Abe will be the first Japanese prime minister to visit these former Eastern Bloc countries. Under the deal, Japan apologized again and promised 1 billion yen ($8.99 million) for a fund to help the women.
Japan, Mr Abe said, "has been carrying out all the promises it made in good faith" and urged South Korea to "firmly follow through" on the pact. Under the agreement, Japan apologized and paid 1 billion yen ($8.9 million) to a Korea-run foundation to support the victims in return for Seoul's promise not to raise the issue again in global forums.
His remarks came after President Moon Jae-in on Wednesday also called the 2015 deal "unsatisfactory" and demanded Japan accept historical truth and offer a honest apology to the victims in a nationally televised press conference.
He said that although it was a political agreement, it excluded the victims and ordinary South Koreans.
The goal is to make use of the strengths of long-term government and expand the frontiers of Japanese diplomacy, as Chief Cabinet Secretary Yoshihide Suga puts it. Tokyo intends to win widespread acceptance for key policies, such as those regarding North Korea and the European Union trade deal.
This is like punching someone in the face and then saying to them, 'I hope it doesn't ruin our relationship.' It just doesn't work that way and Japan needs to stand firm.
Yoshihide Suga, the top Japanese government spokesman and the Chief Cabinet Secretary, said Wednesday: "Japan can never accept it if the South Korean side demands we take further measures despite the agreement that confirmed the final and irreversible solution on the issue".
In an editorial Friday, the Mainichi Shimbun said Abe's absence at the Olympics will only leave a strong impression of cooling ties between Seoul and Tokyo, which would only benefit North Korea.
Between 80,000 and 200,000 women, mostly from the Korean Peninsula, had been recruited to provide sex to the Japanese Imperial Army during World War II.
Japan thought that by striking this deal in 2015, it could put the past away and move forward. Following Moon's remarks, the Japanese government reportedly filed a formal protest with the South Korean Embassy in Tokyo.
Japan bristles at its neighbours continuing to resurface historical wartime issues even after its repeated efforts to resolve them once and for all.
Tokyo has reacted angrily to comments on the issue this week by Seoul. And Japan, increasingly anxious about digital threats from the likes of North Korea and China, is turning to the Baltic state for cooperation in fears of massive harm should such critical infrastructure as nuclear power plants be targeted.
But he said Wednesday: "The fact that there was an official agreement between Korea and Japan can not be denied".