Nadal, who lost to eventual champion Novak Djokovic in the Wimbledon semi-finals last month, beat former Grand Slam victor Wawrinka 7-5 7-6 (7-4) in Toronto.
Tsitsipas, ranked 27, spent just over two hours in advancing to the first Masters 1000 quarter-final of his career.
Tsitsipas reached his first ATP World Tour final earlier this year at a clay-court event Barcelona, losing to Spanish great Rafael Nadal in the championship match.
Tsitsipas will now face either Germany's Alexander Zverev or Russian Daniil Medvedev for the place in the semis.
Tsitsipas set up match point with a scorching cross-court victor before completing the victory in 2 hours, 17 minutes.
"I'm happy to see Stan playing well again".
Novak Djokovic, of Serbia, celebrates defeating Peter Polansky, of Canada, at the Rogers Cup men's tennis tournament in Toronto, Wednesday, Aug. 8, 2018.
He won on a third-set tie-break - his second in as many days - 7-6 (7-1) 3-6 7-6 (7-4). "I'm putting Greece more deep into the map of tennis", Tsitsipas said.
He held serve to open the set and wrong-footed Djokovic after the veteran served at deuce, setting up an overhead smash for the advantage before a brilliant backhand victor secured a 2-0 lead.
Tsitsipas had beaten seventh seed Dominic Thiem to set up a meeting with Djokovic, the ninth seed, but said his triumph over the former world number one was the best of his career. I tried few a things that I didn't try before and I played a bit better.
"I mean, Denis is already established top 50 player, and he's making his way up", he added.
Earlier, fifth-seeded Bulgarian Grigor Dimitrov outlasted American Frances Tiafoe 7-6 (1), 3-6, 7-6 (4) and fourth-seeded South African Kevin Anderson topped qualifier Ilya Ivashka of Belarus 7-5, 6-3.
His trademark retrieving skills were on display throughout the match but Tsitsipas used strong groundstrokes to push the pace and force Djokovic to fully stretch on several rallies. Dimitrov has won six of those matches.
"To be honest, we played three sets and all three sets should have been three, three and three", Zverev said.
Toronto's Daniel Nestor - the oldest player at the tournament at 45 - and Vancouver's Vasek Pospisil were scheduled to play their first-round doubles match in the late afternoon.