Le Clos (20) chased down Phelps in the last 50 metres, touching the wall in one minute, 52.96 seconds (1:52.96) to smash his own national record by well over a second.
Le Clos said he had always admired Phelps. "Ever since 2004 when he won six gold medals, he has been an inspiration and role model," le Clos said.
"I have all his major races on my computer, I think I have watched the 100m butterfly Beijing final, when he beat Cavic by 0.01 seconds, a million times. I have it in seven different languages."
He said that he studied the races and learned from them. "Watching all Michael's races I know he finishes strong. It sounds crazy but I actually thought I was Michael at the last turn," le Clos said.
"When I turned I looked at him and there was a trigger point I just thought I could try do something special. The last 25 metres came in slow-mo and I just remember thinking to myself and my coach saying 'keep it long and make sure you don't shorten up.'"
Le Clos just managed to touch the pad before Phelps, but the South African said he did not really know how he did it. "To be honest, I think I am a bit lucky."
Le Clos's father, Bert, was overcome with emotion when interviewed by the BBC.
Phelps, the defending champion, finished second in 1:53.01.
Le Clos's medal added to the gold won by Cameron van der Burgh in the men's 100m breaststroke final on Sunday.
South Africa currently sits at tenth in medal rankings with two gold medals and although Japan has scooped up 13 medals so far, they sit at number 11. China and the United States have found themselves comfortably seated at the top of the list.
Back in the water
An hour later Le Clos was back in the water, swimming the third leg of the men's 4x200m freestyle relay final.
The South African quartet, consisting of Darian Townsend, Sebastien Rousseau, Le Clos and Jean Basson, got off to a super start, with Townsend going out hard on the first leg.
They faded with Rousseau dropping back on the second leg, and they were unable to close the gap, clocking seven minutes, 09.65 seconds (7:09.65) for seventh place.
The US won in 6:59.70, with Phelps bagging his 15th Olympic gold medal.
Meanwhile, Gideon Louw narrowly missed out on a place in the men's 100m freestyle final, finishing fourth in the second semifinal in 48.44 seconds.
Louw, who set a personal best of 48.29 in the morning heats, was ninth fastest in the penultimate round, falling 0.06 seconds shy of a place in Wednesday's final.
Compatriot Graeme Moore failed to progress beyond the preliminary rounds, finishing seventh in his heat in 49.29.
Best performance of the Games
Elsewhere, the South African men's lightweight fours rowing team of Matthew Brittain, Sizwe Ndlovu, John Smith and James Thompson qualified for Thursday's A final, finishing second of six boats in six minutes, 04.21 seconds (6:04:21) in the second semi-final.
Banyana Banyana produced their best performance of the Games, holding world champions Japan to a credible goalless draw in their last pool match of the women's football tournament.
Holding fort at the back for 90 minutes, the South Africans did well to keep out a relentless Japanese attack in their final appearance on Olympic debut.
The South African women's hockey side lost their second successive match of their campaign, going down 4-1 to world number six New Zealand.
Controversial equestrian Alex Peternell finished 49th of the 53 competitors who completed all three rounds of individual eventing.
Peternell won a case on the eve of the Games, at the Court of Arbitration for Sport (CAS), which forced the SA Sports Confederation and Olympic Committee (Sascoc) to include him ahead of Paul Hart, who was initially selected as the country's only horse rider.
In tennis, Serena Williams was not selected to play in mixed doubles, while her sister Venus moved into the third round after beating Aleksandra Wozniak of Canada 6-1, 6-3.
Williams must play every day to reach the final on Saturday, and the question is whether she can keep it up as she fights an autoimmune disease that causes fatigue. She is also playing doubles with her sister, Serena.
Serena Williams had also wanted to play mixed doubles. But US coach Mary Joe Fernandez said on Tuesday that it would have stretched her too thin.
Andy Roddick played his last day at the games after losing the second round 6-2, 6-1, to Novak Djokovic of Serbia. British hopeful Andy Murray, seeded third, advanced by beating Jarkko Nieminen of Finland 6-2, 6-4.
Olympic tennis also saw a new record when Jo-Wilfried Tsonga of France defeated Milos Raonic of Canada, 6-3, 3-6, 25-23. The previous record was 30 games, set in 2004. – Sapa-AFP