The Dutch striker's earlier penalty miss with his side trailing 2-1 on Sunday looked to have condemned United to a second defeat in their opening three league games but he made amends with two poacher's goals in the dying minutes to steal the points.
Van Persie's thumping 92nd-minute header was his 100th Premier League goal and his fourth in three appearances for United since he joined from Arsenal for £24-million in the transfer window.
Earlier, Lukas Podolski, the man tasked with filling the prolific Dutchman's boots for Arsenal, opened his account in English football in a 2-0 victory at Liverpool.
It was Arsenal's first win, and first goals, of the season after two 0-0 draws, while Liverpool's solitary point from their opening three fixtures is their worst start for 50 years.
In the day's other match, Newcastle United were held to a 1-1 draw at home by Aston Villa, leaving Southampton as the only top-flight side still without a point.
Van Persie scored 37 times for Arsenal last season to clinch the Premier League's Player of the Year award and has carried that prolific form to United.
"He's got four goals in two starts which is a great statistic and he'll get better," United manager Alex Ferguson, whose side climbed to fifth with six points, told Sky Sports.
But until the 87th minute it was Van Persie's fluffed penalty which was set to make the headlines.
Shortly after Patrice Evra's slip allowed Morgan Schneiderlin to restore the home side's lead, Van Persie was felled in the area by Jos Hooiveld.
After placing the ball on the spot he dispensed with his usual penalty-taking style, electing instead to try and dink the ball past Kelvin Davies but his execution was poor and the keeper was able to claw the ball away.
Newly-promoted Southampton had taken the lead after 16 minutes when Rickie Lambert climbed high to head past Anders Lindegaard after United had lost possession in midfield.
Van Persie levelled eight minutes later, chesting down Antonio Valencia's pass and smashing left-footed past Davis.
United never looked secure at the back, though, and Schneiderlin headed Lambert's cross past Lindegaard after 56 minutes, finding himself unmarked following Evra's slip.
After Van Persie's penalty miss the home fans were preparing to celebrate a famous win, only for Van Persie to ruin the party. First he prodded home from close range after Rio Ferdinand's header rebounded off the post.
Then, deep into added time, he made sublime contact with a corner to flash a header past Davis.
"We were a bit lucky with the 2-2 and the dramatic end," Van Persie told Sky Sports.
"I have to say a big thank you to Paul Scholes. When he came on, every single pass he hit was the right one. With him you are always on your toes. For me, he was the man of the match."
Of his penalty blunder, he said: "I don't know what I was thinking with the penalty. I was going to hit it hard but at the last second I changed my mind."
Van Persie's departure had caused furrowed brows among Arsenal fans but the north London club kick-started their season with German striker Podolski proving that they may still prosper despite the sale of the Dutch goal-machine.
When Liverpool skipper Steven Gerrard gave the ball away after 31 minutes at Anfield, Arsenal swept down the other end in a flash and Santi Corzola, another of Arsene Wenger's new signings, played a perfect pass for Podolski to lash a first-time shot past Pepe Reina.
Liverpool, for whom strikers Luis Suarez and Fabio Borini rarely threatened, were lightweight in attack and there was no way back for Brendan Rodgers's side when Spaniard Cazorla sealed the points after 67 minutes.
"Overall, I am pleased with the quality of our performance," Wenger told the club's website. "Everybody understood very quickly that Cazorla would not take six months to settle and Podolski is deadly when there is a chance."
Rodgers, who has few attacking options after Andy Carroll was loaned to West Ham United, finds himself in the bottom three.
"We created enough chances but were not clinical enough," he said. "There is a lot of good to come from the game but we lost and we need to improve. We'll get better as our season progresses." – Reuters