"We have taken a decision to withdraw our soldiers," Zuma was quoted as saying by the SABC on Thursday at the end of a summit of Central Africa's leaders.
The deal is no longer valid due to the Central African Republic (CAR) government's fall, he said.
Zuma faced questions over the presence of the troops following the gunning down by rebels of 13 South African paratroopers in the battle that saw rebels seize the capital Bangui late last month.
"We were in CAR on the basis of the agreement between the two countries," he said at the end of the summit, which ended late on Wednesday.
"Our mission was to help train the soldiers … since the coup and the self-appointment of rebels, it was clear that the government is no longer there," Zuma told SABC at the end of a meeting of the Economic Community of Central African States (Eccas) in Chad's capital Ndjamena.
The decision to withdraw was first announced by summit host, Chad's leader Idriss Deby, late on Wednesday. He added that Zuma was however ready to provide troops in the future if necessary.
The killing of the 13 troops, the country's heaviest military loss since the apartheid era, stirred anger at home.
Official opposition party Democratic Alliance (DA), which threatened to lodge an urgent resolution in Parliament to force the immediate withdrawal of the troops, welcomed Zuma's decision to bring back home the troops.
"This is especially welcoming news for the family and friends of the troops stationed in the CAR," DA leader Helen Zille said in a statement Thursday.
The opposition wants Zuma to ensure that "our soldiers are returned to South African soil immediately".
The opposition however still demands "a full explanation of why South African troops were deployed in the CAR in the first place".
Over 200 troops were in the troubled African country last month, but it was unclear how many remained there after the fall of Bangui. Local media reports suggest a few dozens may still be inside the restive country.
The deaths of the soldiers prompted allegations that the troops were deployed to protect commercial interests, but Zuma rejected those claims.
The Mail & Guardian reported last week that South Africa's military involvement in the Central African Republic has from the start been entwined with ANC-linked business deals.
South Africa sent in extra troops into Bangui this year as protection for troops already on a training mission, which included instructing a VIP protection unit, as part of a 2007 pact.
But it also stated in 2011 that soldiers were deployed to provide VIP protection to fallen president François Bozizé who fled his country after the capital fell.
Meanwhile, African leaders said they would not recognise Central African Republic (CAR)'s new self-proclaimed leader, as the nation's post-coup crisis came under the spotlight at a regional summit in Chad on Wednesday.
A transitional president should be elected in place of Seleka rebel leader Michel Djotodia, Deby told journalists at a meeting of the Economic Community Of Central African States (Eccas).
"As things stand now, it is impossible to recognise a self-proclaimed president," he said.
"A committee selected by national figures must lead the transition. This body will have the executive role and must vote for a transitional president" who would serve for not more than 18 months, he said.
Djotodia grabbed power on March 24 after a rapid assault on the capital Bangui in revenge for unfulfilled promises in a January peace deal with the government.
He has retained Prime Minister Nicolas Tiangaye in his post and says he will hold elections by 2016, a promise the international community is keen for him to keep.
Deby said a legislative body would be created to take on the role of a Parliament and a mission of ministers from Eccas, the African Union and the European Union, among others, dispatched to the Central African Republic's capital Bangui on Thursday "to take the message to Central Africans".
Chad, the region's dominant military power, hosted the Eccas meeting as a way to give the rebel Seleka coalition regime in Central Africa a semblance of legitimacy, according to a diplomatic source.
Missing from the talks was CAR's ousted leader Bozizé, who said he had been refused a seat at the summit.
Leaders nevertheless called on Benin's President Thomas Boni Yayi to grant an asylum request from Bozizé, last reported to be in Cameroon, "if he asks".
"The ex-president Bozizé is in Cameroon, we have asked Benin's president, if [Bozizé] asks, to welcome him there as a brother," Idriss Deby said.
Bozizé, who fled Bangui amid the rebel assault, has accused his former Chadian allies of backing the rebels who unseated him, in an interview with BBC Africa.
Observers have said Bozizé appears to have been left in the lurch by Chad, a once-powerful ally which helped him mount a coup in 2003, while a recent report from the International Crisis Group mentions Chad's "dubious relations with Seleka".
Some 400 Chadian soldiers formed part of a multinational African peacekeeping force (Fomac), sent to stabilise the country in 2008.
However it was South African soldiers stationed in the capital who came up against the rebels on March 23-24, as they stormed the capital after the January peace deal collapsed.
"It was Chadian special forces that led the operation on the Sunday morning and attacked the base of the South Africans," Bozizé told the BBC.
Defence analyst in South Africa Helmoed Heitman backed up these claims in a detailed account of the battle in the Sunday Independent.
"The 'rag tag' rebel force originally reported" … had "flak jackets, new AK47s and heavy weapons up to 23mm cannons," he wrote in the newspaper.
"It was also clear that many were not from the CAR, some speaking with Chad accents and others having distinctly Arabic features." South Africa's heaviest military loss since apartheid raised prickly questions for Zuma over why his troops were sent to an area where South Africa has no immediate strategic interests.
Zuma on Tuesday said a contingent was sent to train local forces and provide protection for the now deposed Bozizé under a 2007 deal.
But CAR sources close to the presidency and security sources say Zuma and Bozizée had signed accords "giving South African businesses access to oil, diamond and gold riches". In exchange, South Africa would defend Bozizé's regime.
Eccas will hold another summit once the international mission to Central African Republic has completed its work, Deby said on Wednesday, to continue to monitor the situation in the unstable country.
Though looting and clashes have petered out in the capital since the coup, reports emerged on Wednesday of pillaging by both pro-Bozizé soldiers and Seleka rebels in the west of Central African Republic, an area previously spared by the fighting.
A senior Seleka rebel who wished to remain anonymous said on Wednesday the coalition was deploying troops to the region. Seleka rebels themselves are now being disarmed, a source close to Tiangaye said. – Additional reporting by AFP