"The military command of al-Shebaab mujahedeen ordered a tactical retreat at midnight," spokesperson Ali Mohamud Rage told AFP.
The announcement came after an assault on Friday on the city by African Union forces, which had been trying to dislodge the insurgents from the key coastal city for days.
Residents in the city confirmed that the Islamist fighters seemed to have moved outside city lines and that their radio station, Radio Andalus, was off the air.
"We don't know where they went to… but early this morning the last military vehicle left the town," said Hassan Ali, a resident.
"Even their radio station is off air," he added.
Al-Shebaab fighters on the ground also confirmed the withdrawal.
"We got orders from our superiors to withdraw from the city ... This is part of broader military tactics we have set for the enemy," Sheikh Mohamed Abu-Fatma, a al-Shebaab commander, told AFP by telephone.
The Kenya Defence Force confirmed the Shebab withdrawal and said they would be moving into the areas that were under al-Shebaab control at the time of the Friday assault.
"As soon as we consolidate, we will move to take the rest of the city," Kenya Defence Forces spokesperson Cyrus Oguna told AFP.
He said the northern part of Kismayo was under the complete control of the allied forces but as of Friday night "some parts of the south were still under the Shebaab."
Residents claim that the withdrawing militia bust open the gates of the main prison in Kismayo and the police station.
"Last night they have released the prisoners from the jail and I saw three civilians shot dead by Shebaab after accusing them of spying, they left and no one of them is here today," Abdikarim Hussein, another resident said.
Kismayo has been a vital lifeline for the al-Shebaab since the African Union force in Somalia (AMISOM) reconquered most of Mogadishu last year and Ethiopian troops stripped them of other key cities in the west.
The looming assault on the port city which has been on the cards for the past four weeks has led to the displacement of an estimated 12 000 people who have fled the city. Kismayo's total population is estimated at between 160 000 and 190 000.
Al-Shebaab seized Kismayo from Somalia's weak central government in 2008.
The Kenyan military has have been aiming to take Kismayo ever since it rolled troops and tanks across the border to fight the Islamist extremists almost a year ago. Reaching its goal has taken longer than anticipated.
The recapture of Kismayo is seen as a major boost to the newly-established central administration in Mogadishu and would pave the way for government troops backed regional forces to reclaim much of southern Somalia. Observers have consistently said that the loss of Kismayo would leave al-Shebaab, who once controlled 80% of the country, unable to retain large swathes of territory.
The other key al-Shebaab-held towns of Afgoye, Baidoa and the port of Marka have all fallen in recent months.
But in places where the al-Shebaab have abandoned fixed positions, most notably in the capital Mogadishu, they have switched to guerrilla tactics and remain a threat.
Experts have warned that the al-Shebaab—who implement an extreme form of sharia law in the areas they control, amputating thieves and stoning "adulterous" women – can sow just as much chaos and death by reverting to guerrilla tactics. – Sapa-AFP