Three foreigners in the United Nations compound died in the attack, along with at least two Somali security guards and three civilians in the surrounding streets, officials said.
Somalia's Prime Minister Abdi Farah Shirdon condemned the brazen daylight raid as a "senseless and despicable attack on innocent UN civilians".
Al-Shabab fighters, who boasted about the killings, used both a car bomb and suicide attackers to blast their way into the fortified base in central Mogadishu near the airport.
Security warnings of an attack have been in place for weeks, and UN staff regularly practise sheltering in a secure bunker inside the compound.
"Our commandos attacked the UN compound … we set off an explosion and entered the compound," a senior al-Shabab official told AFP, saying they entered to attack "the infidel forces."
Somali and African Union troops later moved into the complex – despite the Islamists battling back with heavy gunfire – to end the hour and an half seige.
"The situation is under control … Somali soldiers along with African Union forces stormed the compound and killed the attackers," said Somali police official Abdulahi Osman.
Bloodied bodies
He said three civilians were confirmed killed, caught in the crossfire in the streets near the compound.
UN sources said one staff member and two contractors were killed along with at least two Somali security guards, although the nationalities of the foreigners could not be immediately confirmed.
An AFP reporter saw several bloodied bodies carried away on makeshift stretchers.
"Some of the 'white kuffar'(unbelievers) who tried to engage the mujahedeen in combat inside the offices were killed and thrown out into the compound," the al-Shabab wrote on Twitter.
The UN Assistance Mission in Somalia (UNSOM) said the compound – home to UN humanitarian agencies – had come under " complex attack".
UN sources said a car bomb had been detonated shortly before midday (9am GMT), followed shortly by another blast, believed to have been a suicide bomber.
Series of attacks
An AFP reporter near the scene of the attack reported heavy gunfire after a series of inital explosions, as the fighters battled with security forces.
The compound – including both residential and office areas – is a short distance from the airport zone, the base of the African Union troops, but is guarded by its own security officers.
"The fighting is over, there are many government soldiers all over the area … the attackers appear to have all been killed," said witness Ahmed Sudi.
Somalia's capital has been hit by a series of attacks including suicide and car bombers, mortar attacks and shootings, although in recent weeks the city has been relatively calm.
Islamist al-Shabab militants used to control most of the seaside capital until they abandoned fixed positions in August 2011, but the insurgents have since carried out a string of attacks against the UN-backed government.
The last major attack was in April, when the al-Shabab sent a nine-man suicide commando unit to blast its way into Mogadishu's main court complex.
Courthouse assault
Some of the commandoes blew up their explosives vests while others sprayed gunfire in a rampage that killed 34 people.
Reports suggest that the attack on the UN compound used similar tactics to the courthouse assault.
The 17 000-strong AU force, fighting alongside Somali government troops, has forced the al-Shabab from a series of key towns.
While riven by infighting and hunted by US drones, al-Shabab extremists remain a potent threat, launching car bombs and assassinations, and are still powerful in rural areas as well as reportedly infiltrating the security forces.
Shirdon condemned the attack.
"The UN are our friends and partners, and the UN agencies offer us humanitarian help and support, so I and all Somalis are appalled that they should be the target and victims of such barbaric violence," he said in a statement.
In 2008 a car bomb attack on the compound of the UN Development Programme in the self-declared independent state of Somaliland left two UN staff dead. – AFP