Thirty one people have been confirmed dead.
The St Thomas Aquinas ferry was carrying 831 passengers and crew when the accident occurred on Friday night in a dangerous choke point near the port of Cebu, the Philippines' second biggest city, authorities said.
Coastguard and military vessels, as well as local fishermen in their own small boats frantically worked through the night and Saturday to haul more 629 people out of the water alive, Transport Secretary Joseph Abaya told reporters.
But by Saturday afternoon, 171 people were still unaccounted for and 31 bodies had been retrieved, Abaya reported shortly after navy divers suspended efforts to reach the submerged hull because of bad weather.
"It rained hard… with strong winds and rough seas," navy spokesman Lieutenant Commander Gregory Fabic told AFP, explaining why dives had been suspended.
He also said powerful currents had earlier prevented divers from assessing all of the sunken ferry to determine how many people had died and were trapped inside the vessel.
Fabic said rescuers had not given up hope that there were other survivors who were either still drifting at sea, and searches would continue for them.
Plunged into cold water
But Rear Admiral Luis Tuason, vice commandant of the coastguard, cautioned the death toll would inevitably rise.
"The captain managed to declare abandon ship and they distributed life jackets but, because of the speed by which it went down, there is a big chance that there are people trapped inside," he said, adding the ferry sank within 10 minutes of the collision.
One survivor, Maribel Manalo, 23, recounted to her brother the horror of suddenly being plunged into the cold water in darkness, and emerging from the chaos without her mother.
"She said there was a banging noise then the boat suddenly started sinking," the brother, Arvin Manalo, told AFP.
"They quickly strapped on life jackets and then jumped into the dark sea. She said they felt like they were pulled under. My sister said she pushed our mother up, but they got separated.
"My sister was rescued. My sister knows how to swim, but my mother does not."
He said their mother, 56, remained missing.
The accident occurred at 9pm (1300 GMT) in calm waters near the mouth of the port between two and three kilometres (around one to two miles) from shore, authorities said.
Local fisherman Mario Chavez told AFP he was one of the first people to reach passengers after the ferry sank in the 82-metre-deep (270-feet) channel.
"I plucked out 10 people from the sea last night. It was pitch black and I only had a small flashlight. They were bobbing in the water and screaming for help," he said.
"They told me there were many people still aboard when the ferry sank… there were screams, but I could not get to all of them."
The cargo ship, Sulpicio Express 7, which had 36 crew members on board, did not sink. Television footage showed its steel bow had caved in on impact but it sailed safely to dock.
Tuason said it appeared one of the vessels had violated rules on which lanes they should use when travelling in and out of the port, without specifying which one.
The enforcement office chief of the government's Maritime Industry Authority, Arnie Santiago, said the strait leading into the Cebu port was a well-known danger zone.
"It is a narrow passage, many ships have had minor accidents there in the past. But nothing this major," Santiago told AFP.
"There is a blind spot there and each ship passing through needs to give way in a portion of that narrow strip."
The Thomas Aquinas was a "roll-on, roll-off" ferry, which allows vehicles to be driven aboard and is commonly used in the Philippines.
Ferries are one of the main forms of transport across the archipelago of more than 7,100 islands, particularly for the millions of people too poor to fly.
But sea accidents are common, with poor safety standards and lax enforcement typically to blame.
The world's deadliest peacetime maritime disaster occurred near the capital, Manila, in 1987 when a ferry laden with Christmas holidaymakers collided with a small oil tanker, killing more than 4,300 people.
In 2008, a huge ferry capsized during a typhoon off the central island of Sibuyan, leaving almost 800 dead.