He was speaking alongside key African and European finance minsters and central bankers, in a discussion on Africa's economic outlook at the World Economic Forum on Africa on Friday.
The continent, despite a global economic recession has managed to sustain average growth rates of above 5% in recent years, thanks in large part to demand for its vast natural resource wealth, greater political stability and steady governance and fiscal reforms.
Gordhan echoed Nigeria's Finance Minister Ngozi Okonjo-Iweala, who warned that while many Africans were optimistic about the continent and its prospects, vulnerabilities remained.
"Our economies are not diversified enough and that's a vulnerability," she said.
In Nigeria's case, the country is heavily reliant on oil. According to the 2013 African Economic Competitiveness Report, more than half of public revenues come from mineral resources and well over 80% of Nigeria's exports were generated by this sector.
The country was now looking to agriculture, an area it "neglected for decades" to create jobs, Okonjo-Iweala said.
As well as looking at other sectors within economies, fiscal policies also had to be strong, she said.
Inflows of foreign capital
Growing the pool of domestic savings was critical to stable development for African countries, according to Sweden's Finance Minister Anders Borg.
While inflows of foreign capital was good for countries with low levels of domestic saving, this could lead to current account imbalances.
The influx of too much foreign capital or "hot capital" was dangerous and left countries vulnerable to the reversal of those flows, he noted.
Meanwhile, Gordhan added that a challenge most African countries still had to overcome was building a wider tax base.
Linked to this however was the "leakage of money from the African continent", he stated.
If this money remained on the continent, it would cut the reliance on aid and ensure Africa had access to its own resources and "investable capital".
Intra-African trade and investment
Better intra-African trade and investment was highlighted as critical to help the continent generated growth that was both sustainable and inclusive.
Okonjo-Iweala cited South Africa's investments into Nigeria as a positive example of this type of investment.
South Africa's companies such as MTN and Shoprite were some of the largest investors in Nigeria's economy, she pointed out, "and they are making very good returns on that investment".
Similarly, Nigeria's companies were also making headway in investing on the continent with companies such as Dangote cement expanding into 33 African countries, she pointed out.
This was a new phenomenon that African leaders should "thoroughly encourage".