Why ‘Big Data’ is wearing thin

It is common knowledge that, if you want to get an IT budget approved, there are certain phrases you need to throw into the proposal that hit such a nerve that the Chief Financial Officer will immediately approve the budget. One such phrase is “Big Data”.

However, the phrase is wearing thin. Touted as the key to solving all of a large company’s ills , it has been at the peak of the Gartner Hype Cycle since 2013 and is now falling into the so-called “trough of disillusionment”.

In 2014, according to the IDC, $125-billion was spent by companies on hardware, services and software to deal with Big Data. However it “has not delivered on massive promises”, so these projects are starting to fall out of favour, says SurveyMonkey chef executive Dave Glodberg, speaking at at CeBIT 2015 in Hannover this week.

It would easy to dismiss Big Data as failing due to too much data being collected in too many silos which are still not connected to each other. However, there is another side to Big Data which Goldberg believes is missing from the discussion. He talks about “Implicit Data” and “Explicit Data”, arguing that both distinct types of data are needed.

Implicit Data vs Explicit Data
Implicit Data is what we currently think of when we talk about Big Data. It is data that is routinely collected such as that from clicking on a mouse, listening to song, entering and analysing search queries. We get this type of data on a massive scale, we measure everything and it is this type of data that concerns people about their security and data privacy.

However Implicit data gets it wrong.

The credit card companies analyses everyone’s purchasing habits and suddenly discovers expenses like teeth-whitening, gym membership and hotel rooms. Therefore an assumption is made, based on this Implicit data, that this person is getting divorced. The data does not take into account surrounding circumstances, such as upcoming business travel.

Shopping engines recommend products based on historical implicit data purchases. However, if shopping for a gift for a friend, this would throw off the personal recommendation engine.

“More data doesn’t lead to better information and this is how Implicit Data gets it wrong,” says Goldberg. “If you want to know how someone is feeling, what music they like – you need to ask them. Analysing their searches doesn’t lead you to the correct answer as much as asking.”

Asking is what one can call Explicit data.

Explicit data is data we get by asking questions and receiving answers.

The internet has allowed companies to get this type of data en masse by asking the right questions to the right groups of people.

At one stage, Google realised it was losing female staff, which concerned the company. Instead of relying on Implicit data of female search history, it surveyed the staff and realised it didn’t have a “female vs male” issue but a “new mother” issue. Females who were pregnant or thinking about becoming moms did not like Google’s maternity policy. As soon as Google discovered this, it fixed the issue and retained its female staff.

SurveyMonkey uses Implicit and Explicit data in order to make decisions. Mining the data and asking customers directly allows the company to thrive.  And it is not the only one. In a single day SurveyMonkey has 3 million survey responses, answering 29 million questions, which results in 80GB of new data per day

The key message behind Goldberg’s talk was that relying on causation without correlation didn’t paint a true picture. Rather, companies must have the conversation with their customers and listen to what they have to say. 

“Encourage customers and employees to give you feedback. People feel powerful and loyal when their voices are heard.”

  • Liron Segev is also known as The Techie Guy. You can read his blog here or follow him on Twitter on @Liron_Segev
  • Follow Gadget on Twitter on @GadgetZA

Subscribe to the M&G for R2 a month

These are unprecedented times, and the role of media to tell and record the story of South Africa as it develops is more important than ever.

The Mail & Guardian is a proud news publisher with roots stretching back 35 years, and we’ve survived right from day one thanks to the support of readers who value fiercely independent journalism that is beholden to no-one. To help us continue for another 35 future years with the same proud values, please consider taking out a subscription.

And for this weekend only, you can become a subscriber by paying just R2 a month for your first three months.

Related stories

We must keep Covid-19’s gains in education

COMMENT: The disaster regulations mandated zero-rating for some websites. Now it is time to expand this access

Africa study finds three million new genetic variations

The findings have wide relevance, from learning more about African history and migration to specific variants of people’s health

The future of learning is here — and it’s mobile

The smartphone is a crucial tool for enabling lifelong learning and has the potential to democratise education, particularly with internet penetration and access to data both continuing to rise

Tracking, tracing and transparency

Governments are processing tons of personal information to limit the spread of Covid-19. They must ensure this does not cost us our privacy

South Africa’s southern right whale population is falling, research finds

Researcher Els Vermeulen has a whale of a time conducting an annual aerial survey of the country’s southern right whales. But decreasing numbers reveal the depressing nature of this work

Hollowing of skills and defunding of Stats SA works for a failing state

Perhaps the government does want us to know how bad things really are

Subscribers only

ANC: ‘We’re operating under conditions of anarchy’

In its latest policy documents, the ANC is self-critical and wants ‘consequence management’, yet it’s letting its members off the hook again

Q&A Sessions: ‘I think I was born way before my...

The chief executive of the Estate Agency Affairs Board and the deputy chair of the SABC board, shares her take on retrenchments at the public broadcaster and reveals why she hates horror movies

More top stories

DRC: Tshisekedi and Kabila fall out

The country’s governing coalition is under strain, which could lead to even more acrimony ahead

Editorial: Crocodile tears from the coalface

Pumping limited resources into a project that is predominantly meant to extend dirty coal energy in South Africa is not what local communities and the climate needs.

Klipgat residents left high and dry

Flushing toilets were installed in backyards in the North West, but they can’t be used because the sewage has nowhere to go

Nehawu leaders are ‘betraying us’

The accusation by a branch of the union comes after it withdrew from a parliamentary process

press releases

Loading latest Press Releases…