Big Data Summit 2014: Shortage of talent endangering big data projects

Shortage of data scientists and big data specialists means organisations need to use more automated tools

Organisations' big data initiatives could be derailed by a lack of talent to implement them, according to master data management firm Talend's Mark Carter.

In the US alone, there is an estimated shortage of 200,000 big data professionals who, as a result, can attract bumper salaries - typically between $150,000 and $200,000. "These people are as rare as hens' teeth," said Carter, speaking at Computing's Big Data Summit 2014.

However, four-fifths of their time is typically spent on administration and office politics because organisations simply aren't set up to do big data. "Organisations have lots of different data silos and he or she has to get all that data together first," said Carter.

Instead, he said, organisations need to leverage the talent that they have in-house and to make the process easier with automated tools, cutting the in-house knowledge of complex technologies like Hadoop and MapReduce in the process.

"Why go outside your organisation when you can use the resources you have got inside?" asked Carter.

Trying to do big data on traditional databases, though, is also a mistake, he warned.

But most organisations are perhaps not as behind in big data as they might think: the vast majority worldwide still have not advanced to the stage where they are able to do real-time analytics.

"Some people seem to think that big data is like the Holy Grail, something to be attained or reached. It's not, it's more like a journey," said Carter.