Acquire data skills now, or pay big money for big data analysis

Buy-side IT professionals must adapt - or face paying big money to vendors to mine big data, says Ovum

Data scientists are in short supply and IT professionals must develop their analytical skills now to fill the gap, warns Ovum analyst Mike Davis.

The rise of big data has revealed a skills shortfall in transforming it into useful business information. Failure to address the problem now will leave organisations no choice but to pay big money to vendors.

"Data scientists are people who really understand all of the aspects of how information can be used and exploited, and there is a shortage of them," said Davis. "The best candidates are working for vendors, not end-user customers."

Davis' view chimes with that of HP senior VP of information management Nicole Eagan, who also said last week that analysis skills are in short supply, and that this is a challenge even for vendors such as Autonomy, now owned by HP.

Davis emphasised that end-users need to recruit data scientists from universities or face having to pay vendors for their scientists' time. For this reason, 'big data' could soon mean 'big expense'.

"These scientists are trained within universities, and it is incumbent on organisations that really want to exploit their data to start recruiting them. Otherwise, those data scientists will work within vendors and end-users will have to pay for their advice," he said.

Data scientists often decide to build their own data companies instead of working elsewhere within the industry, explained Davis.

"Autonomy was created by people who studied at the University of Cambridge. And IBM-owned Vivisimo, a big data analytics company, was founded by computer science researchers from Carnegie Mellon University in the US," he said.

Ovum analyst Tony Baer predicted that many organisations will try to fill some aspects of the data scientist role by using automated tools.

"Data scientists will prove a very difficult role to fulfil - it is likely that we'll be seeing tools that automate some of the discovery, domain knowledge and mathematics. That will help with the talent shortage, but the data scientist role will never be completely automated," Baer explained.

Davis said that academic roots are not essential, because there are other routes to acquiring an analytical skill set.

For example, programmers who are dealing with big data could turn their attention to open-source software-framework Hadoop, a tool that supports data-intensive applications, he said.

"The demand for Hadoop programmers will rise and as it is completely open source software, employees can acquire the necessary skills in their own time," said Davis.

Ovum's Baer added that big data would mean "an evolution of roles" and that Hadoop will follow other platforms such as Java in creating a demand for specialists.

"It is just like the shortage of Java developers just after Java emerged. Coders will learn these new platforms based on the laws of supply and demand," he said.

The complicating factor for developers is having the skills to set up and run clusters and being able to work with Hadoop in the cloud, added Baer.