Big data: 2012 in review

Bookended by a picture of confusion at one end and a statement of affirmative action by government at the other, 2012 has been a pivotal year for big data in the UK.

Back in January, Computing asked IT managers at large organisations what the term "big data" meant to them. Twenty-eight per cent replied "vendor hype", 37 per cent viewed it as a big problem (seeing big data as synonymous with increased data volumes), while just 27 per cent said that it represented a big opportunity.

A couple of months later we welcomed a capacity crowd to our Big Data Summit in London. Chatting to attendees we discovered that more than a few had turned up "just to find out what this big data thing is all about".

Fast forward to November, when Chancellor George Osborne made a speech to the Royal Society insisting that big data analysis will be one of the cornerstones of the UK economy in the years to come.

Now, when politicians start to bandy around a previously niche technical term like big data, you can be sure it has finally made its way out of the server room and into the board room. Others agreed.

"Big data has started to go mainstream," Joe Morrissey, VP EMEA of NoSQL firm 10gen, told Computing. "Over the last two years 500 million additional people have gone online. Cloud, mobile and social media are changing the way people use the internet and as a result a lot more data is being generated. The interesting thing going forward is how do we extract value from that data?"

Martin Moran, senior VP and general manager EMEA at vendor ServiceSource, went further: "The explosion of interest in big data in 2012 has created a new focus for businesses of all kinds. The quality and quantity of analytics tools is increasing, as is the demand from the boardroom for business decisions based on quality, timely data. We are seeing the marrying of science and business, creating new opportunities in sectors across technology and IT," he said.

And it was not only business that was seeing the potential.

"The use of big data analytics could save the public sector between £16bn and £33bn a year," said think-tank Policy Exchange in June.

Skills gap

The realisation that there may be gold in them thar big data hills led some to voice concern that qualified data scientists, capable of sifting nuggets from the dirt were dangerously thin on the ground. Both vendors and educational establishments moved fast during 2012 to fill that void.

The UK's first masters degree in data science, to start in 2013, was announced by Dundee University. Meanwhile in December analytics firm SAS announced a sponsored academy at Birmingham City University, in a stated bid to resolve the big data skills gap. According to SAS, demand from employers for the key big data skills NoSQL and Hadoop increased by 295 and 210 per cent, respectively, during 2012.

Not everyone agreed that this was the dawn of a new age, however. "Big data is a big scam most of the time," said Daniel Austin, chief architect at Paypal, who told Computing that for the vast majority of tasks traditional relational database management systems (RDBMS) work just fine.

[Turn to next page]