Although big data is seen by some as little more than a buzz-phrase, many organisations are falling over themselves to recruit staff with the skills to exploit it. If data is the new gold, then this is the gold-rush.
Skilled data analysts and data scientists are in short supply, with the risk being that people with the right skills will be snatched up by vendors, who then sell their expertise on to their customers at a premium. Certainly, that was the opinion expressed by HP to Computing in May 2012 – shortly before the company let some of its most skilled data experts, such as Autonomy’s Mike Lynch, go.
So it seems that not everyone is really convinced that big data analysis skills are needed and that these will be an employment hotspot for IT professionals. There has always been data, so what makes “big data” so special that a new industry needs to be built around it?
Among the naysayers is Kate Craig-Wood, CEO and co-founder of managed hosting provider Memset, which is one of the SME suppliers on the UK government’s CloudStore.
“The use cases for big data techniques [versus those of traditional relational databases] are vanishingly rare; few new skills are needed. I think big data is a bit of a fad, and most cases can be solved by better code – and patience,” she says.
Craig-Wood is herself currently working on a big data project, involving analysis of a large dataset with billions of entries. However, rather than tearing her hair out competing with the rest of the market for those precious analytics skills, she is doing it herself using SQLite, a relational database management system.
“There are some cases, but relatively few, that can’t be handled with decade-old techniques. I think big data is the latest money-spinner for consultants targeting technically clueless mega corps,” she says.
Andreas Bitterer, research vice-president at analyst firm Gartner, largely agrees with Craig-Wood, but suggests that the complexity of data has increased as new technologies have created new types of data, and enabled their capture.
“Big data isn’t really new, largely because ‘big’ is relative, and some organisations addressed these challenges a long time ago. It’s really the variety - with data now coming from sensors, smart meters, or social networking - that adds new complexities to a data-management programme, because organisations need to understand the context and can’t just throw more hardware at the problem.”
Others in the industry argue that the scale of big data alone justifies a drive to acquire in-depth analysis skills.
Geoffrey Taylor, head of academic programmes, SAS UK and Ireland, says that big data is going to be of such importance that employees across the organisation, and not just an expert elite, are going to need analysis skills.
“Data is data, so there’s an element of truth to say that there’s nothing new. The ways in which data is put together, stored then manipulated and analysed are pretty much standard. The big difference is the scale of it.
“So while some techniques might be familiar to people, the size of the data has increased to such a degree that, whereas previously big data was a specialised area in large organisations, those skills will need to be more widespread.
“You need to store the data, manage it and analyse it, so the worlds of analytics, IT and data management cross over. We are often asked by our customers, in the financial services industry and central government especially, where they can find analytical skills to manage increasing volumes of complex data.”
According to SAS, “big data” could represent “big opportunity” for the UK, potentially adding economic value and jobs at a time when both are in short supply.
“We think that big data could add £200bn to the UK economy, and create up to 60,000 jobs. Those positions are in managing and analysing big data, but where are those skills going to come from?” asks Taylor.
To answer this question, the first step is to do some data analysis of our own and examine the skills base of UK plc.
Computing recently carried out a survey of its readers to assess the UK’s readiness to manage and extract value from big data. The results chime with Taylor’s concerns that the UK is not ready.
When asked if they had skills in analytics or data management (including RDBMS, BI, MDM, and data mining), 59 per cent of Computing’s readers (the majority of whom represent firms with over 500 employees) said no.
By eliminating high entry costs for big data analysis, you can convert more raw data into valuable business insight.
A discussion of the "risk perception gap", its implications and how it can be closed