If 2013 was the year that the world at large woke up to big data, 2014 is when it rolled its sleeves up and started to put the theory into practice.
This is apparent from the results of a wide-ranging Computing study conducted between January and March this year, which included in-depth interviews, focus groups and an online survey that went out to 360 IT decision makers.
Of course, big data is still a minority pursuit. Among those surveyed, just 11 per cent had got to the stage of using big data technologies in the production environment. This should not be a surprise: just as not every organisation needs an ERP or ECM system, so big data technologies in their current form are not for everybody.
Hot on the heels of the pioneering 11 per cent were a further nine per cent who were trialling big data technologies in earnest, and another 24 per cent who were investigating the possibilities, perhaps taking advantage of the fact that many of the cornerstones of the technology, such as Hadoop and NoSQL databases, can be downloaded and used for free.
“Whereas last year the focus was on big data to solve questions they hadn’t thought through, this year it’s more about challenges they want to solve,” said a big data lead in a marketing firm during a focus group discussion.
For more and more companies, then, big data is moving from the theoretical to the practical.
There have been other changes, too. Computing asked respondents to pick the most significant big data developments they had seen over the past year in terms of the effect on their business or in general (figure 1).
[Click on image to enlarge]
The increasing use of infrastructure made available as a service was the top answer among less advanced users.
“The big change is Amazon’s maturity. With Amazon Web Services you can spin up a server in five minutes,” said the CIO of a national newspaper.
Cloud-based big data services are also on the rise: “I get probably one call a week from someone saying ‘do you need a big data service’?” said one CIO.
But it certainly isn’t just about cheap and scalable provisioning and cloud services. Interestingly, those organisations that have already got going with big data, either on a production or a trial basis, put something else top of the list: moving from analysing historical data to real-time analytics.
Augmenting the historical analytics provided by traditional BI systems by crunching data as it comes in is of course one of the main promises of big data technologies, with uses such as the analysis of social media streams to discover the public’s view on a product or service, recommenders in online stores, the automation of resource allocation based on real-time data from sensors, and numerous applications in healthcare, manufacturing and engineering. These are areas where big data can make a big difference.
Those who had dipped a toe in the water were also more inclined to think that the past 12 months has brought a more nuanced understanding of big data and that the hype phase is being surpassed by the arrival of genuine use cases. This means that businesses will be more willing to fund initiatives.
“The market is becoming more mature... we are moving away from the hype into the practicality of certain initiatives, we are certainly seeing funding coming through,” said a CIO in higher education.
Funds for applied big data research are also being made available by government, such as in the case of the £100m Genomics England fund for DNA mapping. This was seen as likely to have a galvanising effect outside of the field being funded, by helping to create standards for meta data.
“One of the potential contributions [from the public sector] is they set the rules and the mechanism for well-defined meta data. So the whole industry can follow, instead of coordinating among themselves to set a standard or pseudo-standard. This will help with searching, for sharing in the future,” said the CIO of a larger retailer.
The art of the possible
The increasing number of use cases also revealed the art of the possible. The initial excitement about big data led to much talk about using technology to ask the questions you hadn’t even thought of asking. It is a sign of its growing maturity that we hear less of such talk now and more about practical applications to real business cases.
“We’re getting ideas and inspiration from case studies, whereas last year people struggled to find well-defined cases. There are more case studies on how companies have tied social data to their own data to drive insights. Whereas last year the focus was on big data to solve questions they hadn’t thought through, this year it’s more about challenges they want to solve,” said a big data lead in marketing.
“We’re not really looking at uncovering new opportunities; rather it’s about better management and access,” explained the CIO of a law firm.
There are exceptions to the pure/applied divide, of course. In research, engineering, healthcare and other areas the “business case” may be predicated on opening up previously hidden pathways and following them to see where they might lead, but in other industries the IT or big data team will not have the luxury of time or budget to chase chimeras.
“We are a cost centre, and with trial and error you don’t always see a benefit,” said the CIO at a retail bank, illustrating what is undoubtedly the reality facing many.
A receptive culture
That said, a forward-looking culture that allows for trial and error and can take a few failures in its stride was seen as essential for success with big data technologies, as was a CFO who is willing to stump up budget without expecting an immediate return. This willingness will likely be very dependent on sector: in some data-heavy verticals such as retail, publishing and online services the landscape is changing fast and they need to start developing skills and knowledge to be able to adapt.
“We’ve looked at Hadoop, Mongo and Storm and we’ve had pilot projects that haven’t got anywhere, but the good side is now we have people who know those technologies should we choose to use them again,” said the CTO at an online services firm, explaining why gaining experience can be a benefit in itself.
Panellists, interviewees and survey respondents were asked about the cultural, political and organisational factors they thought were needed to make the most of big data opportunities. The results are summarised in the box below.
Perhaps the most fundamental factor is getting buy-in from the top.
“If there aren’t very senior people driving it, the company stands no chance of making good use of it. No matter how good the skilled persons are in understanding data and exploiting data, they haven’t got political clout,” said the CIO of a supermarket chain.
• Part 2 of this research will be published later this month.