A quick look at the marketplace tells us that big data has come to some kind of inflection point. Venture capital, it seems, cannot get enough of big data start-ups, and the majors are snapping them up too as they seek a piece of the action.
We are moving from a picture in which a few pioneers have been blazing a trail to one where that trail is being followed by more conventional organisations. We are seeing more use cases, a better and more detailed understanding of the possibilities offered by linking different types of data at scale, and the emergence of enterprise-class technologies to allow this to be done.
For more and more companies, big data is moving from the theoretical to the practical. So who are these organisations that are already putting big data theory into practice?
Advanced users, by which we mean those who are using big data technologies and techniques on an operational or trial basis, span all sizes and sectors, but according to a major online survey by Computing they are disproportionately large companies with 5,000 or more employees, and they are disproportionately in the banking and finance, healthcare and IT and telecoms sectors.
While IT is still most likely to be leading a big data project, among more advanced users the business as a whole was found to be taking a more prominent role - 27 per cent compared with eight per cent in the case of less advanced users (those investigating solutions but who have yet to put it into practice).
"Last year it was a pet project within IT. This year we are seeing more demand from the business... The business has seen the benefits of the timings and the information they can get by using these solutions," said the CIO at a bank, interviewed as part of the research.
No doubt related to who is in the driving seat is the way that companies view data with respect to their overall strategy. A massive 44 per cent of more advanced users said, "Our overall strategy, driven by the business, is to find new opportunities by making the most our data", compared with just 28 per cent of less advanced firms.
An even bigger divergence of opinion occurred with the question of competitiveness. "Big data is a key differentiator to ensure success for us and our clients," said 27 per cent of advanced users - those who have already got up and running with big data to some degree - compared with a measly three per cent of less advanced users.
The vision thing
[click to enlarge]
One thing that unites more advanced users is having a data-based vision. Being able to look ahead and spot the opportunities. They are analysing operational data to obtain insights; they can integrate and process data in many different formats; they can store the data in a flexible and agile way; and they have the key skills in house.
Key to supporting the company's vision is having a good understanding of the data it has - or lacks - that can be put to use to increase efficiencies, create new products or services or to provide a competitive edge.
So big data is very much a data governance issue: knowing what you have, knowing where it is, knowing what other data is out there that you could combine with your own internal data to gain insights.
It certainly isn't just dumping everything into Hadoop and hoping rays of insight will start magically shining out.
"Putting everything into Hadoop without thought or expected outcome makes you a dumb organisation ..." said a big data leader in a large marketing organisation.
Indeed it does. Or at least it can. While most organisations need to keep the business case firmly in view, tech companies and researchers may well find it beneficial to spark up a low cost open source solution and play around with it.
For more general business programmes though, respondents voted strongly that big data initiatives should be a shared endeavour, with 55 per cent agreeing that "big data should be managed by a multi-disciplinary teams (including IT, business heads, analysts, industry experts etc...) if it is to be exploited to its full potential" and a further 26 per cent saying that it should be the remit of several departments working together.
"I think it's putting together that cross-functional team is key. Don't just leave it in the hands of the IT guy. Equally, don't just leave it in the hands of the business..." said a project manager in research.
"It's almost like having a virtual team that spans across many aspects of your organisation...," said a CIO in finance.
Not all agreed, however.
"Big Data is just data; it is what IT does," said the head of IT at a legal firm.
Of course, big data is a broad-brush term, covering a lot of different projects, some of which may be purely IT-based. For others though, the creation of a broad based multidisciplinary team helping each department make the most of its data, may be a prerequisite for big data success.
This is a transcript of part of the keynote speech delivered at Computing's Big Data Summit 2014 by chief reporter Graeme Burton (pictured) today.