Welcome to 2013, during which perhaps 30-50 per cent more data will be created than in the previous 12 months. What will this year bring to help us make better use of this increasingly abundant resource?
With big data now hitting the mainstream, we can certainly expect a whole lot more hype and bandwagon-jumping. Just as in the late 1990s it became de rigeur to stick "dotcom" on the end of every other noun, big data will find itself similarly appropriated by all and sundry.
But will investors in big data suffer the same fate as their dotcom counterparts? Will the next Oracle or Microsoft emerge from the new paradigm? And will the technology become usable by non-technical users?
Read on as we recklessly try to draw some linear conclusions from some decidedly non-linear trends.
Will the market for big data technologies increase?
If 2012 saw big data hit the mainstream, 2013 will see more and more organisations attempt to make the technology work for them.
In the same way that early adopters of ERP systems in the 1990s saw their stock rise relative to their competitors as the technology enabled them to scale up their operations while also cutting costs as a percentage of revenue, in 2013 hope is being pinned on big data technologies to perform a similar feat.
Many large organisations already have some sort of big data roadmap in place, but they are now being joined by much less substantial players. Whereas five years ago the customers of NoSQL vendor MarkLogic were exclusively global firms like Dow Jones, CEO Gary Bloom told Computing that they now include publishers with a turnover of less than $25m – SMEs in other words – who are looking to solve exactly the same sort of big data problems.
But a dearth of big data expertise and the complexity and costliness of many integrated systems are often constraints to these smaller firms who wish to make a flying start. They simply can't afford to wait for years to extract value from the data that lies outside of their relational databases. For this reason we will see the continued rise of big data as a service - either cloud-based or through consultancy - in 2013.
Alongside Amazon's on-demand big data service and Google's BigQuery, both of which were launched last year, cloud-based start-ups such as Infochimps and Mortar Data have emerged, offering a short-cut to those without the means or inclination to process and analyse big data in-house.
Back on land, integrators who can knit together the complex on-site multi-vendor environment of a typical Hadoop-based big data system, and those analysts who can tell a compelling story from the results, will find their services in great demand too.
Computing verdict: That was easy! Most definitely yes
What new use cases will we see?
Big data is potentially useful wherever there are a large number of variables that must be analysed to spot small variations or loose inter-relationships.
For this reason it is typically used for fraud detection in data-intensive businesses like banking, government and retail. It is also useful where a multitude of variables can affect prices, such as with financial derivatives and airline tickets, or in complex modelling such as in analysing weather patterns and other scientific research. Expect to see more of the same in the coming year.
New uses are sure to be found in telecoms, where operators now hold enormous amounts of data about subscribers, including a massive increase in geographical and browsing data from smartphones. They will be looking to analyse their various datasets for relationships that can be monetised.
Another growth area is energy. Sensor data coupled with big data analytics can help firms and consumers to reduce their heating and electricity bills by optimising usage, and energy companies can improve efficiency by adjusting load-balancing and transmission through smart grids. US start-up Nest will be setting up shop in the UK this year, providing a big data-enabled smart meter designed by iPod guru Tony Fadell.
The release of government data through the Open Data Initiative will potentially provide opportunities for those who wish to bring new elements into their analysis.
There are new uses in healthcare, transport and insurance too numerous to cover here. Ultimately, though, the majority of new big data adoptions in 2013 will be those that make the most immediate business sense: the quick wins that are easy to sell to the board. These will generally revolve around creating more productive interactions with existing customers, such as providing helpdesks and sales people with real-time information while they are on the phone.
Computing verdict: We will see current practices filtering down to smaller organisations, while firms will start to make use of newly-available government datasets
Will an industry-standard NoSQL database emerge this year?
The fact that many big data systems, including Hadoop itself, are open source and free to download means that anyone can make a start for very little outlay. The past few years witnessed a burgeoning open source NoSQL scene, with some serious investment going into what is still a pretty new technology.
According to some estimates, more than 300 NoSQL start-ups have been formed since 2011; meanwhile analyst 451 Research has compiled a list of 160 databases, a list that lengthens every month. This hive of activity is sure to keep buzzing away throughout 2013.
2013 will certainly be the year that the gloves come off in NoSQL-world. Whereas once the main players were content to plough their own furrows across this fertile landscape, each with its own technical or vertical speciality, increasingly they will find themselves fighting over the same territory, especially around document-oriented databases.