There's much debate over what caused the big data boom and when, but for Darren Dance, Unix technical lead for online payment services provider WorldPay, there's one factor that has caused much of its rise.
"Mobile technology has caused a lot of it because it's starting to create data on a scale we've never seen before," he told Computing at Splunk Worldwide Users' Conference 2013 in Las Vegas.
"We're generating so much that if you believe some of the figures floating around there's going to be petabytes within the next five to 10 years, which is just an insane figure and there's value there."
As a result, Dance believes organisations are beginning to see the potential of properly analysing big data, which has led to a sharp rise in the number of vendors offering solutions. "There's so much value in it and people are starting to realise that so you've got various products that have come into the market," he said
WorldPay uses Splunk to analyse the more than 500GB of data it produces everyday. Dance said the company selected Splunk because it was "kind of there before the boom hit", with technology based on the likes of parallel processing rather than traditional tools, meaning "they broke the barriers where a lot of other tools fail".
Dance revealed that one of his colleagues once described one of Splunk's competitors as "it works and it scales until it doesn't", adding, "all manufacturers will tell you that they scale, but at some point they just die, and when they die they die horribly".
However, as a Splunk customer, that's apparently not a risk for WorldPay because, according to Dance, "the parallel processing capabilities with Splunk mean you can just throw another node at the problem and get it up and running quickly, avoiding a problem occurring".
Returning to mobile's role in the big data boom, WorldPay has several entries in the mobile payment sphere, ranging from face-to-face transactions, to online shopping and microtransactions in mobile games. All of that, Dance told Computing, is providing a new challenge that requires scalability and reliability.
"All of that creates really high impact events, for example when a game releases a content update you'll suddenly have a massive spike in transactions coming through and systems need to be able to cope with that," he said, adding the scope in transaction spikes means WorldPay has to be more dynamic with data volumes than ever before.
"No longer is it 'we processed 50 transactions last week, we'll process 51 this week'," Dance explained. "It's going from processing none one week to five million the next.
"It's massively changing and a lot more dynamic than it ever has been in payments and it means all of our infrastructure has to be built so it can cope with those peaks.
Of course, if the big data boom has only just begun, there's huge potential for organisations of all sizes in every sector to crunch their data, tapping into it for major benefits. For Dance, that means when it comes to big data "it's only going to get bigger", but added that it'll take time before large amounts of data can properly be interpreted, referring to works by statistician and author Nate Silver.
"I really like the fact [Silver] says that the big data revolution actually started with the invention of the printing press, but it takes time for us to turn information into knowledge," Dance explained.
"When the printing press first came along and bits could be replicated relatively cheaply, people had that information but they interpreted it in different ways - and as such there was 200 years of religious war," he continued, arguing it's going to take time for organisations "to catch up and make use of the amount of data that we're actually now generating".
Splunk, Dance added, will be one of the keys to WorldPay overcoming that particular hurdle. "Splunk is actually helping WordPay turn information we collect into knowledge we can use," he said.
The challenge doesn't stop there, however, as WorldPay needs to adapt to future technologies and the data that will come with them, something Dance admits the firm wasn't quite ready for this time around.
"We're building new platforms for the future and the next stage in the journey is to build stuff which is suitable for the future and can grow with the next generation of how people are making payments," he said.