IBM Pulse: Enterprises need to differentiate big and little data

Experts debate real-time flows versus big-picture information

At IBM Pulse in Las Vegas this week, IT and business leaders urged enterprises to recognise the difference between big and little data.

"Big data is lots of data collected over extended periods of time, and the inverse of this is data in motion," said Arun Hampapur, IBM distinguished engineer.

In IBM's parlance, big data furnishes a view of the history and patterns of the data, whereas, "data in motion is little data that reflects something happening now and you get information instantly about it," explained Hampapur.

"What little data doesn't tell you is what happened the week before, or the week before that," he added.

George Hawkins, general manager at US utility company DC Water, urged organisations not to forget big data in the daily deluge of little data.

"Companies have little data that comes in with such quantity that it demands responses every minute," said Hawkins.

"If you aren't careful you will end up just responding to the needs of the little data, because this is likely to reflect something impacting a customer at that moment in time," he added.

"However, it is important to step back and analyse the data over time, because this will build efficiencies and effectiveness into your system."

Companies also need to be aware that little data that highlights problems in real time can disrupt current work flows.

"This completely messes up the service management, and I don't know how much enterprises have realised the impact of this yet," said Julio Olimpio, strategic alliance manager at geographic software company ESRI.

"Companies create daily tasks in their service management systems based on old data, and this is being interrupted by new data coming in," he added.

"How do you manage the old static data as well as continually manage the real-time data that points to errors that need correcting? You can't just put it at the back of the queue, you have to put this into the queue."