Big data: the view from the foothills of the IoT

We ain't seen nothing yet

Part one of a two-part look at the intersection between big data and the Internet of Things, based on a major Computing research study.

At Computing we're interested in the broad technological movements. What are the drivers? Where are the big players investing their money? How are the planets aligning, and what are the likely consequences for business and society?

For the last few years the big data revolution has been one of these mega-trends. Technological developments such as Hadoop, NoSQL and Spark have allowed the ingestion, processing and analyisis of huge swathes of data that were hitherto out of reach.

This has led to new possibilities for businesses to be more much data driven and predictive, driving significant changes in industries such as financial services, scientific research, media and retail.

Now big data itself is being driven to the next level. There is a huge increase in the volume, variety and velocity of data coming our way in the looming shape of the Internet of Things.

What we now refer to as the IoT goes far beyond sensors. It encompasses smart cities, robotics and interactive health monitoring. It pulls in scientific research, weather forecasting and climate models. And it includes driverless cars, automation and robots.

Download the Computing Big Data & IoT Review 2017

All of these applications are highly data-intensive, and that data needs to be verified, processed, transmitted and analysed faster than ever before.

From an analytics standpoint, therefore, the IoT is big data on steroids.

While those getting their hands dirty with the IoT are still in a minority, Computing's research into this area noted a marked change in activity since we asked about perceptions last year. Twice the number of respondents said the IoT "is happening or will happen, and we are preparing for it."

How detailed these preparations are and whether they are in response to a predicted need or simply the result of of hype and exposure is anyone's guess.

"It's in the media, it just brings an awareness, so you think ‘OK well there is something that's happening, all of these devices spewing data - what am I going to be able to do with Internet of Things?" said the head of technology in a non-governmental organisation.

Nonetheless, our research found a definite correlation between those organisations "doing" big data and those active in or preparing for the IoT. The two are intimately intertwined.

How to extract value from data hitherto thought worthless is the big data conundrum. Do you just collect as much as possible in the hope it will be useful or do you take a more directed "small data" approach?

"There's going to be shedloads of data but what do we use it for? God knows," said a service delivery manager in banking.

However others seemed to have a much clearer view as to how they would make use of the mountains of data heading their way from the sea of sensors.

"All of the stars are aligning - data, digital and devices, all of those things are coming together" said an interviewee in the professional services sector.

"We are now beginning to understand machine learning, we are now understanding automation. Previously all these things were disconnected in different ways, but we've now finally started connecting them together and we are now just waiting for the sensors. We are about to hit that crossroad and we will probably hit it in the next 3 years."

The RoI of IoT

In fact the three year horizon was pretty much the time that most of our survey respondents expected to see a return on any investment in IoT.

This, of course, could be due to the realities of making a business case. It is hard to get a budget for longer projects.

A lag of three or more years between investing in IoT and seeing a return looks reasonable when we see where most companies have reached in their journey. At the moment they are thinking about deployment of sensors, and how to store and process the data from them. Square one, in other words.

This order of priorities was the same whether they had gone ahead with an IoT deployment or not.

Big data: the view from the foothills of the IoT

We ain't seen nothing yet

Security and privacy

The lack of emphasis on security, both in this study and by the industry more generally, was a worry for many of the experts we shared the survey results with. We have already seen DDoS attacks made possible by poorly secured IoT devices, and there will no doubt be many more. Time and time again it has been shown that security is very hard to bolt on after the fact. It has to be integral to the technology.

With the IoT, human intervention in terms of patching and updates is limited by the sheer number of things and their miniscule scale. This means the security question really must be addressed before any large scale rollout occurs. The recent WannaCry ransomware "outbreak" shows us just how vulnerable national infrastructure can be.

Connected cars have already been hacked, and think of the havoc that could be created if a city's automatic traffic controls, power grids or other national infrastructure were to be compromised by an attacker analysing, restricting or corrupting the data flows.

"There are issues in terms of how to get the data and do it in a secure way. It's a massive challenge and we haven't yet figured it out," an analytical consultant in financial services told us.

The impact of the technology on personal privacy is another cause for concern. The IoT is all about making technology invisible. Rather than having to type instructions on a keyboard, the idea is that technology will respond to our needs naturally, without us really noticing. This sounds fantastic, but essentially it makes us all passive rather than active participants.

The data used to operate these smart devices is derived from our own behaviour, but we have little control as to what happens to it. And smart does not necessarily mean benevolent.

The tech-dystopian TV series Black Mirror was mentioned more than once during our interviews and focus groups, as was an earlier literary example.

"There was the case of Alexa and the law enforcement agency in the US who were turning round and trying to subpoena Amazon. It's a cliché to say it but, you know, Big Brother…" said an IT director in textile services.

So, there remain a fair number of blockers to the progress of the IoT, which let's not forget is still in the very early stages of adoption in most industries. Even for those who have dipped their toes in the water, very few have moved onto full-scale deployment.

Having said that, the IoT - although it wasn't called that then - has been deployed for decades in industries like oil and gas and also in industrial production lines. Then there's weather forecasting which has always relied on data feeds from large numbers of sensors and more recently use cases include renewable energy installations and agriculture.

But even where sensor networks are nothing new, the focus now is now all about cross-matching the data they produce.

"The thing about IoT is it's another way of looking at stuff, it's another way of measuring," a business consultant told us. "We've had sensors out in cities and what have you for donkeys' years. You've had sensors in your car for donkeys' years. It's now about bringing that data back and using it."

While its use cases are many and various, the key value proposition of the IoT lies in the ability to predict what's going to happen and to respond automatically and appropriately.

This ranges from a driverless car being able to sense when someone is about to step out into the road where the value is saving a life, to predictive maintenance which allows aircraft to spend more time in the air saving the airlines billions and improving safety too, to instore sensors that tell retailers what products to store depending on the weather forecast, the value being increased revenue and less waste.

All of these applications depend on data, and lots of it, so that the past can be compared with the present with conclusions drawn from subtle patterns that humans could never spot.

Not all IoT applications are analytics intensive. However, many of the large-scale and more complex applications certainly do fall into that category, and as even simple industrial control systems start taking data from more and more touchpoints so the processing and analysing of that data will become more and more important.

"In many applications, although not all, you have to go big with IoT, so the big data in the IoT worlds are inextricably linked, said the CIO in an environmental services agency.

We'll be looking at the big data response to the IoT in part 2.