Everyone knows that we live in a data economy and organisations are gathering more information than ever before about customers, buying habits, team performance, internal operations, finance, and the supply chain. And as the Internet of Things spreads, the number of channels and devices that draw data into the enterprise can only explode - an information big bang creating a new data universe.
It should come as no surprise, therefore, that this shift towards focusing strategic business decisions on data analysis is changing the nature of the jobs market. In 2018, the World Economic Forum predicted that 75 million traditional jobs would be lost worldwide to automation, AI, robotics, and other Industry 4.0 technologies - notably in once-safe white-collar careers, such as legal services, financial advice, accounting, and auditing.
However, the WEF also predicted that 133 million new jobs would be created, many of them in functions such as data analysis and data science, a net gain of 58 million human jobs, including in those industries that are in the vanguard of technology-driven change.
Challenges in the new data economy
But the transition to this new data economy is not running smoothly in many organisations, according to a recent Computing Research survey of over 150 IT and data leaders in professional markets, such as banking, finance, manufacturing, and government - which are among those most affected.
While many of these organisations are engaged in digital transformation and enlisting the services of data scientists and analysts, these expensive experts' time is not being used effectively and their skills are often being wasted as a result.
Asked how much time their data teams spend on mundane tasks such as data collection and preparation, over one-third (34 percent) said between 21 and 40 percent, and the same number answered between 41 and 60 percent.
A further eight percent answered between 61 and 80 percent, while two percent said from 81 to 100 percent. This means that a combined total of 78 percent of skilled data analysts are spending between 20 and 100 percent of their time on routine, low-value-adding tasks, a clear waste of a critical resource. Just 22 percent of organisations are making effective use of their skilled analysts, therefore.
And that's not all. Only one-quarter of respondents to the Computing survey are based in enterprises where non-analyst users are empowered to access, visualise, and analyse all the relevant data. In the remaining organisations, a separation of data, applications and analytics - together with the inadequacy of their on-premise visualisation tools - means that while data scientists are rightly acting as gatekeepers of the analytics function, they are also creating a bottleneck that restricts business insight.
At this early stage of the shift towards a more analytics-based economy, the difficulty of recruiting enough data science experts has been documented in several recent reports, despite data volumes increasing tenfold over the past three years.
According to recruitment firm Robert Walters, for example, the number of data roles has increased by 80 percent since 2015, with data scientist vacancies growing by 110 percent year on year and data engineers by 86 percent. At present, many of those roles are not being filled.
The Computing survey results reveal that not only should more data scientists be engaged in finding the answers to complex challenges, they should also be defining the challenges in the first place - not spending their time on routine admin work.
So what is the solution to this thorny Industry 4.0 problem? The use of more powerful self-service analytics tools in the cloud should remove the bottleneck, freeing up data scientists to do what they are employed to do, while empowering other users to find their own insights faster.
The ‘siloing' of internal data in disparate on-premise systems, combined with the separation of data from expert data analysis teams, via technical bottlenecks and clunky platforms, places a limit on real insight. In turn, this limits the realisation of strategic data objectives, along with digital transformation and agility in what everyone now recognises are hyper-competitive markets.
Integrating internal functions such as HR, Finance, and Accounting via a cloud-based platform that is accessible anywhere and from any device, should provide the data visibility that decision-makers need, as well as allowing businesses to gain benefits from the compelling economics of the public cloud.
After all, collecting reams of data is pointless if businesses are unable to gain critical insights and value from that resource. And if decisions are to be made at the speed demanded by today's business world, in-depth analysis should be taking place where the data actually is, and not subjected to long delays as data science teams prepare and present the raw material.