The revolution in self-service analytics starts now

Computing research finds that in most companies top-down BI still prevails, but things are about to change in a big way

The way that information is propagated through businesses and government departments has been changing for years, leading to a gradual flattening of decision making hierarchies. Every now and then, though, something happens to accelerate the process, and we may be on the cusp of such a step change right now.

In the traditional business reporting process, IT interrogated enterprise databases and data warehouses and created reports when business users requested them. If the business wanted a new report it had to go back to IT. IT had its hands on all the levers of control.

Then, as the tools improved in terms of usability, perhaps adding a friendly web UI, business users were able to create their own reports, so long as they had a good understanding of the data, the tools and the querying process. However, such users were few and far between

Now, we are entering the age of self-service analytics, where business users help themselves to pooled data to get at the information they need, in the way they need it, and perhaps even in real time.

Analyst Clive Longbottom of Quocirca describes this as "proper business intelligence", as opposed to the business reporting and business analytics that went before.

"The new, proper BI tools allow different roles to utilise the tool in different ways - but all at a self-service level," he explained.

Self service means that non-technical employees are actively encouraged to query core data in order to make decisions, and offered training to be able to do so with minimal assistance from IT.

On the face of it then, responsibility for analytics is moving away from the IT department and into the hands of users in marketing, finance, product development, or anywhere that data can help with decision making. Business will drive analytics rather than IT, so the theory goes.

However that hasn't happened yet. Computing surveyed nearly 400 decision makers from UK organisations and found that there has been little movement over the last 12 months (figure 1). Results from 2013 were very similar too.

Because of the way that business reporting and/or analytics has always been done, and the complexity of the tools to hand, IT and data have long been virtually synonymous. And so it remains for now.

"Historically, this is where the data resides and the technical ability is within the IT department," a CIO at an online recruitment firm told us.

"In traditional organisations, IT owns the data, the smart people in IT understand the data and make suggestions to the business."

Self service may not have taken off yet, but the signs are that it is about to - and in a big way (figure 2 - above). If our respondents are correct, soon IT will no longer be the sole custodian of the core data, and neither will it have a monopoly on the smart people who understand it.

Currently in 70 per cent of organisations, analytics is restricted to the upper echelons of the decision-making hierarchy, but in three years' time that figure is predicted to fall to less than 30 percent. At the same time, more than half predict that analytics in their organisation will be democratised using self-service tools that are much more intuitive to use.

A CIO working on a major infrastructure project told us that for his firm the change would come sooner than three years, because they will be able to leapfrog the traditional BI tools - or "Oracle stuff", as he put it - and move straight into next generation analytics. Others will have the two systems running side by side for some time.

Shifting gear

The democratisation of analytics has the potential to unleash an explosion of creative thinking as boundaries are torn down and individuals gain a new picture of the organisation and their own role within it.

New ideas can be tested against data models as what-if scenarios. The decision-making process can be speeded up, successful ventures can be expanded more quickly and unsuccessful ones dropped before too much time and resource have been wasted. In short, a data-driven business can be more agile, rational and efficient, from top to bottom.

"We want to use more evidence-based decision making. Everyone should have an equal voice in putting forward something to test. It could be a user test or a piece of analysis on our customers. This has worked very well in our product development, but we are now trying to take that approach across our whole business," said our CIO in online recruitment.

In contrast to the static reporting solutions that have defined analytics thus far, the new breed of analytics tools offer the ability to view data-points over time and to drill down into points of interest. This means that very different job roles can interrogate the same pool of data in very different ways. The CEO might have a dashboard showing macro trends, the engineer can deploy one that alerts him to events as they happen and allows him to quickly find the root cause, and then there are frontline workers who just need to be able to change the parameters of a pre-constructed model, to see which product to put on which shelf for example.

To allow for such a wide range of use cases the tools need a efficient and flexible back-end engine, a slick user interface, and the ability to enforce access and security rules so that sensitive data can only be seen by those authorised to do so.

The market for such tools is growing fast, with newish companies such as Pentaho, Tableau, Qlik and Birst all doing well, along with Tibco's acquisition Sportife and any number of open source tools.

However, as impressive as these new discovery and visualistion tools may be, they are worthless if the central common pool of data - what has become known as the "data lake" - is polluted. A central piece in the democratisation process is making sure that everyone is on the same page, and that the data is reliable. This is where the hard grind of master data management, quality control, data cleansing, de-duplication and governance comes in.

But how accurate does the data need to be before it can be used for high level decision making?

While a few sticklers said they would not make any important decision unless the data was was 100 per cent accurate (good luck with that!), the consensus was that a level of accuracy between 80 and 90 per cent is good enough.

"It's about supporting gut feeling and experience, not replacing it," said a data infrastructure director in a large global bank. If data is 80 per cent accurate you leave room to make the actual decision."

Without sufficient quality there can be no trust and without trust, the whole business of democratising decision making will fall over.

"It's very difficult to re-build confidence. If data is found to be incorrect it is very damaging. It only has to happen once for a user to question every subsequent piece…" said a CIO in the services industry, continuing: "But it is difficult to create trust when data comes from so many disparate sources, even if it has been checked and re-checked."

Pulling it all together

As well as ensuring quality there is also the small matter of identifying relevant sources of data and allowing that data to flow into the data lake, which can be an enormous task in itself, especially in a big diverse organisation like a global bank.

"We operate in 55 countries, each with their own rules," said the CIO of such an organisation. "Our main challenge is with raw data because of the sheer number of systems. If I look at transaction data alone, there are about 50 different services."

In general, however, of the three steps in the process of creating value out of data, gathering the raw materials was thought to be the least problematic (figure 3).

The most challenging step was thought to be turning raw data into useful information, closely followed by providing actionable insights - turning information into knowledge. The former tends to be the job of the BI team whereas creating knowledge is done by analysts. Perhaps these teams do not work as closely as they should.

"The hand-over of the information and understanding from the BI teams to business analysts is not as seamless as it could be, although they work pretty much as one team," said one CIO.

Another common problem is where the process has not been driven by real business need, leading to swathes of information that is of no particular value.

"It's knowing what you want. You've got all this information and you don't know what to do with it," opined a solutions manager in an NGO

"One of the things we've found quite tricky is that information to knowledge bit where there are shifting sands around requirements. We find it quite difficult sometimes because the dashboards are so flexible it's hard to actually tie down how we want to use them," added a CIO in a local authority.

Turning data into information and information into knowledge that anyone can use to do their job more effectively or creatively is what democratising analytics is all about.

For IT this means a change in role. As the old top-down business intelligence model dwindles, a number of changes will work their way through the business. And since knowledge is often synonymous with power, changes to its distribution will inevitably become political. There will be losers as well as winners.

However, IT people are good at adapting and perhaps there is not too much to fear. The importance and the amount of data are growing exponentially and all businesses will need knowledgeable people to find and integrate the data sources and to select the tools with which to interrogate it. Because driving a business via dashboards is about much more than just a pretty interface.

@_JohnLeonard