BBC Worldwide rolled out commercial mobile viewing apps - with no user analytics

"We've been pretty much blind on mobile," admits Zoe Bolton, head of service operations at BBC Worldwide

BBC Worldwide rolled out the BBC Store, a commercial mobile app to enable viewers in the UK to buy and download programmes no longer available on its free iPlayer app, but didn't build in any user analytics to monitor the performance of the mobile app until months after it was launched.

On top of that, because so much IT capability has been outsourced at the BBC, it lacked the resources to manage any analytics app implemented in-house, when service operations staff were tasked with making good the omission.

"In December 2015, we launched the mobile apps... When we launched the mobile apps we didn't put any analytics in, so we've been pretty much blind since that time around our mobile app usage," said Zoe Bolton, head of service operations at BBC Worldwide. "Everyone was bothered about launching the product rather than taking on additional work," she added.

On top of that, users were receiving unhelpful error messages, while design shortcomings couldn't easily be highlighted. For example, anyone from outside the UK, who aren't supposed to be able to use the service, would only be blocked when they tried to pay with a credit or debit card registered to a non-UK address.

But without monitoring and analytics, none of this could easily be identified.

Bolton was therefore given the brief to "start monitoring". But the organisation didn't know what it needed to monitor. "I had to [keep] asking 'what am I monitoring?' It took a really long time," said Bolton.

Complicating that, many components in the BBC Worldwide downloads service are delivered by third parties. "I needed a solution that gave us end-to-end visibility of the customer journey," said Bolton, speaking at this week's .conf2016 Splunk user conference in Orlando, Florida.

She continued: "I decided I'm not going to monitor infrastructure because we are paying third parties to manage our infrastructure, and they are not going to be bothered about the 'journey' of the overall customer experience.

"And I needed to know more than just 'the website is available'. It's 'the website is available and customers can sign in' and, when they sign in, can they sign in quickly? When they buy can they buy quickly? And when they browse can they do that quickly? And [what about] when they play back? These are the core journeys."

Hence, Bolton decided that analytics that could provide insight into the kind of service that customers were receiving was essential, rather than one based on purely technical metrics, which often don't tell the whole story from the end user's perspective.

The organisation engaged and implemented Splunk in August 2015, putting staff through Splunk training in October, according to Bolton. Phase two of the implementation was conducted in April 2016 and Splunk MINT, which is purely for mobile apps, was implemented in July 2016.

Bolton also warned that outsourcing, especially in terms of managed services, could have been a barrier to the organisation getting hold of all the log files pertinent to the services BBC Worldwide needed to monitor, but that "we have still been able to do what we want to do" by other means.

Now, though, having implemented Splunk, other parts of the BBC are now looking at the software as a cost-effective means of providing monitoring for IT projects elsewhere in the organisation.

Bolton was one of a number of UK end-users appearing at the company's event this week. Other presenters included Ministry of Defence CIO Mike Stone, while Splunk used the opportunity to detail new versions of a number of core analytics and security products.

Corrections: The BBC has contacted Computing to say that it "had analytics in place for the website, which takes the vast majority of our traffic and all purchases", and that analytics were always in place for the BBC Store service delivered via the website.