How Nielsen is evolving into an AI-first business

We have had to embrace three things: cloud, open source and mobile and we use AI where it makes sense, says chief research officer Mainak Mazumdar

Nielsen, the market research and customer information measurement company, is on its way to becoming an AI-first business. So says chief research officer Mainak Mazumdar.

But what does that mean in reality? After all, AI has become an umbrella term covering everything from chatbots to deep learning and neural networks.

In Nielsen's case it's very simple, he told Computing during the recent Spark + AI Summit Europe. It means using automation wherever it makes sense in the company's day-to-day work, such as sending clients regular reports, setting up accounts, onboarding new users with usernames and passwords, following up requests, making phone calls and other routine and repetitive tasks. Robotic process automation (RPA) is now active in many parts of the business, Mazumdar said.

But it goes further than that simply automating back-office functions. Nielsen is a major processor of data. With operations in more than 100 countries, the company pulls in data on market share, pricing, TV viewing, ad performance, customer loyalty, branding and promotions from a large number of touchpoints. The company employs more than 1,000 data scientists to analyse this data and provide meaningful insights to Nielsen's customers.

"We have had to embrace three things: cloud, open source and mobile," Mazumdar said. "Cloud, because of the computation power - there's a lot of data and we didn't have computation power on our laptops or our on-prem servers so we had to move to the cloud."

The embrace of open source is because almost all the groundbreaking advances in analytics at scale is coming from such collaborative projects, including Apache Spark.

"We use the Databricks service and this has allowed us to operate with Spark in the cloud. There's a lot of innovation in open source that we cannot replicate internally. We have our own proprietary stuff, but we don't want to be the best Spark ninjas or TensorFlow on steroids, right? We want to use these tools so that we can move fast so we have a lot of partnerships."

The third pillar, mobile, is important to Nielsen as mobile devices are an increasingly important means for collecting data including speech, location, images and other types of information.

Nielsen's data science ‘crown jewels' is a core reference set of labelled data drawn from all over the world. This is validated data of known provenance with plenty of metadata attached which can be used to train the machine learning models the data scientists use to make their predictions, ironing out biases and filling gaps. Mobile data forms an important component of Nielsen's labelled data collection because metadata such as time and location are automatically included, for example, in a photo of a supermarket's shelves.

"We use this data to train other types of datasets, from set-top box data in media to point-of-sale in retail to image databases. Labelled data is very powerful for us," Mazumdar explained.

An AI-first architecture

At the back end being AI-first means being able to distribute and democratise this data and prevent it from getting stuck in silos. Currently, Nielsen ingests data from billions of global transactions every day, a number that's set to increase dramatically as more IoT devices come online. All the data sits in Hadoop data lakes in the cloud (AWS and Azure), and is accessed by data scientists, data engineers and application developers using mainly open source tools.

Proprietary approaches are perishable, they have a use-by date

Image credit: Nielsen

"Proprietary approaches are perishable, they have a use-by date", said Mazumdar, returning to the theme of open source. "A combination of cloud and Spark addresses the issue of manual partitioning, and Python and its libraries and notebooks means data scientists can seamlessly work in this environment."

This architecture has also served to increase the level of collaboration between data engineers, data scientists and the app developers who build dashboards and other products for customers, he said. "The performance, scale and speed have all improved."

What can artificial intelligence and machine learning do for you and your organisation? If you don't know yet, or want to make sure that you're not missing out, Computing's first AI & Machine Learning Live event is for you. To find out more, check out the Computing AI & Machine Learning Live website. Attendance is FREE to qualifying IT leaders and senior IT pros, but places are going fast

[Turn to next page >>]

How Nielsen is evolving into an AI-first business

We have had to embrace three things: cloud, open source and mobile and we use AI where it makes sense, says chief research officer Mainak Mazumdar

But for any AI endeavour to be succesful must be getting the data and data flows must be right.

"If you start digitising your processes then you have the data," Mazumdar said. "The first step is digitisation, and I think people kind of gloss over that part. Once you digitise, it doesn't matter if there's a person sitting behind that or not, you can build some decision making around it."

That doesn't mean automating everything, he added: "You can use RPA with people still sitting behind it but using it as a tool for better decision making."

For more complex tasks, though, operationalising AI requires a deal of prior study about where it will have the greatest impact. To be amenable to automation, processes must be well understood and have plenty of data and metadata attached to them.

Not all such processes require massive learning datasets, however, and operations conducive to the operational application of machine learning vary from customer to customer, and country to country. Nielsen is very much led by the wishes of its clients and partners, from whom much of the data that feeds the machine learning models is derived.

We are a near hundred-year-old company and we take a long view

The company's AI-first strategy began two years ago "not because of a fad or because it's a new thing to do. We are a near hundred-year-old company and we take a long view," said Mazumdar.

"It's still very early days and we don't have all the answers, but we have the intent. We have the intent and clients are asking us to get there, and we're moving along with the clients. Our clients are Fortune 500 companies. They're going through the same process. We are learning from them, they're learning from us, it's a community. It's a team sport. You can't do it all by yourself; everything is a team sport."