The Met Office: Data science in an organisation full of scientists

Data scientists better for finding answers to business questions, rather than scientific ones, suggests Met Office CIO Charles Ewen

Met Office CIO Charles Ewen has questioned whether an organisation such as his needs data scientists - when it has hundreds of its own, real scientists to draw on.

The Met Office, says Ewen, generates multi-petabytes of data every month, which is analysed by its climate scientists in order to forecast the weather and wider weather patterns. Its analysis goes as far out as deep space, where it is also bidding to determine the influence of "space weather" on the Earth's own atmosphere and weather patterns.

However, for an organisation as replete with scientists as the Met Office, trained in physics and statistical analysis, it raises the question of whether an organisation that does so much proper science actually needs data scientists?

For Ewen, the answer is "yes and no".

He says: "You can think about data at the Met Office in three 'domains'. The first thing is data analysis to describe what has happened and what is happening. The Met Office's 'observations programme', for example, is all about establishing a picture of the atmosphere. That's one domain.

"The second is, why did it happen? And that's a different question to 'what's happened?' Then, based on understanding what happened and why it happened, you stand a fighting chance of predicting what will happen. So that's three clear domains: what happened? Why did it happen? And, what will happen?

"If you break those domains down, and you ask the question: what would a data scientist add? Well, in the area of what's happened, potentially quite a lot because there's quite a lot of value to be added in the 'what happened area?' And, because it's largely the realm of statistics, big data and all the kinds of things that data scientists are about, that's good," Ewen told Computing.

If the question is 'why did it happen?' continues Ewen, that's largely driven by the laws of physics and highly specialist knowledge, "so rather than have a data scientist answer that question, you're much more likely to get a better answer if you have a physicist answering that question," he says.

"In the domain of 'what will happen?' purely on weather forecasts, once again, they'd probably not be much help. But if you think more broadly that's not the question that people are often asking. People are asking questions like 'will my high street likely be busier tomorrow than it was today?'" says Ewen.

That might involve basic climate data, but mixed with business data that the average data scientist ought to be most comfortable working with. "If you're looking for an answer to that kind of question, a data scientist potentially has tremendous use, because he or she can use their statistical, correlative, non-diagnostic techniques to make predictions - but in the day-to-day business of weather forecasting and climate projections? Not so much."

Charles Ewen is the CIO at the Met Office, the UK's national weather service, based in Exeter. He was a keynote speaker at Computing's recent IT Leaders' Summit in London, which brought together CIOs, IT directors and other IT leaders from across the UK. Stay tuned for our forthcoming CIO interview with Charles Ewen in which we will explore the IT behind the Met Office, big data, and the organisation's new super computer