Met Office deploys NoSQL intelligent-learning security system to monitor network traffic

Self-learning systems and a dedicated team of engineers help keep the Met Office secure

The Met Office is moving to a big-data-driven security model that will see it handle security threats by using a NoSQL-based system that automatically monitors network traffic for abnormal activity.

It follows the creation of a dedicated Cyber Security Operations Centre at the Met Office, and the development of an information security strategy based on "trust zones". These identify core information that require toughened defences, but which also accepts that security based on rigid outer defences will almost certainly be breached.

"At some level, ranging all the way from insider threats to advanced persistent threats (APTs) and zero-day malware, we are going to be compromised. So it's a question of doing your best to make sure that doesn't happen, combined with monitoring and reaction," says Met Office CIO Charles Ewen.

"Those are key words. We certainly plan to reduce our systematic security barriers in favour of technological monitoring and reaction. We have quite a funky project going on in-house using Elasticsearch, one of the new age NoSQL databases. This collates all of the logs from all of our systems.

"It's a self-learning system. It will build a picture of what's normal and, therefore, as it learns it can start to tell us when something has happened that isn't normal. When it's not normal, that's where the focus of our diagnostic teams will come into play," says Ewen.

In the early days, he adds, the system threw up "an awful lot of false alarms", but as it has bedded in, those false positives have declined in number.

"As the 'machine' has become more capable and learnt more about what normal looks like, it has become an increasingly effective approach to enable our teams to know something odd has happened, where it has happened, and to start to use their skill to then investigate," says Ewen.

The creation of a dedicated security team reflects Ewen's appreciation of the unique skills and attitudes security requires.

"We've put real engineers in that team," he says.

"That gets back to my belief in real engineers. You do need class-qualified consultants that are largely analysts that know the policies and how to implement them. You definitely need all of that, but you also need skilled engineers who can roll their sleeves up and who can do both the diagnostics, and the follow-up forensics and preventative measures to make sure that something that's happened doesn't happen again.

"We used to rely on our operational staff to do that role. And it's just not quick enough. You need people with a blend of skills that are a bit different from your operational technologists to be the people who diagnose," Ewen told Computing.

The Cyber Security Operations Centre is also responsible for managing the Met Office's more traditional suite of security tools and software, including its intrusion detection system and processes.

The Met Office's deputy director of applied science and scientific consultancies Doug Johnson will be speaking at Computing's forthcoming Big Data & Analytics Summit 2016 in London in March. Find out more and reserve your place now