How the Crown Estate built a system to store and retrieve hundreds of terabytes of marine data

The Marine Data Exchange was built after the Crown Estate was inundated with hundreds of terabytes of marine data from offshore wind-farm operators

The surface of the Earth - particularly around developed countries like the UK - is pretty well-mapped, and the tools for examining the results of that mapping equally well developed.

Mapping the ocean floor, however, is another matter. Indeed, it's somewhere that the Google Streetview car rarely ventures, and for organisations that need to operate offshore it can represent a major cost and a big barrier.

Furthermore, the sheer area that needs to be covered isn't small. "The Crown Estate owns the sea bed out for 12 nautical miles and it has rights to do many different things on the sea bed out to 200 nautical miles and beyond to the continental shelf limits," says Peter Edmonds, The Crown Estate's marine data manager and head of the Marine Data Exchange project . "It's about four or five times the UK land mass and one of our key remits is around renewable energy," he adds.

Indeed, as interest in setting up offshore wind farms - not to mention the potential for tidal energy generation in the future - grows, the need for better and more easily accessible mapping of the seabed around the UK has become increasingly important. However, mapping the seabed isn't quite as easy (or as cheap) as sending out a Google Streetview car.

The Crown Estate therefore hit upon a solution, says Edmonds. When prospective offshore wind farm operators do their preparatory surveys, the agreement obliges them to hand over their survey data so that a detailed map of the UK offshore seabed can be built up at their expense rather than taxpayers', and can be re-used at a later date.

The only trouble with this cost-saving idea was that very quickly the Crown Estate became snowed under several terabytes of marine data, which came in on various different physical media - with no real means to manage it and efficiently re-distribute it. "We had data coming out of our ears and we also had a delivery problem because, to realise its value, we needed to get the data back out into the public domain," says Edmonds.

In a bid to thrash out a system to overcome this problem, Edmonds led a series of internal brainstorming sessions to scope out requirements, with IT services provider Clear People being brought in to help with the implementation.

What The Crown Estate's IT team alighted on was a Microsoft SharePoint-based system that utilised the Azure cloud to aid distribution of environmental and geographic marine data beyond the organisation - a system it calls the Marine Data Exchange.

In the process, The Crown Estate also had to standardise its meta-data formats to the MEDIN standard as it migrated both legacy marine data and more recent geographic information submitted by wind farm operators.

Launched in February 2013, the system is architected, says Edmonds, to maintain a clear separation between the geographic data held on the intranet and the information requested via the public portal by users: when a request for data is made, that request is sent to the Maritime Data Exchange repository on the Crown Estate intranet, but it is uploaded onto Microsoft Azure where it is cached until the end-user downloads it.

The idea, says Edmonds, is not just to keep The Crown Estate's infrastructure secure, but also to keep its cloud-storage costs to a minimum and to help balance network demand.

The ease with which data can be submitted and acquired has also caused the amount of data held on the Exchange to explode: from just 1TB at launch to more than 100TB today - and growing.

Today, the Crown Estate's Maritime Data Exchange stores data estimated to be worth £100m, helping to cut the cost not just of offshore wind farm development, but any developments around Britain's coast in the future.