Case study: Morrison's data warehouse

Supermarket chain bemoans lack of technical skills in the north, but benefits from IT layoffs at HBOS

Supermarket chain Morrisons is an Oracle shop, using a variety of the firm's products including Enterprise Linux, BIEE (Business Intelligence, Enterprise Edition) for reporting, Oracle Data Integrator (ODI) and real-time data integration tool GoldenGate. Speaking exclusively to Computing, Jonathan Walsh, MIS manager at Morrisons, described the five-year replatforming exercise that his firm is currently just over half-way through.

"We're in the process of a big replatforming exercise from a mainframe to an Oracle footprint. The core transactions are now going into the new data warehouse, and as the system continues to be rolled out, new transactions will go straight into it."

Walsh said that the decision to replatform was taken four years ago, as the previous mainframe environment was not fit for purpose for the size of company; it failed to scale with the business.

Although Oracle has recently announced the new version of its database appliance, Exadata X3, Morrisons is currently in the process of implementing the older iteration, X2.

"We've just gone live with X2, it gives us fantastic levels of data compression. We're seeing 20 times compression, so we go from 20TB to one. X3 however can put your entire warehouses in-memory," he added, although admitted that there are no plans as yet to upgrade to the newer model.

The reason for choosing Exadata, according to Walsh, was performance.

"Some queries used to take 10 minutes with the previous platform we were on, and that's now down to a few seconds."

But how does this vastly increased performance help the business? Walsh said that it creates a different way of working with the data.

"People used to be inclined to take the data out of the database and put it into a big spreadsheet and work with it there, rather than using the power of the platform [because queries took so long]. Now they use the system for the purpose it was designed for."

While this implementation has been a success, there is still a lot of data migration activity to complete.

"We're on a journey," said Walsh. "We've got a lot more data to put in, including years of basket-level transaction data. There's a lot of value in that data, we've just needed a platform to allow us to leverage it."

[Turn to next page]

Case study: Morrison's data warehouse

Supermarket chain bemoans lack of technical skills in the north, but benefits from IT layoffs at HBOS

It seems that the data itself will be kept on-premise for the foreseeable future. Walsh is not himself an advocate of the cloud, saying that he has kept his own services out of the public cloud, due to security fears. However on the software side, the firm is a relatively new Salesforce.com customer for its CRM needs.

One problem with the new platform Walsh has found is in finding people with the right skills to help set up and manage the appliances and the data warehouse that resides on it. He describes the data warehouse itself as a "house".

"It's a very complicated architecture, you can't just build another room in it without understanding how it all fits together. We get third parties to do a lot of the bricklaying, with a mix of onshore and offshore support. But we need the control in the middle to make sure the architecture fits together properly. That has created a bottleneck in terms of our ability to recruit and retain the necessary skills."

He adds that he needs to build up the level of knowledge within his team, knowledge specific to Morrisons' use of the Oracle stack.

"It takes a year for new starters to become effective. ODI is a relatively new technology, so it's hard to find those skills. We quite often get people with strong ETL [extract, transform and load] skills, but they don't know the specific technology, they might have to move from a non-retail environment into retail, so there's still lots of learning to be done."

And even once the staff has been skilled up in the right areas, there is a further problem in retaining them.

"In two or three years of gaining this experience, their market value has shot up so it's a struggle to retain them," Walsh admits. "We've struggled a lot to find people with strong data warehousing skills particularly focused on Oracle. There's not a massive pool of them available, and the fact that we're in the North of England doesn't help [where fewer people with the right skills are available].

"We've actually benefitted from the decimation of the financial services industry in places like Yorkshire – from firms like HBOS who have laid off IT staff."

Although the replatforming project is technically close to the half-way point, Walsh doesn't yet see an end in sight.

"The original plan said that the programme would take five years, and we're about three years in. But we're now starting to consolidate other projects in - like CRM and customer analytics - so it effectively becomes a rolling programme. There isn't a specific date which we're working to."