Overload averted: Essential guide to storage solutions

Research company IDC last year forecast that 2010 would see the volume of digital data stored by everyone on the planet reach 1.2 zettabytes (1.2 billion terabytes), representing growth of 62 per cent over 2009. And by 2020, that volume will have grown by a factor of 44 to 35 trillion gigabytes.

Some of that data will be stored on personal PC or laptop hard drives, smartphone Flash memory, and portable or removable data storage devices. But a far larger volume will come under the enterprise storage remit, meaning IT chiefs will be responsible for finding the space to host it on company servers or storage arrays, and the additional resources required to manage and secure it according to rules and regulations around data protection and retention.

Research from Gartner estimates data capacity in those enterprises is growing by 40-60 per cent per year on average, not enough to cope with demand if IDC’s statistics prove accurate. Its survey of 1,004 companies in eight countries conducted in August 2010 identified data growth as the top data-centre challenge, followed by system performance and scalability, and network congestion and connectivity architecture.

It is the explosion in the volume of unstructured data that is proving especially difficult to handle. Strictly speaking, unstructured data is any electronic file that does not conform to a pre-defined data model or fit into relational database tables, but the term pretty much extends to anything and everything that is not a database file: email, text and word processing documents, web pages, videos, images, audio files, health records, log files, instant messages, social media data and so on. And the increasing tendency among some organisations to create and retain multiple copies of that data for backup, disaster recovery, business analytics and e-discovery purposes tends to swell its mass even more.

So how can storage managers predict in advance when they will run out of space? One way is to use storage resource management (SRM) tools to track how full disks currently are and the rate at which information is being added, but these only make sense for larger organisations and can be expensive.

An alternative, more manual approach is simply to look at old backups and use their average rate of growth over time to calculate future rates of growth. From a cost perspective, it is important to fit storage upgrade investment as closely as possible to capacity requirements, and it is usually better to add a small amount of extra disk at regular intervals than perform large-scale upgrades every couple of years or so, say experts.

Whatever approach individual IT de-partments choose to take, there is no doubt that monitoring and managing storage capacity upgrades use up time and resources that could be put to better use elsewhere within the organisation. This is one reason why a growing number of companies are choosing to outsource responsibility for data storage, or specific storage-orientated applications, to third-party providers, with online backup becoming particularly popular and other storage-as-a-service business models beginning to emerge.

Of those polled in a survey of 2,600 IT staff and end users from 13 different countries, conducted by research firm InsightExpress on behalf of networking giant Cisco last year, one in three IT professionals said more than half of their company’s data and applications will be hosted in private clouds by 2013, with others moving at least some of their data to public clouds within the same timeframe.

To date, it is predominantly small businesses that use public cloud-based storage services, such as Amazon Web Services Elastic Compute Cloud, the recent outage of which highlights the risks in trusting mission-critical data to a third party without the reassurance of service level agreements (SLAs) or keeping local copies of the same information.

Many companies also worry about maintaining the control and integrity of data hosted outside their own firewalls, and remain committed to keeping the data itself, if not the applications accessing it, either in-house or within private clouds that have security, reliability and SLAs built into their architecture.

Of those questioned in the Gartner survey, for example, 30 per cent said they planned to build new datacentres of their own by the end of 2011, with more than half saying they would expand their existing datacentres, simultaneously consoli-dating or upgrading existing storage architecture, adding storage management tools, and building extra security measures into in-house systems.