Spotify gears up for growth

Spotify is growing fast

Online music site Spotify last month installed new servers from supplier SGI at its co-location facility in Stockholm to cope with growing demand that has seen its users top five million.

The music-streaming service was launched in October 2008 and offers subscribers free, advert-supported access to a vast database of songs via the internet. The platform is being watched closely by other businesses looking to capitalise on web-streaming services.

The huge and growing demand for the service has vindicated the company’s original decision to design a storage infrastructure that could be expanded rapidly to handle extra traffic.

Spotify operations director Emil Fredriksson said of the storage system: “We copied the design from Google pioneers – and designed the system to scale servers to grow with the business, so we were always able to add hardware to cope with more users.”

Each of the SGI 3000 servers installed in Spotify’s Stockholm datacentre hosts up to 12 hard disk drives, meaning there are up to 24 drives per three units of height in SGI’s Foundation Series server racks.

Each server features up to two dual/quad core Intel Xeon 5100/5200/5300/5400 or AMD Opteron 2200/2300 series CPUs, up to 128GB of RAM, and either 400GB of SAS or 1TB of Sata II hard disk capacity.

“The total amount of storage we have is more than one petabyte, but in terms of active songs and those we have recently streamed, it is probably more than 100 terabytes,” said Fredriksson.

Whereas the free, ad-supported version of the Spotify site offers streamed music files sampled at approximately 160Kbit/s in the Vorbis format, the paid-for premium version ups the audio quality to 320Kbit/s. As such, the user experience is closely tied to the performance of the underlying storage architecture, said Fredriksson.

“In addition, Fibre Channel SANs are becoming a thing of the past, so we rely more on Ethernet and servers with direct attached storage running our customise d software,” he said. “Fibre Channel is also very expensive and our solution is more cost effective, power efficient and based on commodity storage, RAM and processors.”

Spotify’s indexing and search software was custom built in house, though it relies to a certain extent on existing software platforms beneath it. The Spotify software client needs to be downloaded before users can begin using the service. It examines the cache of the client PCs accessing Spotify, which then informs it of other users (or peers) closer to them from whom it can retrieve the same audio stream to increase performance further.

“We are currently running out of two datacentres and our goal is to always have as high an availability as possible so we replicate data between them,” said Fredriksson. “We also need to get as close to the user as possible and try to add more capacity that way.”