Follow Us

We use cookies to provide you with a better experience. If you continue to use this site, we'll assume you're happy with this. Alternatively, click here to find out how to manage these cookies

hide cookie message

Facebook prepares 'cold storage' to deal with data volumes

Facebook rethinks the data centre to reduce costs and still be able to store billions of photos effectively

Article comments

Facebook is rethinking the way it stores data to cope with the seven petabytes of new photos the social network's users upload every month. As the number of photos grows, Facebook needs to find cheaper, less power-hungry ways to store them all, according to the company's vice-president of infrastructure engineering.

Users upload about 300 million photos a day, more on special occasions, Facebook's Jay Parikh told the Structure Europe conference in Amsterdam yesterday. "Halloween is one of our biggest photo upload days of the year. We will get somewhere between probably one and two billion photos uploaded just in a single day," he said.

Photos like the ones taken at Halloween soon lose their interest, with no one looking at them after a few days or weeks, but "our contract with our users is that we can't delete the data when it is not accessed, we have to keep it," he said. That led to the idea of putting the photos into a sort of "cold storage," Parikh said. To do that, Facebook plans to build a new data centre with different types of storage, server hardware and network equipment that consumes less power and costs less than existing data centres - all without changing servers response times, said Parikh.

But how efficient can Facebook make its cold storage? When costs and power consumption in data centres are lowered, this usually happens at the expense of access speeds.

Storing data on tapes, for instance, lowers power consumption but severely slows down data access.

Amazon Web Services is following a middle path with its Glacier cloud storage service, which it pitches as an alternative to tape. The service is optimised for data that is infrequently accessed and for which retrieval times of several hours are acceptable.

That's much too slow for Facebook, according to Parikh. "I can't have a photo that you go access from five or ten years ago, and for me to show up a banner to the user that says: 'Hey, why don't you try again in 24 hours?' It's still got to be relatively real time," he said.

Most data centres that are used today are optimised to use a lot of power to deal with tasks that need big computing power. The "cold storage" technology Facebook is thinking of is at the other extreme, said Parikh. "You need lots and lots of space but you don't need as much power," he said, adding that everything about the data center needs to be rethought to handle the problem at the scale Facebook faces.

At a high level, Facebook is working on software that will figure out how and where to store a piece of content in the infrastructure when it ages, said Parikh. "That will mean that the copies of the data will move around over time and utilise the different pieces of infrastructure that we will have optimised for the age of content." Some of the inventions in the software layer will allow Facebook to still respond quickly but to store data more cost effectively, he said.

Cold storage will be part of Facebook's infrastructure in the next year or two, he said. Facebook plans to disclose and share the parts that it thinks are relevant through the Open Compute Project, an initiative started by Facebook to apply the open source software collaboration model to the world of data centre hardware.



Share:

More from Techworld

More relevant IT news

Comments



Send to a friend

Email this article to a friend or colleague:

PLEASE NOTE: Your name is used only to let the recipient know who sent the story, and in case of transmission error. Both your name and the recipient's name and address will not be used for any other purpose.

Techworld White Papers

Choose – and Choose Wisely – the Right MSP for Your SMB

End users need a technology partner that provides transparency, enables productivity, delivers...

Download Whitepaper

10 Effective Habits of Indispensable IT Departments

It’s no secret that responsibilities are growing while budgets continue to shrink. Download this...

Download Whitepaper

Gartner Magic Quadrant for Enterprise Information Archiving

Enterprise information archiving is contributing to organisational needs for e-discovery and...

Download Whitepaper

Advancing the state of virtualised backups

Dell Software’s vRanger is a veteran of the virtualisation specific backup market. It was the...

Download Whitepaper

Techworld UK - Technology - Business

Innovation, productivity, agility and profit

Watch this on demand webinar which explores IT innovation, managed print services and business agility.

Techworld Mobile Site

Access Techworld's content on the move

Get the latest news, product reviews and downloads on your mobile device with Techworld's mobile site.

Find out more...

From Wow to How : Making mobile and cloud work for you

On demand Biztech Briefing - Learn how to effectively deliver mobile work styles and cloud services together.

Watch now...

Site Map

* *