More than one petabyte. Three sites.
That was the storage deployment described in an announcement draft on my desk.
A petabyte is one thousand terabytes. At Shutterstock, that capacity held photo and video files used by the systems that processed and delivered them.
The storage problem was not only about buying enough disks. We had to move large files through the system quickly, add capacity as the collection grew, and keep the product available when hardware failed. We also had to do it at a cost that made sense for the business.
We built the storage system around an open-source, scale-out filesystem and Coraid storage. Coraid used standard Ethernet rather than a separate Fibre Channel network.
Before choosing it, we compared iSCSI, Fibre Channel, Fibre Channel over Ethernet, and the Coraid approach. The differences were cost, performance, complexity, and fit with our open-source systems.
Shutterstock needed a lot of capacity, and the image and video collection kept growing. The draft framed the requirements as capacity, performance, cost, and interoperability with open-source systems.
The deployed system was spread across three sites. That scale and distribution were part of the announcement draft, not a future target.
The draft included a quote from me as Director of Web Operations and IT. It described the Coraid platform as the foundation for the distributed storage system and emphasized capacity, performance, cost, and interoperability with open-source systems.
The file is still a draft. It is prepared announcement language, not a public release yet. I need to keep that separate from the system already running.
Coraid said the system could deliver up to 1,800 megabytes per second and offered five to eight times the price-to-performance of competing storage. Those were vendor claims, not independent measurements from Shutterstock. The draft places them beside the capacity, performance, cost, and interoperability requirements in the prepared quote.
The architecture was intended to grow in smaller steps. Capacity could be added as the collection expanded instead of replacing one large array.
What we have deployed is more than one petabyte across three sites, using an open-source scale-out filesystem and Coraid storage over standard Ethernet. The performance and price comparisons belong to Coraid. My prepared quote belongs to the draft.
One petabyte sounds like a big milestone in 2011. It is also the amount of deployed storage described in the draft sitting in front of me.