By October 2011, Shutterstock had deployed more than one petabyte of storage across three sites.
A petabyte is one thousand terabytes. At Shutterstock, that capacity held millions of photo and video files and supported the systems used to process, search, license, and deliver them to customers.
The storage problem was not only about buying enough disks. We had to move large files through the system quickly, add capacity as the collection grew, and keep the product available when hardware failed. We also had to do it at a cost that made sense for the business.
We built the storage system around an open-source, scale-out filesystem and Coraid storage. Coraid used standard Ethernet rather than a separate Fibre Channel network. That fit well with the rest of the infrastructure we were already operating.
Before choosing it, we compared iSCSI, Fibre Channel, Fibre Channel over Ethernet, and the Coraid approach. Each option could provide shared storage. The differences were in cost, performance, complexity, and how well the equipment would work with our open-source systems.
Traditional storage vendors sold complete systems with controllers, software, support, and a large price tag. They could be a reasonable answer for some companies. Shutterstock needed a lot of capacity and expected the number of images and videos to keep increasing. The cost per usable terabyte mattered.
We also did not want one storage system to become an expensive box that the rest of the engineering team could not inspect or change. The open-source filesystem gave us more control over how data was distributed and how the system expanded. Standard Ethernet made the network easier for our team to understand and operate.
The final deployment was spread across three sites. That helped with capacity and availability, but it also created more work. Data had to be placed and moved carefully. The sites needed enough network bandwidth. Monitoring had to show whether the limit was disk, network, or a service using the storage.
The vendor announcement included a quote from me as Director of Web Operations and IT. I said the Coraid platform was the foundation for our distributed storage system and gave us the capacity, performance, cost, and open-source interoperability we needed.
That was accurate, although it was written for a press release. The day-to-day work was less polished. Drives failed. Traffic moved between sites. New images and videos arrived. Processing jobs read and wrote large amounts of data. The team had to add hardware without turning every expansion into a special project.
Coraid said the system could deliver up to 1,800 megabytes per second and offered five to eight times the price-to-performance of competing storage. Those were vendor claims, not independent measurements from Shutterstock. Our decision was based on our own requirements and comparisons, not only the numbers in a press release.
The useful part of the architecture was that it could grow in smaller steps. We could add capacity as the collection expanded instead of replacing one large storage array. We could use the same networking skills we already had. We could connect the storage to open-source software without waiting for a vendor to support every change.
There were tradeoffs. When a company assembles storage from open-source software, standard networking, and hardware from different sources, its own team owns more of the integration. There is no single vendor responsible for the complete system. We had to understand the filesystem, network, disks, and application workloads well enough to operate them together.
That was work we were prepared to do. Shutterstock's image marketplace already depended on large-scale storage, and the volume was going in one direction. More contributors were uploading content, more customers were searching it, and video files were becoming a larger part of the collection.
One petabyte sounded like a big milestone in 2011. For the operations team, it was also the current amount of storage we had to keep working while we planned the next increase.