In November 2013, I went to Hong Kong to talk about how we were using OpenStack at Shutterstock. More than 3,000 people from 50 countries attended the summit. A lot of the conversation around OpenStack was still about whether companies could run it in serious production environments.
We were already doing that.
Shutterstock had thousands of nodes, several data centers connected at more than 10 gigabits per second, and multiple petabytes of disk. The application teams were deploying more than one hundred times a week across dozens of REST services. We were using Ruby, Perl, Node, MariaDB, HDFS, and plenty of other systems.
The operations team could not support that pace by waiting for tickets and manually typing the same commands. We needed to build software for the rest of engineering to use.
One slide in the talk said, “everyone ships code.” Operations was included in that.
OpenStack handled our virtual machines. We also had a lot of dedicated hardware, especially for databases and other high-I/O systems. Those two kinds of capacity went through different systems underneath, but developers did not need two completely different ways to ask for them.
We put a lightweight provisioning layer in front of OpenStack and Foreman. A request for a virtual machine went to OpenStack. A request for dedicated hardware went through Foreman. The developer used one interface, and the provisioning system handled the difference.
That was useful for more than convenience. It kept a lot of data center knowledge out of individual requests. A developer did not need to know every hardware, network, and provisioning detail before getting capacity. The operations team could change the implementation without teaching the entire engineering group a new process.
Storage was more complicated. A disposable virtual machine did not have the same requirements as a large data set, search index, backup, or persistent snapshot. We used open-source filesystems, standard interfaces, and automation around the storage hardware. We were also working on defining more of the block-storage layer in software and sharing the tools where we could.
We automated network work too. Our code made changes across core networking, load balancers, and firewalls. We used routing for internal failover and connected our physical environment to a virtual private cloud in Amazon when we needed extra capacity.
The data centers were divided into zones. Each zone had enough network and compute resources to operate independently through the stack. Database failover still happened at the database tier. We were not pretending every system was stateless or that all hardware could be swapped without thought. The platform had to account for those differences.
The main point of the talk was not that we used a particular list of open-source products. It was how we put those products together and made the result available to the rest of the company.
A one-off script could save an operator time. It did not help much if only that operator knew how to run it or fix it. We wanted repeatable provisioning that developers could use themselves. We wanted physical and virtual capacity to follow the same basic request path. We wanted the network and availability model represented in code instead of a checklist somebody had to remember.
Open source worked well for us because we could inspect and change the software. We could connect OpenStack to the tools we already had and adapt the system to Shutterstock’s traffic, storage, and database needs. The tradeoff was that our team owned the full result. We chose the architecture, wrote the integration, and operated it.
By the end of the talk, the summary was pretty straightforward: OpenStack for virtual machines, one provisioning path across virtual and physical servers, network automation, and zones for availability and distribution.
It was a practical system built for the company we had at the time. It also gave our application teams a way to keep deploying without waiting for operations to handle every request by hand.