On June 11, 2007, I answered a Craigslist job post by sending a list.
Nine load-balanced Apache and PHP web servers. Five load-balanced MySQL database servers. A 30-terabyte SAN that could grow to 191 terabytes. High availability on the network appliances. About 80 terabytes of traffic each month. Uptime above 99.99 percent.
I asked them to call me. I believed I could provide the server-management and IT-infrastructure experience they needed, and the best proof I had was Beatport.
Beatport had launched in 2004 as an online download store for DJs. I joined in September 2006 as a senior systems administrator, when the company was still young and growing quickly. In January 2007 I was promoted to lead the technology function. That happened fast, which was exciting, but the title did not reduce the amount of work. It just made the whole system my problem.
An online record store has no closing time. A DJ in another country does not care that the team in Denver is asleep. Labels have music to deliver. Customers have files to download. Search, payment, processing, databases, storage, and network traffic all meet at the same storefront. When one part stops, it is not an abstract infrastructure issue. The store is broken.
The effect of a systems problem was usually clear, and the responsibility was substantial.
The architecture in my email was the result of decisions about where failure could happen and how much of it we could absorb. Multiple web servers meant traffic did not depend on one machine. Multiple database servers meant the data layer needed its own plan for load and failure. High availability on the network gear allowed one appliance to fail without taking the store down.
Storage was the number I kept coming back to. Thirty terabytes was substantial in 2007, but the more important phrase in the email was "scalable to 191 TB." We were not only operating the catalog we had. We were trying to stay ahead of the catalog, the file processing, and the traffic still coming.
The web tier used Apache and PHP. The database tier used MySQL. The SAN held the music and related data moving through the store. Load balancing distributed the requests, and the network appliances were configured for high availability. None of those parts was unusual by itself. The amount of traffic and the need to keep the complete service available made the combination demanding.
I knew the equipment well enough to list it from memory. I was also learning the rest of the job while doing it: budgets, hiring, priorities, architecture, the engineering process, and how to explain technical choices outside the server room.
I was very much still proving myself. The email started, formally, with my name. It attached a resume. It called Beatport a prime example of my work. I wanted another organization to see the size of the responsibility and conclude that I could handle theirs too.
At the same time, Beatport keeps getting larger. The catalog, traffic, and team are moving fast enough that I cannot solve only the problem in front of us. I have to know what sits on the other side of the load balancer, what can fail, and whether the next version will work before the current version runs out of room.
My role also included the team responsible for the environment. Capacity planning had to turn traffic and storage growth into purchase decisions before we hit a limit. Monitoring needed to cover the full path through the store. When we changed the platform, we had to understand what the change did to the databases, storage, network, and customer downloads.
The numbers were the main evidence in the application. The system was handling real traffic and staying available. I was proud of the work, but I was still willing to answer a Craigslist post only a few months after taking over the technology function.
The email was short. It described nine web servers, five database servers, thirty terabytes of storage, and eighty terabytes of monthly traffic. I attached my resume and waited to see whether the company wanted to talk. In the meantime, the work at Beatport continued. The immediate job was to keep the current platform running while planning enough capacity for the next stage of growth.