Beatport's data-center environment now contains more than 300 devices.

I wrote that number today in a note describing my work. The facility runs twenty-four hours a day, every day, because the store serves DJs, labels, and partners across time zones. There is no useful closing window where the whole platform can be taken apart.

The team is now seven systems administrators and network engineers. We are responsible for day-to-day operations as well as the architecture, implementation, and project work needed to keep expanding it.

The change from 2007 is easy to see. At that point I described nine web servers, five database servers, and a 30-terabyte storage system. That already felt substantial. The current environment is measured across hundreds of devices instead of a few server groups.

The work is not only adding equipment. It is making routine operations repeatable enough for seven people to share them. Identity, monitoring, configuration, documentation, capacity, and recovery all have to work across the environment. A manual step that is manageable on nine servers becomes a different problem when it has to happen everywhere.

The same is true for troubleshooting. One alert can begin in the application, database, storage, network, or operating system. The team has to know which layer changed and which systems depend on it before making the next change.

I am still close to the implementation, but the job now includes building the team and organizing the work around continued growth. More than 300 devices is the current count I gave. Seven people are responsible for keeping that environment operating while it continues to change.

Archive