Beatport's data-center facility now contains more than 300 devices.
The physical center of it is a 750-square-foot colocation cage that we designed and deployed around one requirement: no single layer should be able to take the store down by itself.
That starts below the servers. Redundant power. Redundant network core. Redundant firewalls and BGP routers. Redundant load balancing. Network-interface bonding on the Linux nodes. Multiple paths through switching and routing. Storage arrays that can move load between systems.
The parts have to be configured so one failure does not stop the rest of the environment.
We use Juniper firewalls and BGP routers, Foundry core routers and load balancers, and commodity Dell server hardware. The exact brands matter when I am configuring or repairing them. The design matters when a device stops answering and customer traffic still needs somewhere to go.
Beatport is a record store without a closing time. DJs buy music from different countries and time zones. Labels deliver content. The application has to search, sell, process, and deliver files while the technology team changes the machinery underneath it.
In 2007 I could describe the store with nine web servers, five database servers, and a 30-terabyte SAN. That list was already substantial to me then. Now I manage a team of seven systems administrators and network engineers responsible for day-to-day operations across hundreds of devices.
With nine web servers, I could still carry most of the topology in my head. With more than 300 devices, undocumented state becomes a system of its own. A switch port, firewall rule, DNS record, storage mount, monitoring exception, or one-off server build can work perfectly and still create a future outage because only one person knows why it exists.
We are trying to build the alternative into the platform. Centralized identity and certificate systems. Monitoring and trending. Configuration management. Scripts for the manual tasks that should not require somebody to remember fifteen steps. A routed network using OSPF, BGP, and VRRP. Available Layer 2 access using spanning-tree controls. Internal and external DNS that do not depend on one machine.
The tool list is specific: FreeIPA and DogTag for identity and certificates, Spacewalk for system management, Zenoss for monitoring, and func plus Puppet for remote execution and repeatable configuration. My estimate is that centralizing those tasks has cut manual labor by more than half while giving the team more capacity to troubleshoot the parts of the environment that actually need judgment. Bash and Python handle a lot of the glue.
Storage has its own tiers. We use commercial arrays where their capabilities justify the cost and commodity hardware with attached disks where cost and performance point in that direction. The arrays have backup partners and hot-fail methods. I keep evaluating global file systems and the possibility of moving more of the file-system layer back toward open-source systems.
Redundancy adds equipment and configuration. Two paths can still share the same mistake. A second firewall can carry the same bad rule. Two power feeds can meet upstream. A standby server is only useful if it has been tested.
The team has to understand the paths and exercise them. Monitoring has to tell us which layer is unhealthy before customer traffic proves it. Capacity planning has to show where growth is narrowing the margin. Maintenance has to improve the environment without treating the live store as a lab.
The cage makes the architecture physical. Every redundant path requires another cable, port, device, rail, power connection, and label. The design only works if those parts match the diagram and remain maintained.
Customers will not notice that the platform has more than 300 devices. They will notice if the store is unavailable or a download fails. Our next work is to keep documenting the environment, test the redundant paths, monitor capacity, and replace manual steps with repeatable configuration.
The team of seven systems administrators and network engineers will do that work while the store remains online.