The firewall was going into the colocation facility that night.

I sent a three-part plan before starting. Install the firewall and connect the external router. Build the internal network for two new Linux Virtual Server machines. Rebuild the load-balanced services without moving any production hosts yet.

The existing switches could keep the two sides of the network separate for now. It was not the finished design, but it was enough to create a WAN side and a LAN side until better switching and VLAN support arrived.

The new load balancers would sit behind the firewall on their own internal network. Once that path worked, each production service could move separately. Create the mapping on the firewall, remove the service from the old load balancer, change the backend addresses, test it, then move to the next one.

There would be downtime, but it should be brief and limited to one service at a time.

This was side work after the Beatport day. The plan was not a formal migration document. It was an email that began, “Here is a brief plan for what I am going to do tonight.”

That was enough. Everybody knew the order, what could go down, and what would still be left for later.

By just after midnight, the firewall issue was fixed and the first virtual load balancers were ready. The production cutovers still needed to be scheduled, but the new path existed and had passed the initial tests.

The important part was not pretending the whole environment could change in one shot. Get the network ready. Get the replacement services ready. Then cut them over one at a time and keep the outages small.

Archive