We were preparing to take Shutterstock down on purpose Thursday night.
A hardware failure had left the production firewalls out of sync. Returning them to a healthy pair required replacing hardware, upgrading the software, and rebooting. Until we did that, another firewall failure could force the same work under emergency conditions.
The planned outage was the safer option.
We had tested the maintenance in Texas, where the firewall reboot took about 45 minutes. We reserved a two-hour window so the team could test the sites before restoring traffic and still have room if the reboot ran long. I thought we would probably finish in an hour and were very unlikely to need more than two.
The ugly scenario was a failed software update that left the devices unusable. In that case, we would have to rebuild them and restore the configuration from backup. I put the probability below one percent, but the possible outage could stretch from two to six hours.
That risk needed to be stated plainly. “Routine maintenance” was not enough information for the people running the company. They deserved the expected case, the worst credible case, the reason we could not avoid downtime, and the resources we would have in place.
Thursday at 10 p.m. gave us the best combination of low customer impact and strong support. The internal team would be ready. The network vendor had reviewed the plan. Remote hands would be available in the primary data center.
At that point our sites were not active-active across locations, so there was no clean way to perform this maintenance without a full outage. That architectural limitation was real. Waiting would not remove it. Waiting would only let the next hardware failure choose the time for us.
I was confident because the plan was concrete and tested, not because the work was risk-free.
The go/no-go meeting still mattered. If a new fact changed the risk, we could stop. But based on what we knew, a controlled outage with the right people awake was better than an uncontrolled outage with everyone scrambling.