I want a phone call any time the site goes down.
I joined Shutterstock in October, and I am still learning the systems and how the team responds when something breaks. A dashboard alert is not enough for me right now. If there is a critical failure, call my cell and pull me into it.
I want to hear what failed, what fixed it, and what we need to go back and repair properly. I also want to speak with the person who brought the site back and give them some credit.
We added me to the notification group. I tested my notifier and it called. Then I asked the team to trigger a failure from the site side so we could verify the full route. It is easy to test the phone in isolation and assume the system works. I want to know the production alert reaches the notifier, the notifier reaches me, and the person receiving it knows what to do next.
That site-side test still needs to happen. The phone test alone is not the full path.
Monday's office firewall failure had extremely poor timing. We were already rebuilding the network during the sprint, but the old problem arrived before the new path was ready. The immediate response moved us onto new firewalls, and new switches are planned before the end of the year.
I sent a note to the office afterward. I also asked for a broader state-of-the-union conversation when I return next week. There is more old data-center and network equipment to replace, and the company needs a straight list of what is broken, what we are replacing, and what will take longer.
Before people headed out for the holiday, I told the systems team I wanted us to talk as a group about the current issues, alerts, and on-call availability. They could choose the time. I was available whenever they called.
The request was simple: review the open issues, make sure the alerts are understood, and know who is available. If something critical happens, call me.
For now, the notifier itself works. The full site-side alert still needs to be tested. The office is on the replacement firewalls. The team call still needs to happen before everybody disappears for the holiday.