The mobile front page at Shutterstock was failing on a Sunday afternoon. We restarted the service supervisor on the mobile nodes. The alert kept firing.
Someone from the mobile team tested the site from a phone. The health check loaded. The front page loaded. Login worked. Search worked. Adding an image to a lightbox worked. Everything a customer was likely to do in that test worked.
The question was whether the person on call could acknowledge the alert and get back to his Sunday.
My answer was yes. We did not have evidence of a customer outage, and the obvious restart had already been tried. There was no reason to keep the operations person actively engaged while another team investigated a narrower problem.
But there was still a problem. The automated test was failing, and we did not know why.
The manual check told us the main mobile functions worked at that moment from one phone and one account. The automated test was checking a specific path that had worked before. It might have been a bad test, or it might have found a smaller failure that the manual test missed. Either way, the mobile team needed to look at it.
I wrote that there was definitely a failing test even though the human checks appeared okay. We had looked at the logs, and the problem was beyond restarting services. Operations could acknowledge the page. The mobile team should investigate the programmatic failure as soon as possible.
That kept the response proportional to what we knew. It was not a full outage. It was also not something to ignore.
We had learned not to keep restarting services just to see whether an alert would turn green. Once the normal restart did not work, repeating it could hide useful evidence or create another problem. The team that owned the mobile code and the test had a better chance of finding the cause.
The alert itself also had to remain credible. If everybody learned that this check could fail while the site was fine, they would eventually stop taking it seriously. Then it would not help us when there was a real customer problem. A broken test was work, even when it was not an emergency.
We also needed the mobile team to compare the automated path with the actions that had just been tested by hand. The failed assertion might have depended on a response, page element, or sequence that a person did not notice. If the assertion was obsolete, it should be updated. If it found a real defect, the team needed to understand its customer impact.
This happened less than three months after Shutterstock went public. At the time, the marketplace had more than 20 million images and had completed more than 250 million downloads. The company had priced 4.5 million shares at $17.
Those were the numbers people outside the company saw. Inside the company, the day-to-day work was still a Sunday alert, a restart, a phone test, and a short email deciding who needed to keep working on it.
The company being public did not change that basic operating process. It did increase the cost of weak monitoring and unclear ownership. A customer-facing ecommerce site at that scale needed tests people trusted and a team that knew which group took over when a check failed.
I wanted the person carrying the pager to know when he could step away. If every alert stayed active until every possible cause was resolved, on-call work would become miserable. I also wanted the product team to understand that a passing manual check did not close the issue.
So we split the response into two parts. First, confirm that customers could use the mobile site. That lowered the urgency and released operations. Second, leave the failing automated test with the mobile team so they could find out what had changed.
There was no larger incident in the source material I have from that day. There was no dramatic fix. The site worked when tested, and the automated check did not. We gave the remaining work to the right team and moved on.