The POP nodes died overnight. About a thousand users were affected for four hours because nobody answered the monitoring call at three in the morning.
I added the incident to the outage tracker.
I joined Peek this month as Director of Technology Operations. Peek sells a dedicated mobile device for email and messaging, which means the device can look simple while the service behind it is doing a lot of work. AOL, Gmail, Yahoo, Exchange, old and new POP services, IMAP Idle, texting, sending, receiving, registration, attachments, and every strange formatting decision an outside email provider can make all meet inside the product.
When one of those pieces dies overnight, the person holding the device does not see a node. They see missing email.
The operating history is rough and useful. Availability was 97.8 percent in the fourth quarter of 2008. January and February were both below 99 percent. There were database failures, provider outages, capacity problems, web-scrape changes, memory issues, routing failures, deployment errors, an attack, and slow queries. Some incidents affected a small group for days. Others hit everybody for an hour or two.
The numbers improved. June reached 99.998 percent by the tracker. July slipped to 99.4 percent, with most of the recorded outage time tied to deployment activity. August is at 99.7 percent so far.
On August 2, a deployment error left the web tier without enough capacity and affected roughly half the user base, about 3,300 people, for three hours. Today's incident affected about a thousand. A monthly availability number compresses those people into a decimal. The incident list puts them back.
The monthly percentage does not show the impact by itself. The incident list records the number of users, duration, cause, and proposed fix.
It also makes the operating problems harder to excuse. Too many deployments are causing outages. Slow queries are back. We still have missing-email questions and possible timestamp problems. The tracker lists systems that crash on malformed email, lost sends, lost reply bodies, domain guessing, TLS requirements, registration failures, formatting issues, and administration tools that cannot find an account cleanly.
The incident tracker gives us a long work list.
One project is automated deployment, called Operation Porter. The first nodes are live in production, with more coming. It should reduce hand-built differences and deployment mistakes that turn a release into an outage.
Capacity work is another project. We need to benchmark the database, Apache, and web servers, understand actual traffic patterns, and know when another node changes the answer. Security is still being audited. Operational reporting has a first version live. An email integration is breaking messages into pieces in a way nobody would call pretty.
Then there is the call at three in the morning.
Monitoring can detect that a node stopped, but it cannot answer the phone. We need better alerts, clear coverage, escalation, and a named person responsible for overnight failures.
I know this problem from Beatport. A service open across time zones does not share the office schedule. Peek adds another layer because the customer carries the expectation in a physical device. They bought a thing for email. If email is not there, explanations about upstream providers, queues, memory, or POP nodes do not restore the promise.
I am still learning the system and the team. The outage tracker shows the hours, affected users, cause, and proposed resolution. Sometimes the resolution is specific: add memory, keep a standby Apache node, fix the code, or roll back. Sometimes the root cause is still unknown and needs more investigation.
The POP nodes are back and the thousand affected users have service again. Next we need to confirm the on-call coverage, test the escalation path, and make sure the next overnight node failure gets a response before four hours pass.
We also need to finish the automated deployment work and review the deployment failures already recorded in July and August. Those are separate problems, but both affect the same availability number.