Datadog asked what we were using to monitor the infrastructure at Shutterstock.
The honest answer was that we had built a highly customized system and were collecting roughly 800,000 data points every second. Shipping all of that to a hosted service did not look realistic.
That did not make the conversation useless. It changed the question.
I already knew the company and had spoken with its cofounder before. The product’s event correlation and interface were interesting, especially in places where our own monitoring experience could improve. I asked how long the free trial lasted and suggested I might hack on sending a lower-volume stream of discrete events instead of the full metric firehose.
This is the part of technical evaluation I enjoy. A vendor arrives with a broad pitch. The easy reactions are to buy the whole thing or dismiss it because one dimension does not fit. The more useful reaction is to identify the narrow piece that might solve a real problem and test that.
Our volume was not a vanity number. Monitoring can become an infrastructure system of its own. Collection, storage, dashboards, alerts, and retention all have costs. Moving 800,000 observations per second across the internet would introduce expense and another dependency. It would also ask a hosted product to absorb a workload very different from a normal trial account.
Events were another matter. A smaller stream of deployments, configuration changes, alerts, and operational milestones could help explain why a graph moved. Metrics tell you that something changed. Events can give the graph context.
I did not promise a rollout. I asked for room to experiment.
That distinction is important. Good vendor conversations are not procurement theater, and they are not architecture by sales deck. They are a chance to compare an outside product with the system you actually run, including its awkward scale and the tools you have already invested in.
The full metric stream was a bad fit. A focused experiment around event isolation, correlation, or a better interface might not be. That was enough reason to keep the door open and spend a little time hacking on it.