“Fun shiz” was MogileFS.
I had just met with the team at Blip.tv to talk about their experience with MogileFS. I came away interested in how it handled the problems I cared about: multiple data centers, large files, and a release line that appeared stable.
I came back ready to build a test pool.
The note to the team also pointed everybody toward our Zenoss and deployment graphs. We were the people with access to fix production. That meant we needed to know what normal looked like before something broke.
The weekend plan was to get virtual machines running on Xen with a manageable interface, then stand up MogileFS and start trying to hurt it. I asked everybody to think of fun test programs and bring over any other storage technology worth exploring.
I meant fun literally. I wanted to load it, break it, and understand what happened.
The order mattered. First, get the virtual machines running. Then build the storage pool. Then write the tests and push it. I was not asking the team to install a new production answer over the weekend. I was asking us to create a place where we could find out what the software actually did.
The graphs were part of the same work. Before changing storage, we needed to watch the systems we already had and recognize their normal behavior. Zenoss and the deployment graphs were sitting there for us to use. If we were the people who could fix production, we should be looking at them.
We had not selected the answer. First we had to build the test pool and run the tests.
That was the plan for the weekend in the office.