“Fun shiz” was MogileFS.
That probably explains a lot about me.
I had just met with the team at Blip.tv. They were running roughly 1.5 petabytes of media on MogileFS and were happy with it. The project had worked through the problems I cared about: multiple data centers, large files, and a release line that appeared stable.
I came back ready to build a test pool.
The note to the team also pointed everybody toward our Zenoss and deployment graphs. We were the people with access to fix production. That meant we needed to know what normal looked like before something broke.
Graphs were not decoration. They were how we learned the shape of the site.
The weekend plan was to get virtual machines running on Xen with a manageable interface, then stand up MogileFS and start trying to hurt it. I asked everybody to think of fun test programs and bring over any other storage technology worth exploring.
This was not a vendor bake-off with matching slide decks. It was engineers comparing notes with people already running the system, then putting the software on machines and seeing whether our own workload agreed.
I meant fun literally. A new system was a thing to poke, load, break, and understand.
There was a practical goal underneath the enthusiasm. Shutterstock's image library was not getting smaller. Storage had to grow without becoming one very expensive box that could ruin everybody's night.
We had not selected the answer. We were doing the work required to deserve an opinion.
That part was fun too.