Thumb01 finished syncing last night.
Thumb02 is at 687 gigabytes. Thumb03 is at 735. The full thumbnail data set is just over one terabyte.
I restarted the second copy after the first one completed and left it running in a screen session. The load looked good. This morning I changed the path again: I stopped the copies pulling from the systems under production load and made Thumb02 and Thumb03 sync from the completed Thumb01 instead.
That should reduce the performance impact on the live thumbnail servers and get both remaining copies finished faster.
This is the kind of work I expected at Shutterstock, although the specific names and directory trees are still new to me. The company sells stock images online. The product looks like search results, preview images, and files delivered to customers. Underneath that experience is a storage system full of derived assets that have to exist in the right place, with the right names, permissions, and timestamps.
The full collection is large enough that copying it affects production systems.
One terabyte in 2010 is not an impossible amount of data. It is large enough that copying it becomes an operational event. The source system has a real workload. The destination disks have a limit. Network paths have capacity. rsync has to walk the directory tree and compare state. A job can run for hours, appear healthy, and still finish with the wrong ownership or an incomplete branch if we are not checking the result.
Running the job inside screen matters for a simple reason: the copy should continue if my terminal disconnects. I can detach, reconnect, inspect progress, and leave the long transfer alive overnight. That solves the session problem. It does not solve verification. We still have to inspect the destination after the bytes arrive.
The team is watching those details. The directories receive the correct permissions once the files underneath them arrive. Timestamps are coming across. The remaining copies are close enough that we can see the finish, but not close enough to stop paying attention. A percentage complete is useful, but the usable result is a full tree the application can actually read.
My first instinct was to run the copies in parallel from the original source. That is direct, and parallel usually sounds faster. It also makes the same production side serve two long-running readers while continuing its normal job.
Once Thumb01 was complete, it became a better source. The finished copy could carry the read load for the other two. The production servers could return to their normal workload. The total amount of data moving did not change, but the source of that work did.
The network can have enough total bandwidth while one disk, process, mount, or application host remains the narrow point. Adding another copy job to the same production source can slow the source and the copies. Moving the remaining reads to Thumb01 removes that pressure from the live thumbnail servers.
I have been at Shutterstock for about ten weeks. I am asking to be called on critical incidents, rebuilding parts of the office network, setting up DNS, and now learning the storage paths through a terabyte of thumbnails. The work moves between architecture and commands very quickly.
The copy gives me a direct view of how the current storage system behaves under load.
The emails around this are short because there is not much value in decorating the status. Which copy is done. How far the others have moved. What the full size is. Why I killed the existing jobs. Where the new jobs are pulling from. What the load looks like.
The remaining work is straightforward. Thumb02 and Thumb03 need to finish. Then we need to verify permissions, timestamps, directory completeness, and production load. They are still syncing from Thumb01, and I am watching the load until both jobs complete.
I will send the team another status when they do.
-Chris