The first copy finished syncing last night.
The other two are at 687 and 735 gigabytes. The data on these thumbnail systems is just over one terabyte.
I restarted one copy after the first one completed and left it running in a screen session. The load looked good. This morning I changed the path again: I stopped the two remaining jobs from pulling against the systems under production load and made them sync from the completed copy instead.
That should reduce the performance impact on the live systems and get both remaining copies finished faster.
One terabyte is enough that copying it becomes an operational event. The source system has a real workload. The destination disks have a limit. rsync has to walk the directory tree and compare state. A job can run for hours and still finish with the wrong ownership or an incomplete branch.
Running the job inside screen matters for a simple reason: the copy should continue if my terminal disconnects. I can detach, reconnect, inspect progress, and leave the long transfer alive overnight. That solves the session problem. It does not solve verification. We still have to inspect the destination after the bytes arrive.
The team is watching those details. The permissions and timestamps need to come across correctly. A number showing progress is useful, but the usable result is a full tree the application can actually read.
My first instinct was to run the copies in parallel from the original source. That is direct, and parallel usually sounds faster. It also makes the same production side serve two long-running readers while continuing its normal job.
Once the first copy was complete, it became the better source. It could carry the read load for the other two while the production systems returned to their normal work. The amount of data did not change, but the source of the work did.
The status emails are short: which copy is done, how far the others have moved, why I killed the existing jobs, where the new jobs are pulling from, and what the load looks like.
The remaining work is straightforward. The other two copies need to finish. Then we need to verify permissions, timestamps, directory completeness, and production load. They are still syncing from the completed copy, and I am watching the load until both jobs complete.
I will send the team another status when they do.