The GetGlue web servers need to stop keeping old logs forever.
I wrote the first version of the archive script today. Files older than fourteen days get copied to Amazon S3. Each server has its own directory on the other side, and the script is scheduled to run automatically.
I am not letting it delete anything yet.
For the first few days, a file that appears to upload successfully gets moved into a local directory called tmptrash. If I screwed up the S3 path, the upload was incomplete, or the script did something else dumb, the original is still sitting on the server.
Once it has run for a while and the S3 copies check out, I can purge tmptrash by hand. I am also looking at ways to confirm the remote object before allowing the local delete to happen automatically.
The first rollout is only two web servers. I will check the result with the person who knows the existing environment, then turn on the rest. Database dumps come after that.
The scripts follow the backup structure that is already here. Same sort of paths. Same sort of filenames. Nothing clever enough that the next person has to reverse engineer my reverse engineering.
This is week one of the consulting job. Moving fourteen-day-old logs is not the largest problem in the world, but it is exactly the kind of thing that becomes a problem if everybody assumes it is handled.
For now the files go to S3 and then into tmptrash.
I will delete them when I know I can.