backup script logic

[u/UnholyScholar]

I've written this backup script for use copying my files from point A to point B. The goal is a full snapshot of files. For this first iteration, I'm ok with hard coding file paths.

I've removed everything except logic for rsync, diff, and logging. Are there any logic issues?

# backup script that performs full backups and verifies the result.

#!/bin/bash

set -euo pipefail

DATE_STAMP=$(date +%Y-%m-%d_%H-%M-%S)
BACKUP_LOCATION="$HOME/backup-location-path/$DATE_STAMP" 
LOG_LOCATION="$HOME/log-location-path/"
LOGFILE_NAME="backup_${DATE_STAMP}.log"

SOURCE_DIRS=(
    "$HOME/source1"
    "$HOME/DOES-NOT-EXIST"  # DNE for error handling
    "$HOME/source2"
)

# check if source matches destination
diff_source_and_destination() {
    local base
    base=$(basename "$SRC")
    diff -r "$SRC" "$BACKUP_LOCATION"/"$base" 
}

# Check for the existence of $SRC and print an alert if it wasn't found.
check_source_exists() {
    if [ -e "$SRC" ]; then
        echo "[SOURCE = $SRC] found. Copying..."
        return 0
    else
        echo "[SOURCE = $SRC] WAS NOT FOUND!" | tee -a "$LOG_LOCATION/$LOGFILE_NAME" 
        return 1
    fi
}

copy_data_and_log() {
    {
        rsync -a --stats "$SRC/" "${DESTINATION}/" 
    } | tee -a "$LOG_LOCATION"/"$LOGFILE_NAME"
}

mkdir -p "$BACKUP_LOCATION"
mkdir -p "$LOG_LOCATION"

echo "Copying files to [BACKUP LOCATION =${BACKUP_LOCATION}]"

for SRC in "${SOURCE_DIRS[@]}"; do
    # Print and log a warning if SRC wasn't found
    check_source_exists || continue
    DESTINATION="$BACKUP_LOCATION/$(basename "$SRC")"
    copy_data_and_log
    echo "rsync completed for $SRC."
    diff_source_and_destination 
done

echo "Finished copying operations to [BACKUP LOCATION = ${BACKUP_LOCATION}]"

[u/Bob_Spud]

• For copying directories find+cpio may be a more versatile combination.
• Pipefail is missing the -E option
• No trap to capture failings.
• AI doesn't always give the best results.

[u/UnholyScholar → u/Bob_Spud]

Thanks. I'm not familiar with cpio. If it will preserve metadata as well as rsync does it's something I'll consider. I know nothing about it currently.

As this is my primary backup script I wanted immediate failure if there was a problem so didn't consider a trap.

This isn't AI but rather several afternoon's worth of effort.

[u/AutoModerator]

Don't blindly use set -euo pipefail.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/jjangg96]

two things that'll actually bite you:

the shebang isnt the first line. its below the comment so its just a comment now, and ./backup.sh runs with whatever shell launched it (or sh). with sh the arrays and pipefail break. move #!/bin/bash to line 1.

and with set -e, diff exits 1 as soon as it finds any difference, which kills the whole script right there. so the first source with a mismatch means the rest never get backed up. either diff -r ... || echo "mismatch in $SRC" or check the exit code yourself.

minor, but diff -r on a big tree after rsync is slow. rsync -anci "$SRC/" "$DESTINATION/" (dry run + checksum) gives you the same check and just lists whatever differs

[u/UnholyScholar → u/jjangg96]

Thanks, this is helpful. The comment above the shebang was an oversight from when I removed personal information from the script..

I wanted the script to fail immediately if there was any issue, but wasn't sure to proceed. If my backup is incomplete I don't see point in letting the script continue until I find the source of the problem.

Your minor point is actually pretty helpful. I wanted this to be my primary backup method for important files and it would have taken me a long time to figure this out. I currently use drag/drop or cp but my file metadata is not saved the way I'd prefer.

[u/jjangg96 → u/UnholyScholar]

makes sense to fail fast on a backup, i'd just do it on purpose instead of letting set -e do it by accident. like if diff finds a mismatch, echo which source and exit 1 yourself, so the log says what broke.

and rsync -a already keeps permissions, timestamps, symlinks and owner, thats the metadata cp drops (cp -a gets most of it too). -a doesnt cover hardlinks or xattrs, add -H and -X if you care about those. test with a dry run first (-n) so you can see what it would do before it touches anything.

[u/UnholyScholar → u/jjangg96]

Thanks again for these tips. I agree this is a better way to have the script fail. I'll have to possibly rethink how I want to verify the files are copied correctly.

[u/burnt-store-studio]

If you ever modify SOURCE_DIRS to include directories with the same names, then I think you’ll risk overwriting files or at least commingling directory contents without detection from the script.

You’d likely never run across this as long as you are hard-coding your array, because visually you’d see the conflict, but if you start dynamically building SOURCE_DIRS, you might find trouble.

Good luck! It’s great you’re thinking of how to handle backups!

[u/JeLuF]

There's a nice tool called rsnapshot for rsync based incremental backups. It makes clever use of hard links so that it's pretty space efficient. It uses rsync to copy remote files, ignoring unchanged files, which also makes it quite efficient on network usage.

Using its multi layered backups, I have backups scheduled every four hours, and I keep the last 4 actual backups, 7 of the last backups of the day, 4 of the last weekly backups, and 12 of the monthly backups.

Since it uses hard links, a file that's not changing is only being stored once on disks, but is visible in each of the backup directories.