/Programming

It's three in the morning and the script that runs the database backup just failed. It didn't tell anyone. It didn't produce any useful log. It simply stopped halfway through, left a two-gigabyte temporary file sitting on disk, and moved on as if nothing had happened, because the next command in the script ran anyway. Nobody noticed until the next morning, when someone went to restore a backup and found that the most recent one was three weeks old. If you've written bash for more than a few months, you've probably lived some version of this story. And it's not because of carelessness. Bash was designed at a time when scripts were short sequences of commands typed by hand, not pieces of infrastructure that run on their own in cron, in CI pipelines, inside containers, sometimes for years without anyone rereading the code. The language hasn't changed to match how we use it today, so it's up to whoever writes the script to make up for that. ## Why bash fools even experienced people The root of almost every problem of this kind is simple to state and easy to forget: by default, bash keeps executing after a command fails. If line three of a script errors out, line four runs normally, as if everything were fine. In a script that deletes temporary files, spins up containers, or moves data around, that's the difference between a handled error and a silent disaster. Comparing this to other languages makes it stand out: in Python an unhandled exception stops the program, while in bash a failing command only stops the program if you explicitly say that's what you want. And that's exactly where most of the scripts that "work on my machine" and then fall apart in production come from. ## The three flags every script should have The first line after the shebang, in most serious scripts, should be this: ```bash #!/usr/bin/env bash set -euo pipefail ``` The `-e` makes the script stop as soon as a command returns an error. The `-u` makes the script stop if you try to use a variable that was never defined, which prevents that classic case of `rm -rf $DIR/` turning into `rm -rf /` because `$DIR` was empty. The `-o pipefail` makes sure that in a pipe like `command1 | command2`, the script notices if `command1` failed, even if `command2` ran fine afterward. That said, these three flags have limits worth knowing, because trusting them blindly creates a false sense of safety. `set -e` doesn't stop the script if the failing command is inside an `if` condition, because in that context bash assumes you want to test the result, not halt everything. It also doesn't work inside functions called in certain contexts, nor in command substitution used as the value of an assignment. In practice, this means these flags are the foundation, not the full solution, and every critical part of the script still deserves explicit checking. ## Trap: making sure the mess gets cleaned up even when everything goes wrong Every script that creates a temporary file, opens a lock, or spins up a child process will, sooner or later, get interrupted mid-run, whether because of an error, because someone hit Ctrl+C, or because the parent process was killed. The question that matters is what's left behind afterward. ```bash tmpfile=$(mktemp) trap 'rm -f "$tmpfile"' EXIT # the rest of the script uses $tmpfile normally ``` The `trap` tied to the `EXIT` signal guarantees that cleanup command runs no matter how the script ends, whether it succeeds, errors out, or gets interrupted manually. That's the difference between a server that accumulates junk from months of failed runs and one that cleans up after itself. ## Quoting variables: the silliest mistake that breaks the most things in production Forgetting to quote a variable looks like a style detail, but it's probably the most common cause of bizarre bugs in bash. Without quotes, bash splits the variable's value into pieces separated by whitespace and then expands anything that looks like a filename pattern, which means a filename with a space in it, or an empty value, can make a loop behave completely differently than expected. ```bash # dangerous: if $file has a space or is empty, this breaks for f in $(ls $dir); do rm $f done # safe for f in "$dir"/*; do rm -- "$f" done ``` The practical rule is: every variable that holds a path, a filename, or any text coming from outside the script should be in double quotes every time it's used, no exceptions. And `"$@"` always quoted, never `$@` unquoted, because only the quoted version preserves each argument as a separate unit. ## Validate before acting, not after A robust script is suspicious of everything before it does anything irreversible. That means checking whether the required arguments were passed, whether the files and directories the script depends on actually exist, and whether the external commands the script relies on are installed in the environment where it's running. ```bash command -v jq >/dev/null 2>&1 || { echo "jq not found, aborting"; exit 1; } [[ -d "$backup_dir" ]] || { echo "directory $backup_dir does not exist"; exit 1; } ``` This seems obvious until you remember that most scripts that break in production break precisely because they were written and tested in an environment that had everything set up, and then ran on a new server, inside a minimal container, or on another team's machine where a dependency simply wasn't there. ## Logs that help someone at three in the morning, not just logs that exist A log that says "error" doesn't help anyone. A useful log records when it happened, what was being done, and why it failed, in a way the on-call person can understand without having to reread the whole script. ```bash log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$1] $2" } log "INFO" "Starting backup of database $DB_NAME" log "ERROR" "Failed to connect to $DB_HOST on port $DB_PORT" ``` It's worth the habit of sending these logs both to the screen and to a file, and including enough context that someone who has never seen the script before can diagnose the problem just from reading the message. ## Preventing the same script from running twice at once Scheduled cron scripts have a common problem: if one run takes longer than the interval between schedules, the next one starts before the previous one finishes, and having two instances of the same script touching the same files or the same database table at once is a recipe for data corruption. ```bash exec 200>/var/lock/my_script.lock flock -n 200 || { echo "an instance is already running"; exit 1; } ``` This lock using `flock` guarantees that if the script is already running, the new attempt simply gives up instead of running in parallel and creating a race condition. ## Testing before trusting ShellCheck is the non-negotiable minimum for any script that goes beyond three disposable lines. It catches exactly the kinds of errors described here, forgotten quotes, undefined variables, wrong comparisons, before the script gets anywhere near production, and it runs in seconds. ```bash shellcheck my_script.sh ``` For those who want to go further, there's Bats, which lets you write real tests for scripts, simulating inputs and checking outputs, the same way you'd test a function in any other language. That's overkill for a script that runs once a month, but essential for anything that supports part of the production infrastructure. ## What to check before putting a script into production Before a script goes into cron, into a CI pipeline, or anywhere it will run without direct supervision, it's worth going through one final check: whether it has `set -euo pipefail` on the first line, whether it has a cleanup `trap` for any temporary file or lock it creates, whether every variable representing a path or filename is quoted, whether it validates dependencies and arguments before acting, whether the logs would be enough for an on-call person to understand a failure without rereading the code, whether there's protection against concurrent execution when that matters, and whether ShellCheck passed without ignored warnings. A script that passes this whole list isn't a perfect script, but it's a script that's unlikely to wake you up at 3am over something silly. --- If you've ever had a bash script blow up in production, this is the moment to share the story in the comments. There's almost always someone who's hit the same mistake and has a tip that would have prevented it.