Quest for improvement - shell-scripting

Shell scripting patterns: paths, file globs and and current dir

quest — Mon, 04 Jul 2011 22:01:21 +0000

The basis of this pattern is simple: never ever change current working directory in your scripts. In fact, never ever assume that you have a working directory unless your command explicitly works from local directory (e.g. find). Consequently, you should never use relative paths in your script unless you got it from the caller; since your script doesn't change working directory, it's fine to just use whatever path the caller handed you.

There are numerous reasons for this. The most straightforward reason for this is that cd executed without an argument will change directory to the executing user's home directory, which is arguably where the script can do the most damage.

There is one case where your script needs to refer to a relative path, namely if your script is distributed together with other files (e.g. in a zip file) and should be runnable after being unzipped. The script must be able to find files/directories relative to wherever the script resides, so relative to the current working directory does not cut it if the user tries ./zipdir/script.sh. Fortunately, all shells have a $0 that is the path of the script itself. Thus, you do this:

DATADIR=$(dirname $0)/data

While you are avoiding things, also avoid shell expanded glob expressions. They are simply too imprecise and it is too easy to get bitten by paths with space in them. In 99 of 100 cases, find is better than shell glob. find is your friend.

find /dir/with/cruft -mindepth 1 -maxdepth 1 \
     -type f -regex '.*\.sh' | frobnicate_script

There are numerous things going on here. mindepth/maxdepth describes how deep to look; a depth of 0 means the path supplied to find, so 1/1 means "look in the directory /dir/with/cruft". type f means "look only for files". regex uses regular expressions to match files, which is a lot more expressive than globs (normally supplied to find via -name). It is tempting at this point to reach for -exec, but please refrain. The exec option creates code that is very difficult to read.

Instead xargs is the friend of your find. If all you want to do is a simple operation on each found file, you can do something like this:

find $rootdir -mindepth 1 -type d -cmin -10 -print0 | xargs -0 chmod o-rwx

Or, in English: remove world permissions on all directories under $rootdir that I created the last 10 minutes. Find option -print0 will terminate each line with a null character instead of a newline and -0 will tell xargs to ignore whitespace and instead react to null characters. Presto, no problem with space (or even newlines) in paths.

Discouraging users from using delimiter characters (such as space, tab, newline, comma, pipe, et.c.) in file and directory names is part of the holy cause. Nevertheless, some file will have them, sometimes even for good reasons. So your script needs to be prepared for them. Quote all data that comes from outside.

As an aside, it is an annoying aspect of shell scripting that most syntax highlighters will treat e.g. "$1" as a string and give it string colour, but $1 as a variable and give it variable colour. Most unhelpful.

Tags:

Shell scripting patterns: errors and failure

quest — Mon, 04 Jul 2011 05:22:28 +0000

In my opinion, error handling is the area where shell script clearest shows its age. The idea that the shell will by default ignore that commands signal an error state (i.e. a non-zero exit code) seems very strange when viewed through the lens of modern programming theory. Thus, all scripts should start with this instruction, that tells the shell to care about exit codes:

set -e

Now, the following code will no longer result in tears when /destination does not exist:

process < /source/file > /destination/file
rm /source/file

To explicitly ignore the exit code of a command, use:

process < /source/file > /no/such/file || true

... or if you care, but don't want to just bail, you can:

may_explode || result=$?

Just remember that $result will be empty rather than 0 if all goes well, so you cannot do [ $result -gt 0 ].

In rare cases, it may genuinely be that you have a section of instructions that you want to execute even if the previous fail. If so, use a subshell:

(
    set +e
    ...
)

The set operation will only be active in the subshell.

It is quite possible to achieve something akin to modern exception handling in bash shells. Consider the following:

(
    do_this || exit 1
    do_that || exit 2
) || {
    fail=$?
    case $fail in
    1) handle_case_1 ;;
    2) handle_case_2 ;;
    *) exit $fail ;;
    esac
}

Of course, do_this, do_that may already return decent exit codes such that you don't need the '|| exit' bit. More generally, now that set -e is turned on, you need a mental model of execution flow, much like any other programming language. Just remember the law: any context (be it subshell, function or block) has an exit code/return value of the first statement executed that returns a non-zero exit code/return value. Somewhat simplified, as soon as a statement returns non-zero, execution will abandon successive scopes until someone "handles" the non-zero state. The three main ways to "handle" the non-zero state are pipe "|", command substitution "$()", or "||". The background control operator "&" also does not propagate the exit code (it can later be retreived with wait).

It may be tempting to use exit to make life simpler inside functions, but you should resist. exit will abort execution of the current shell, but parentheses and pipes create subshells, so a function cannot know whether it will abort the master shell process or just a subshell. Use only return statements in functions and save exit for top-level code. exit should only be needed in top-level code to tell the caller that parameters or input were invalid.

Tags:

Shell scripting patterns: returning from functions

quest — Tue, 28 Jun 2011 22:12:51 +0000

One shortcoming of shell scripting is the inability to return anything of significance from a shell script function. Consider: get a function that returns the youngest file in a directory. The basic moving part in this is ls -tr /da/dir | tail -1. Abstracting this to a function seem problematic, given that we cannot return a value. However, functions is very similar to external commands in bash, so you can do this:

functions latest_file() {
  ls -tr "$1" | tail -1
}

... and then simply invoke it thus:

latest=$(latest_file /da/dir)

From this point, we can write what effectively is a generator:

function find_magic_scripts() {
    find $1 -type f -name '*.sh' | while read fname ; do
        grep -l 'magic word' $fname || true
    done
}

We can now use it as a source for a consumer:

find_magic_scripts /da/dir | link_and_version

These two processes will be run in parallel, which is good since they both take a little time. Of course, find_magic_scripts will still block as soon as it has output a file, but will restart as soon as link_and_version has read that line.

However.

There is an issue with this pattern: a sequence of pipes will have the error code for the last command, ignoring any previous failures. So, if /da/dir does not exist, link_and_version will simply get EOF and be happy. Unfortunately, there is no neat workaround that solves all problems. The best strategy is usually to introduce an explicit failure marker in the protocol over the pipe, something like:

function foo() {
    {
        ...
    } || echo "FAIL $?"
}
foo | bar

Or, where left side is an external command, simply:

{ find /may/not/exist -name '*.sh' || echo "FAIL $?" ; } | bar

Now bar() can know that something went wrong and abort.

Tags:

Shell scripting patterns - a personal perspective

quest — Sat, 25 Jun 2011 11:15:00 +0000

It is often said that no serious development project should ever use shell script.

While it is certainly true that there are some serious shortcomings is the bash strain of shells, this is not to say that these shortcomings cannot be overcome. Like so much else in programming, it is mostly a matter of finding good patterns that contain those shortcomings.

I want to write something on this matter, so I'm planning a series of blog posts on important patterns that help in keeping bash shell scripts readable and maintainable while keeping a reasonable performance. With luck, they will have something for you as well.

Tags: