Quest for improvement - software-development

What does it mean to develop software?

quest — Thu, 04 Oct 2012 12:14:48 +0000

Consider the following:

In principle, in the general purpose computing world, once you check in to your version control system a solution to a certain problem, you will solve that problem for everyone everywhere for all eternity.

You may counter that if that were true, there would be only one Get Things Done app for Android, rather than 100. However, this springs more from disagreement on what the problem is.

If we take that basic premise to be true, we need to gradually increase our capacity to solve problems. If we do not, logically we must sooner or later run out of problems that we can actually solve.

Or, bluntly put, writing software is the craft of producing unemployment.

Now, Joseph Stigliz may say that this has a lot to do with cheap capital favouring capital investment (such as massive compute farms and powerful software systems) over investment in manpower. This may be so, but this has been the norm for the last decade and a bit, and is likely to remain true for a decade to come while banks and states deleverage. Thus the principle will stand for the forseeable future (several aeons by ICT yardsticks).

Tags:

The twelve-factor app

quest — Sat, 12 May 2012 08:03:13 +0000

I do not normally do "look-it-I-found" posts, but sometimes you encounter some article, or even a whole community, that makes you go "This is exactly what I feel!"

The Twelve-Factor App is such an article. It collects in one compact article several relevant points about writing software-as-a-service, that I have been trying to make. In particular, the article articulates the relationship between such things as dependencies, releases and deploys, which I have long felt to be marginalized in the greater software development discussion.

Before you think "But I don't do SaaS stuff," be advised that all of these recommendations are good practices even for more traditional software development. In fact, I think it excessive modesty on the part of the author to limit himself to this scope.

I agree with and would emphasize each and every rule in The Twelve-Factor app, bar one. I have one real piece of beef with the article, namely its view of configuration.

The world is not so simple

Says section III:

The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.

There are several problems with using environment variables as the main configuration vehicle:

environment variables can not easily be changed after startup,
many applications have far too much configuration for environment variables to be useful, and
environment variables are too visible.

Changes after startup: The article envisions that running a process should perform a task once and then return/exit. Thus, configuration changes can occur between restart. However, there are numerous legitimate use cases where you want to change configuration of a running process inside the scope of its task. For example, you may have a web trawling process and you want to change its logging verbosity or what back-off delay it uses.

Configuration volume: Certain applications have a large volume of configuration. In particular, commercial software typically pass several slightly different use cases over its lifetime as it passes from one financing customer to the next. This means it will accumulate configuration options to select between those use cases or simply need to be configurable by customer or similar. This will quickly grow unmanageable if set through environment variables. One may argue that these are not optimal circumstances for software development, but the alternative is typically bankruptcy.

Environment variables are too visible: It is difficult to manage security of environment variables. Some ways they may leak:

ps e
Set in shell before startup
.bash_history or .zhistory

By contrast, a configuration file can be referred to by name without revealing its content, which makes it much preferable; most configuration files will end up containing credentials sooner or later.

Making room for configuration management

The brilliance of The Twelve-Factor App is that it carves out a distinct niche for Configuration Management, essentially by excluding it from the software development process altogether. Also, it pushes much of process management to systems administrators, by empowering them.

However, those systems administrators may have to manage numerous different installations with different configuration sets. For example, an organization may run one instance per customer. At this point, the configuration will have to be versioned and managed in its own right, whereby the environment variables will have to be written to and loaded from file anyway.

Alternative recommendation

While environment variables may be useful to name the configuration file or configuration backing service, using them as the main configuration strategy will lead to grief in most non-trivial cases. It may work in the early stages of the life cycle, but sooner or later, chances are you will out-grow that strategy.

Rather, my recommendation would be to use either of two strategies, depending on the use case:

plain configuration file, using whatever standard tools are available in your environment, or
configuration backing service, when configuration becomes complex; in particular, when different processes need to behave differently depending on e.g. connecting user.

This critique notwithstanding, my hat is off to Adam Wiggins of Heroku fame.

Tags:

Shell scripting patterns: configuration and option parsing

quest — Wed, 06 Jul 2011 20:49:51 +0000

Adhering to software patterns lowers the threshold for doing the right thing, even when the quick hack beckons. This is particularly true in shell script handling of options and configuration. Essentially, the top of all scripts should look like this:

OPTION1="default_value"

. /etc/my.conf

function usage() {
    echo "./dascript [-o <value>] -p <value>"
    echo " -o: set the value of option1"
    echo " -p: set the value of option2"
}

while getopts "ho:p:" opt ; do
    case "$opt" in
    o) OPTION1=$OPTARG
    p) OPTION2=$OPTARG
    h) usage ; exit 0 ;;
    *) usage >&2 ; exit 1 ;;
    esac
}

[ -n "$OPTION2" ] || {
    echo "Please supply -p <value>"
    usage
    exit 1
} >&2

Copy the above example each time you get to the point where you need any configuration whatever for a script. It's much easier than trying to cheat.

So what is going on?

List all parameter with their default values.
Source a configuration file, which will overwrite the default value.
Define a function usage that prints all available options (so that we can use it when parsing options).
Parse options supplied on the commandline to allow overriding the configuration file.
Verify that options have good values, including mandatory options.

The only time this pattern fails is when you need to allow supplying the name of the configuration file.

Tags:

Shell scripting patterns: paths, file globs and and current dir

quest — Mon, 04 Jul 2011 22:01:21 +0000

The basis of this pattern is simple: never ever change current working directory in your scripts. In fact, never ever assume that you have a working directory unless your command explicitly works from local directory (e.g. find). Consequently, you should never use relative paths in your script unless you got it from the caller; since your script doesn't change working directory, it's fine to just use whatever path the caller handed you.

There are numerous reasons for this. The most straightforward reason for this is that cd executed without an argument will change directory to the executing user's home directory, which is arguably where the script can do the most damage.

There is one case where your script needs to refer to a relative path, namely if your script is distributed together with other files (e.g. in a zip file) and should be runnable after being unzipped. The script must be able to find files/directories relative to wherever the script resides, so relative to the current working directory does not cut it if the user tries ./zipdir/script.sh. Fortunately, all shells have a $0 that is the path of the script itself. Thus, you do this:

DATADIR=$(dirname $0)/data

While you are avoiding things, also avoid shell expanded glob expressions. They are simply too imprecise and it is too easy to get bitten by paths with space in them. In 99 of 100 cases, find is better than shell glob. find is your friend.

find /dir/with/cruft -mindepth 1 -maxdepth 1 \
     -type f -regex '.*\.sh' | frobnicate_script

There are numerous things going on here. mindepth/maxdepth describes how deep to look; a depth of 0 means the path supplied to find, so 1/1 means "look in the directory /dir/with/cruft". type f means "look only for files". regex uses regular expressions to match files, which is a lot more expressive than globs (normally supplied to find via -name). It is tempting at this point to reach for -exec, but please refrain. The exec option creates code that is very difficult to read.

Instead xargs is the friend of your find. If all you want to do is a simple operation on each found file, you can do something like this:

find $rootdir -mindepth 1 -type d -cmin -10 -print0 | xargs -0 chmod o-rwx

Or, in English: remove world permissions on all directories under $rootdir that I created the last 10 minutes. Find option -print0 will terminate each line with a null character instead of a newline and -0 will tell xargs to ignore whitespace and instead react to null characters. Presto, no problem with space (or even newlines) in paths.

Discouraging users from using delimiter characters (such as space, tab, newline, comma, pipe, et.c.) in file and directory names is part of the holy cause. Nevertheless, some file will have them, sometimes even for good reasons. So your script needs to be prepared for them. Quote all data that comes from outside.

As an aside, it is an annoying aspect of shell scripting that most syntax highlighters will treat e.g. "$1" as a string and give it string colour, but $1 as a variable and give it variable colour. Most unhelpful.

Tags:

Shell scripting patterns: errors and failure

quest — Mon, 04 Jul 2011 05:22:28 +0000

In my opinion, error handling is the area where shell script clearest shows its age. The idea that the shell will by default ignore that commands signal an error state (i.e. a non-zero exit code) seems very strange when viewed through the lens of modern programming theory. Thus, all scripts should start with this instruction, that tells the shell to care about exit codes:

set -e

Now, the following code will no longer result in tears when /destination does not exist:

process < /source/file > /destination/file
rm /source/file

To explicitly ignore the exit code of a command, use:

process < /source/file > /no/such/file || true

... or if you care, but don't want to just bail, you can:

may_explode || result=$?

Just remember that $result will be empty rather than 0 if all goes well, so you cannot do [ $result -gt 0 ].

In rare cases, it may genuinely be that you have a section of instructions that you want to execute even if the previous fail. If so, use a subshell:

(
    set +e
    ...
)

The set operation will only be active in the subshell.

It is quite possible to achieve something akin to modern exception handling in bash shells. Consider the following:

(
    do_this || exit 1
    do_that || exit 2
) || {
    fail=$?
    case $fail in
    1) handle_case_1 ;;
    2) handle_case_2 ;;
    *) exit $fail ;;
    esac
}

Of course, do_this, do_that may already return decent exit codes such that you don't need the '|| exit' bit. More generally, now that set -e is turned on, you need a mental model of execution flow, much like any other programming language. Just remember the law: any context (be it subshell, function or block) has an exit code/return value of the first statement executed that returns a non-zero exit code/return value. Somewhat simplified, as soon as a statement returns non-zero, execution will abandon successive scopes until someone "handles" the non-zero state. The three main ways to "handle" the non-zero state are pipe "|", command substitution "$()", or "||". The background control operator "&" also does not propagate the exit code (it can later be retreived with wait).

It may be tempting to use exit to make life simpler inside functions, but you should resist. exit will abort execution of the current shell, but parentheses and pipes create subshells, so a function cannot know whether it will abort the master shell process or just a subshell. Use only return statements in functions and save exit for top-level code. exit should only be needed in top-level code to tell the caller that parameters or input were invalid.

Tags:

Shell scripting patterns: returning from functions

quest — Tue, 28 Jun 2011 22:12:51 +0000

One shortcoming of shell scripting is the inability to return anything of significance from a shell script function. Consider: get a function that returns the youngest file in a directory. The basic moving part in this is ls -tr /da/dir | tail -1. Abstracting this to a function seem problematic, given that we cannot return a value. However, functions is very similar to external commands in bash, so you can do this:

functions latest_file() {
  ls -tr "$1" | tail -1
}

... and then simply invoke it thus:

latest=$(latest_file /da/dir)

From this point, we can write what effectively is a generator:

function find_magic_scripts() {
    find $1 -type f -name '*.sh' | while read fname ; do
        grep -l 'magic word' $fname || true
    done
}

We can now use it as a source for a consumer:

find_magic_scripts /da/dir | link_and_version

These two processes will be run in parallel, which is good since they both take a little time. Of course, find_magic_scripts will still block as soon as it has output a file, but will restart as soon as link_and_version has read that line.

However.

There is an issue with this pattern: a sequence of pipes will have the error code for the last command, ignoring any previous failures. So, if /da/dir does not exist, link_and_version will simply get EOF and be happy. Unfortunately, there is no neat workaround that solves all problems. The best strategy is usually to introduce an explicit failure marker in the protocol over the pipe, something like:

function foo() {
    {
        ...
    } || echo "FAIL $?"
}
foo | bar

Or, where left side is an external command, simply:

{ find /may/not/exist -name '*.sh' || echo "FAIL $?" ; } | bar

Now bar() can know that something went wrong and abort.

Tags:

Shell scripting patterns - a personal perspective

quest — Sat, 25 Jun 2011 11:15:00 +0000

It is often said that no serious development project should ever use shell script.

While it is certainly true that there are some serious shortcomings is the bash strain of shells, this is not to say that these shortcomings cannot be overcome. Like so much else in programming, it is mostly a matter of finding good patterns that contain those shortcomings.

I want to write something on this matter, so I'm planning a series of blog posts on important patterns that help in keeping bash shell scripts readable and maintainable while keeping a reasonable performance. With luck, they will have something for you as well.

Tags: