Simple regression testing using zsh script

From raju

Task

I want to add new functionality to an existing script without introducing any "regressions", i.e. without breaking any existing functionality.

Solution

Here is a simple way to do it by using a zsh script.

% cat regr_test.zsh
#! /bin/zsh

# set -x
set -u

run() {
    local script=$1
    local file=$2
    local prefix=$3
    local odir=$4
    local id=$5
    $script $file $prefix > $odir/${id}.txt
}

# using relative path names. But can use absolute path names if necessary.
script_new=get_name.pl
script_old=get_name_old.pl
printf "performing regression test for\n"
printf "    $script_new vs. $script_old\n"

# using relative path names. But can use absolute path names if necessary.
odir_new=1
odir_old=2
mkdir -p $odir_new $odir_old
printf "output is stored in $odir_new , $odir_old\n"

test_arg=(
    "/usr/bin/ls"
    "$HOME/.vimrc"
    )

test_name=(
    "abs_path"
    "dot_file"
    )

prefix="RK_"

ntest=${#test_arg};

for i in `seq 1 $ntest`
do
    run $script_new ${test_arg[$i]} $prefix $odir_new ${test_name[$i]}
    run $script_old ${test_arg[$i]} $prefix $odir_old ${test_name[$i]}
done

diff -rq $odir_new $odir_old > /dev/null
dret=$?
if [ $dret -eq 0 ]; then
    printf "\nSuccess! No differences are found between legs $odir_new and $odir_old\n"
else
    printf "\nFailed! Differences found between legs $odir_new and $odir_old\n"
    diff -rq $odir_new $odir_old
fi

where get_name.pl and get_name_old.pl are new and old versions of a script that tries to add a prefix to a given file name.

% cat get_name.pl
#! /usr/bin/perl -w

use strict;
use warnings;
use File::Spec;  # for rel2abs
use File::Basename qw(dirname);

my $file_arg=$ARGV[0];
my $prefix=$ARGV[1];

my $file_abs = File::Spec->rel2abs( $file_arg );
(my $base_name = $file_abs) =~ s!.*/!!;
my $dir = dirname($file_abs);

my $new_file = "$dir/$base_name$prefix";
print "$new_file\n";
% cat get_name_old.pl
#! /usr/bin/perl -w

use strict;
use warnings;
use File::Spec;  # for rel2abs
use File::Basename qw(dirname);

my $file_arg=$ARGV[0];
my $prefix=$ARGV[1];

my $file_abs = File::Spec->rel2abs( $file_arg );
(my $base_name = $file_abs) =~ s!.*/!!;
my $dir = dirname($file_abs);

my $new_file = "$dir/$prefix$base_name";
print "$new_file\n";

Here, I purposefully introduced a bug in the new version of the script which appends the prefix instead of prepending.

% diff get_name.pl get_name_old.pl
15c15
< my $new_file = "$dir/$base_name$prefix";
---
> my $new_file = "$dir/$prefix$base_name";

The idea is to make sure that the regression test catches it.

First make all the scripts executable

% chmod +x get_name.pl get_name_old.pl regr_test.zsh

Run the regression test

% ./regr_test.zsh
performing regression test for
    get_name.pl vs. get_name_old.pl
output is stored in 1 , 2

Failed! Differences found between legs 1 and 2
Files 1/abs_path.txt and 2/abs_path.txt differ
Files 1/dot_file.txt and 2/dot_file.txt differ

Here is the directory structure once the script is run.

% tree
.
|-- 1
|   |-- abs_path.txt
|   `-- dot_file.txt
|-- 2
|   |-- abs_path.txt
|   `-- dot_file.txt
|-- get_name_old.pl
|-- get_name.pl
`-- regr_test.zsh

2 directories, 7 files

Looking at the output files, it becomes obvious as to what went wrong

% head [12]/*
==> 1/abs_path.txt <==
/usr/bin/lsRK_

==> 1/dot_file.txt <==
/home/rajulocal/.vimrcRK_

==> 2/abs_path.txt <==
/usr/bin/RK_ls

==> 2/dot_file.txt <==
/home/rajulocal/RK_.vimrc

Once the bug is corrected, a successful run will look like

% ./regr_test.zsh
performing regression test for
    get_name.pl vs. get_name_old.pl
output is stored in 1 , 2

Success! No differences are found between legs 1 and 2

Side Effects

This page also demonstrates how to:

  • pass arguments to functions in zsh shell scripts

Related links