Bash notes

From raju

Note:- This page will be migrated to http://www.kamaraju.xyz/dk/bash_notes. Once the migration is done, it will be deleted.

Scripts

Check for exact number of arguments

    $cat check_arg_num.sh
    #! /usr/bin/env bash
    set -e
    set -u
    if [ "$#" -ne 1 ]; then
      echo "Usage: $0 foo" >&2
      exit 1
    fi
    foo=$1
    echo $foo
    

Sample run without an argument

    $./check_arg_num.sh
    Usage: ./check_arg_num.sh foo
    

Sample run with an argument

    $./check_arg_num.sh something
    something
    

Snippet to echo multiline message:

    if [ "$#" -ne 1 ]; then
      echo "
    Usage: $0 foo_date
    where foo_date is of the form YYYYMMDD" >&2
      exit 1
    fi
    foo_date="$1"
    

check that number of arguments are in a range

    $ cat check_arg_range.sh
    #! /usr/bin/env bash
    set -e
    set -u
    if [ "$#" -lt 1 ] || [ "$#" -gt 2 ]; then
      echo "Usage: $0 foo <bar>" >&2
      echo "Number of arguments should be between 1 and 2"
      exit 1
    fi
    foo=$1
    bar=${2:-default}
    echo $foo
    echo $bar
    
    $ ./check_arg_range.sh
    Usage: ./check_arg_range.sh foo <bar>
    Number of arguments should be between 1 and 2
    
    $ ./check_arg_range.sh valA
    valA
    default
    
    $ ./check_arg_range.sh valA valB
    valA
    valB
    
    $ ./check_arg_range.sh valA valB valC
    Usage: ./check_arg_range.sh foo <bar>
    Number of arguments should be between 1 and 2
    

file extension

To get the extension

    echo ${filename##*.}
    

To get the last three characters in a filename

    echo ${filename: -3}
    

Sample use case:

    $ filename="d u m m y.b l a h"
    
    $ echo ${filename##*.}
    b l a h
    
    $ echo ${filename: -3}
    a h
    

characteristics of the input: Contains spaces in both file name and in extension, extension is not three characters.

full name, file name, file head, extension

    for fullname in /path/to/*.csv
    do
    fname="${fullname##*/}"
    fhead="${fname%.*}"
    ext="${fname##*.}"
    echo $fullname $fname $fhead $ext
    done
    

Ref:- https://stackoverflow.com/questions/965053/extract-filename-and-extension-in-bash

replace string

    $a="space_kama_raju_kama_raju_kama_raju"
    
    $echo $a
    space_kama_raju_kama_raju_kama_raju
    
    $echo ${a/kama/foo}
    space_foo_raju_kama_raju_kama_raju
    
    
    $echo ${a//kama/foo}
    space_foo_raju_foo_raju_foo_raju
    

looping on the command line

    for i in foo.csv bar.csv; do echo "==>" $i "<=="; cat $i; echo ; done
    

default arguments

The idea here is to have some default values assigned to a variable if nothing is specified on the command line. If some values are specified on the command line, they will be used instead.

    $cat default_args.sh
    #! /usr/bin/env bash
    
    if [ "$#" -eq 0 ]; then
        dirs=(
            "path1"
            "path2"
            )
    else
        dirs="$@"
    fi
    
    for d in ${dirs[@]}
    do
        echo $d
    done
    
    $./default_args.sh
    path1
    path2
    
    $./default_args.sh foo bar
    foo
    bar
    

Used in git_status.sh to get the list of directories on which we want to run "git status" type of commands.

Demonstrates | check number of arguments in a bash script

code snippets

check if a variable is empty

    if [ -z "${BASE}" ]; then
        # do something
    fi
    

check for two conditions inside if

This checks if BASE variable is empty or if LOCAL and BASE are the same.

    if [ -z $BASE ] || [ $LOCAL = $BASE ]; then
        # do something
    fi
    

Check if a directory exists

    if [ -d "$directory" ]; then
        echo "$directory exists"
    else
        echo "$directory does not exist"
    fi
    

To check if a directory does not exist, use

    if [ ! -d "$directory" ]; then
        echo "$directory does not exist"
    fi
    

compare numbers

Sample code

    # --autostash is available only if git version is >= 2.6
    git_version=`git --version | cut -f3 -d' '`
    major_version=`echo $git_version | cut -f1 -d.`
    minor_version=`echo $git_version | cut -f2 -d.`
    
    if (( $major_version >= 2 )) && (( $minor_version >= 6 )); then
            use_autostash=1
    else
            use_autostash=0
    fi
    
    if [ $use_autostash == 1 ]; then
            git pull --rebase --autostash -v origin
    else
        git stash
        git pull --rebase -v origin
        git stash pop
    fi
    

To compare floating point numbers

    $ [[ 2.5 > 2.10 ]] && echo 1 || echo 0
    1
    $ [[ 2.5 < 2.10 ]] && echo 1 || echo 0
    0
    

It also works for comparing integers.

    $ [[ 3 > 2 ]] && echo 1 || echo 0
    1
    $ [[ 3 < 2 ]] && echo 1 || echo 0
    0
    

-gt, -lt etc., will not work for floating point numbers

    $ [[ 2.5 -gt 2.10 ]] && echo 1 || echo 0
    bash: [[: 2.5: syntax error: invalid arithmetic operator (error token is ".5")
    0
    
    $ [[ 2.5 -lt 2.10 ]] && echo 1 || echo 0
    bash: [[: 2.5: syntax error: invalid arithmetic operator (error token is ".5")
    0
    

But they work for integers

    $ [[ 3 -gt 2 ]] && echo 1 || echo 0
    1
    $ [[ 3 -lt 2 ]] && echo 1 || echo 0
    0
    

tested with

    $ bash --version
    GNU bash, version 4.4.12(1)-release (x86_64-pc-linux-gnu)
    
    $ cat /etc/debian_version
    9.4
    

cd into script directory

    cd `dirname "$0"`
    

which can also be written as

    cd "$(dirname "$0")"
    

source a script from current script's directory

    source "$(dirname "$0")"/foo
    

join two strings to get a path

    full_path="$part1/$part2"
    

check status of previous command

Sample use case

    git rev-parse @ > /dev/null 2>&1
    if [[ $? == 128 ]]; then
        echo "Not a git repository. Exiting."
        exit 1
    fi
    

remove characters from the beginning and end

tags | remove first N characters, remove last N characters

To remove N characters from the beginning and M characters from the end of a string

    ${i:N:-M}
    

Example:

Prepare some data

    mkdir -p ~/x
    cd ~/x
    
    mkdir -p a/.git b/.git c/d/.git
    
    $ for i in $(find . -name .git); do
    > echo $i
    > done
    ./a/.git
    ./b/.git
    ./c/d/.git
    

To remove "./" from the beginning and "/.git" from the end

    $ for i in $(find . -name .git); do
    > echo ${i:2:-5}
    > done
    a
    b
    c/d
    

using while loop to process the output of find

    find blah blah | while read i; do
      echo $i
    done
    

is equivalent to

    for i in $(find blah blah); do
      echo $i
    done
    

For example, to get all the git repos under a directory, we can either do

    find . -name .git -type d -prune | while read d; do
      echo $d
    done
    

or

    for d in $(find . -name .git -type d -prune); do
      echo $d
    done
    

output the current directory name in green

    echo "$(tput setaf 2)$PWD$(tput sgr0)"
    

or more generically

    green=$(tput setaf 2)
    reset=$(tput sgr0)
    echo "$red$PWD$reset"
    


Ref:- https://mywiki.wooledge.org/BashFAQ/037

quoting related

    echo "$@"
    echo "${@:2}"
    cd "$i"
    

parent directory

    dir=/foo/bar
    parentdir="$(dirname "$dir")"
    

increment number in for loop

code snippet

    i=0
    for hash in $hashes
    do
        git checkout $hash -- $file
        mv $file m${i}_$file
        ((i++))
    done
    

Ref:- https://stackoverflow.com/questions/20681210/incrementing-a-variable-inside-a-bash-loop

where I needed it:- https://github.com/KamarajuKusumanchi/rutils/blob/master/bin/xtract_hist_git

Scripting

Difference between $* and $@

formatting if and for statements

    for f in *; do
        if [ -d "$f" ]; then
            # $f is a directory
        fi
    done
    

To do it on one line

    for f in *; do if [ -d "$f" ]; then echo $f; fi done
    

Ref:- https://unix.stackexchange.com/a/86727/198064

sort after

ISO-8601 compliant dates

    $ date +%Y-%m-%d
    2017-10-30
    
    $ date +%Y-%m-%dT%H:%M:%S%z
    2017-10-30T17:44:13-0400
    

You can also achieve this by -I

    $ date -I
    2017-10-30
    
    $ date -Iminutes
    2017-10-30T17:47-04:00
    
    $ date -Iseconds
    2017-10-30T17:47:39-04:00
    

Equivalence between -Iseconds and -u

    $date +%FT%T%z
    2017-10-30T17:53:06-0400
    
    $date -u +%FT%TZ
    2017-10-30T21:53:06Z
    

Basically UTC time + offset printed by %z is equal to the local time.

Ref:-

redirect both stdout and stderr to a file

    foo.sh > out.txt 2>&1
    

To append both stdout and stderr to a file

    foo.sh >> out.txt 2>&1
    

controlling history

Add the following to ~/.bashrc

    # This is useful if you run more than one bash instance in parallel. Without
    # this option, the history file only contains the contents of the last exiting
    # shell. For more details, see
    # https://unix.stackexchange.com/questions/6501/why-would-anyone-not-set-histappend-in-bash
    shopt -s histappend
    
    # prevent duplicate entries of a single session from being saved to $HISTFILE.
    export HISTCONTROL=ignoreboth:erasedups
    
    export HISTSIZE=20000
    
    # Up arrow will show commands from other shells as well
    export PROMPT_COMMAND="history -a;history -c;history -r;$PROMPT_COMMAND"
    
    # When a command is duplicated, store only the latest.
    # Adapted from
    # https://unix.stackexchange.com/questions/48713/how-can-i-remove-duplicates-in-my-bash-history-preserving-order
    tac $HISTFILE | awk '!x[$0]++' | tac > /tmp/tmpfile && "mv" -fv /tmp/tmpfile $HISTFILE
    wc -l $HISTFILE
    

What they do

    * ignoreboth - does both ignorespace and ignoredups
    * erasedups - 
    * ignorespace - ignore commands that start with spaces
    * ignoredups - ignore duplicates
    
    * history -a  # append history lines from this session to the history file.
       
    History file may contain history from other terminals not in this one so:
    
    * history -c # clear [in-memory] history list deleting all of the entries.
    * history -r # read the history file and append the contents to the history list instead.
    

redirect output and errors to /dev/null

Append this to the command

    > /dev/null 2>&1
    

PS1

Some PS1 entries I like

    # host@user:pwd
    # Use $PWD instead of \w so that the home directory is not abbreviated as ~
    PS1="\h@\u:$PWD $"
    

Missing Features

  1. Currently, it is not possible to list files by modification time using tab completion. Use zsh instead.

Ref:- http://lists.gnu.org/archive/html/bug-bash/2014-12/msg00161.html
http://www.zsh.org/mla/users/2015/msg00078.html

Useful links

Documentation links