Shell Scripts

  • Variables
    • ${var}
    • ${var/pattern/replacement}
    • Quoting
    • Command substitution
In [1]:

echo $PWD

/Users/cliburn/_teach/BIO821_2017_notebooks/scratch

Match and discard the longest pattern from the front.

In [2]:

echo ${PWD##/*/}

scratch

Match and discard the shortest pattern from the front.

In [3]:

echo ${PWD#/*/}

cliburn/_teach/BIO821_2017_notebooks/scratch

Match and discard the longest pattern from the back.

In [4]:

echo ${PWD%%/*}

Match and discard the shortest pattern from the back.

In [5]:

echo ${PWD%/*}

/Users/cliburn/_teach/BIO821_2017_notebooks

Another way to get basename

In [6]:

echo ${PWD##/*/}

scratch

In [7]:

basename $PWD

scratch

Another way to get dirname.

In [8]:

echo ${PWD%/*}

/Users/cliburn/_teach/BIO821_2017_notebooks

In [9]:

dirname $PWD

/Users/cliburn/_teach/BIO821_2017_notebooks

Simple substitution.

In [10]:

echo ${PWD/2017/2018}

/Users/cliburn/_teach/BIO821_2018_notebooks/scratch

Single quote does not expand variables.

In [11]:

echo '${PWD}/foo.txt'

${PWD}/foo.txt

Double quotes expand variables.

In [12]:

echo "${PWD}/foo.txt"

/Users/cliburn/_teach/BIO821_2017_notebooks/scratch/foo.txt

Command substitution with S(command).

In [13]:

echo $(printf "%5.2f%%" 3.14)

3.14%

Another command substitution.

In [14]:

echo $(ls -l)

total 5888 -rw-r--r-- 1 cliburn staff 1067008 Oct 30 13:44 Chinook_Sqlite.sqlite -rw-r--r--@ 1 cliburn staff 41276 Sep 19 08:29 Python01.ipynb -rw-r--r--@ 1 cliburn staff 27153 Sep 21 08:09 Python02.ipynb -rw-r--r-- 1 cliburn staff 40464 Sep 21 19:45 Python03.ipynb -rw-r--r-- 1 cliburn staff 13814 Sep 29 09:01 Python04A.ipynb -rw-r--r-- 1 cliburn staff 22925 Sep 29 09:01 Python04B.ipynb -rw-r--r-- 1 cliburn staff 27355 Sep 27 08:54 Python05.ipynb -rw-r--r-- 1 cliburn staff 11068 Oct 3 10:49 Python06.ipynb -rw-r--r-- 1 cliburn staff 34613 Oct 6 17:57 Python06A.ipynb -rw-r--r-- 1 cliburn staff 22266 Oct 6 18:05 Python06B.ipynb -rw-r--r-- 1 cliburn staff 470003 Oct 17 10:31 Python06C.ipynb -rw-r--r-- 1 cliburn staff 24515 Oct 6 19:29 Python07A.ipynb -rw-r--r-- 1 cliburn staff 161864 Oct 7 10:18 Python07B.ipynb -rw-r--r-- 1 cliburn staff 435109 Oct 29 11:50 Python08A.ipynb -rw-r--r-- 1 cliburn staff 173446 Oct 30 13:44 Python09A.ipynb -rw-r--r-- 1 cliburn staff 279122 Nov 2 09:31 Python09B.ipynb -rw-r--r-- 1 cliburn staff 11642 Oct 31 07:16 Python9A_Exercise.ipynb -rw-r--r-- 1 cliburn staff 5929 Oct 31 07:21 Python9B_Exercise.ipynb -rw-r--r-- 1 cliburn staff 3203 Aug 31 19:19 UnixShell01.ipynb -rw-r--r--@ 1 cliburn staff 22561 Nov 2 11:26 UnixShell02.ipynb -rw-r--r--@ 1 cliburn staff 22784 Sep 12 09:33 UnixShell03.ipynb -rw-r--r-- 1 cliburn staff 5117 Sep 12 09:37 UnixShell04.ipynb drwxr-xr-x 3 cliburn staff 102 Oct 23 16:23 __pycache__ -rw-r--r-- 1 cliburn staff 14 Nov 2 11:25 bar.txt -rw-r--r-- 1 cliburn staff 28 Nov 2 11:25 big.txt drwxr-xr-x 3 cliburn staff 102 Oct 17 10:29 figs -rw-r--r-- 1 cliburn staff 139 Nov 2 11:25 humpty.txt -rw-r--r-- 1 cliburn staff 135 Oct 23 16:23 mystery.py -rw-r--r-- 1 cliburn staff 68 Nov 2 11:35 s4.sh -rw-r--r-- 1 cliburn staff 31 Nov 2 11:35 s5.sh -rw-r--r-- 1 cliburn staff 31 Nov 2 11:35 s6.sh -rw-r--r-- 1 cliburn staff 26 Nov 2 11:34 script2.sh -rw-r--r-- 1 cliburn staff 14210 Oct 30 13:44 tips.xlsx
  • Functions
    • Special variables
    • $1, $2, $*, $@, $#
    • $?
    • Writing a function

There are two equivalent ways to write a function.

We show one way below.

Note the use of positional arguments $1 and $2` to capture function inputs.

In [15]:

add_int () {
    echo $(($1 + $2))
}

In [16]:

add_int 1 2

3

Here’s the other way.

Note also that $ is optional within $((arithmetic expression)).

In [17]:

function add_int {
    echo $((1 + 2))
}

In [18]:

add_int 1 2

3

In addition to the positional arguments $1, $2, ..., you can also show all arguments with $@

In [19]:

show_args () {
    echo $@
}

In [20]:

show_args one two three four

one two three four

The shift command lets you skip arguments.

This is often useful when the function has some arguments that set up the context, and others that are then passed on as real command arguments.

In [21]:

show_args_with_shift () {
    ARG1=$1
    ARG2=$2
    printf "There are %d arguments before shirt\n" $#
    shift 2
    echo $@ $ARG1 $ARG2
    printf "There are %d arguments after shift\n" $#
}

In [22]:

show_args_with_shift one two three four

There are 4 arguments before shirt
three four one two
There are 2 arguments after shift
  • Shell scripts
    • Writing a script
    • Sourcing a scirpt
    • #! magic
    • chmod +x
    • Where does Unix look for executable programs?
    • The PATH environment variable

Writing a shell script typically involves the following steps.

  • Write the script
  • Add a #!/path_to_interpreter
  • Give execute permission with chmod +x
  • Put in path in $PATH variable

Note: Steps 2-4 are optional if you don’t mind using bash <script_name>.

In [23]:

cat > script1.sh <<EOF
#!/bin/bash

echo "My first script"
EOF

In [24]:

bash script1.sh

My first script

In [25]:

./script1.sh

bash: ./script1.sh: Permission denied



In [26]:

ls -l script1.sh

-rw-r--r--  1 cliburn  staff  36 Nov  2 11:38 script1.sh

In [27]:

chmod +x script1.sh

In [28]:

ls -l script1.sh

-rwxr-xr-x  1 cliburn  staff  36 Nov  2 11:38 script1.sh

In [29]:

./script1.sh

My first script

In [30]:

script1.sh

My first script

In [31]:

export PATH=$PATH:$PWD

In [32]:

script1.sh

My first script

In [33]:

rm script1.sh

In [34]:

cat > script2.sh <<'EOF'
#!/bin/bash

echo $ANSWER
EOF

Regular variables are not visible within a script. Only environment variables are.

In [35]:

ANSWER=42

In [36]:

bash script2.sh

You can make a variable visible to a script by prefixing it on the same line as the call.

In [37]:

ANSWER=42 bash script2.sh

42

Or you can create an environment variable with the export statement.

In [38]:

export ANSWER=42
bash script2.sh

42
  • Control flow
    • Return values
    • Conditional expressions
      • Logical operators &&, ||, !, -a, -o
      • Arithmetic tests wiht (( ))
    • if
    • for
    • while
    • until

The last evaluation outcome is stored in $_. A successful outcome is indicated with 0.

In [39]:

echo $_

0

In [40]:

ls non_existent_direcotry

ls: non_existent_direcotry: No such file or directory



In [41]:

echo $_

1

Conditionals test for a successful outcome, that is, 0.

In [42]:

if [[ 0 ]]
then
    echo YES
fi

YES

For example, a common test is to check for file existence.

In [43]:

if [[ -e UnixShell01.ipynb ]]
then
    echo YES
fi

YES

In [44]:

if [[ -e foo.txt ]]
then
    echo YES
fi

In [45]:

touch foo.txt

In [46]:

if [[ -e foo.txt ]]
then
    echo YES
fi

YES

In [47]:

rm foo.txt

Some common file test operators

Operator Meaning
-e exists
-s exists and has non-zero size
-f is a regular file
-d is a directory
-b is a block device
-r has read permission
-w has write permissions
-x has execute permissions

Use (( arithmetic expression )) to evaluate arithmetic expressions.

In [48]:

if (( (2 + 3 > 6) && (2 + 3 > 4) ))
then
    echo YES
fi

In [49]:

if (( (2 + 3 > 6) || (2 + 3 > 4) ))
then
    echo YES
fi

YES

Date formatting.

In [50]:

date +"%d%b%y_%H:%M:%S"

02Nov17_11:38:27

Check out

man strftime

to see the full list of formatting options.

Using a for loop.

In [51]:

for FILE in $(ls *ipynb)
do
    echo "Old name" $FILE
    NAME=${FILE/UnixShell/Bash}
    echo "New name" $NAME
    echo "EXAMPLE: mv ${FILE} ${NAME}"
    echo
done

Old name Python01.ipynb
New name Python01.ipynb
EXAMPLE: mv Python01.ipynb Python01.ipynb

Old name Python02.ipynb
New name Python02.ipynb
EXAMPLE: mv Python02.ipynb Python02.ipynb

Old name Python03.ipynb
New name Python03.ipynb
EXAMPLE: mv Python03.ipynb Python03.ipynb

Old name Python04A.ipynb
New name Python04A.ipynb
EXAMPLE: mv Python04A.ipynb Python04A.ipynb

Old name Python04B.ipynb
New name Python04B.ipynb
EXAMPLE: mv Python04B.ipynb Python04B.ipynb

Old name Python05.ipynb
New name Python05.ipynb
EXAMPLE: mv Python05.ipynb Python05.ipynb

Old name Python06.ipynb
New name Python06.ipynb
EXAMPLE: mv Python06.ipynb Python06.ipynb

Old name Python06A.ipynb
New name Python06A.ipynb
EXAMPLE: mv Python06A.ipynb Python06A.ipynb

Old name Python06B.ipynb
New name Python06B.ipynb
EXAMPLE: mv Python06B.ipynb Python06B.ipynb

Old name Python06C.ipynb
New name Python06C.ipynb
EXAMPLE: mv Python06C.ipynb Python06C.ipynb

Old name Python07A.ipynb
New name Python07A.ipynb
EXAMPLE: mv Python07A.ipynb Python07A.ipynb

Old name Python07B.ipynb
New name Python07B.ipynb
EXAMPLE: mv Python07B.ipynb Python07B.ipynb

Old name Python08A.ipynb
New name Python08A.ipynb
EXAMPLE: mv Python08A.ipynb Python08A.ipynb

Old name Python09A.ipynb
New name Python09A.ipynb
EXAMPLE: mv Python09A.ipynb Python09A.ipynb

Old name Python09B.ipynb
New name Python09B.ipynb
EXAMPLE: mv Python09B.ipynb Python09B.ipynb

Old name Python9A_Exercise.ipynb
New name Python9A_Exercise.ipynb
EXAMPLE: mv Python9A_Exercise.ipynb Python9A_Exercise.ipynb

Old name Python9B_Exercise.ipynb
New name Python9B_Exercise.ipynb
EXAMPLE: mv Python9B_Exercise.ipynb Python9B_Exercise.ipynb

Old name UnixShell01.ipynb
New name Bash01.ipynb
EXAMPLE: mv UnixShell01.ipynb Bash01.ipynb

Old name UnixShell02.ipynb
New name Bash02.ipynb
EXAMPLE: mv UnixShell02.ipynb Bash02.ipynb

Old name UnixShell03.ipynb
New name Bash03.ipynb
EXAMPLE: mv UnixShell03.ipynb Bash03.ipynb

Old name UnixShell04.ipynb
New name Bash04.ipynb
EXAMPLE: mv UnixShell04.ipynb Bash04.ipynb


In [52]:

for IDX in $(seq 5)
do
    TIMESTAMP=$(date +"%d%b%y_%H:%M:%S")
    sleep 1
    FILE=$(printf "Experiment_%03d_%s.txt" $IDX $TIMESTAMP)
    echo $FILE
done

Experiment_001_02Nov17_11:38:28.txt
Experiment_002_02Nov17_11:38:29.txt
Experiment_003_02Nov17_11:38:30.txt
Experiment_004_02Nov17_11:38:31.txt
Experiment_005_02Nov17_11:38:32.txt

This form of the for loop should be familiar to C, C++ and Java developers.

In [53]:

for (( IDX=1 ; IDX <= 5; IDX++))
do
    echo  $IDX
done

1
2
3
4
5

The while loop.

While and until loops are prone to infinite looping. It is usually a good idea to have a loop counter and exit after it reaches a maximum count during development.

In [54]:

COUNT=1
while (( $COUNT <= 5))
do
    echo $COUNT
    ((COUNT++))
done

1
2
3
4
5

In [55]:

COUNT=5
until (( $COUNT == 0))
do
    echo $COUNT
    ((COUNT--))
done

5
4
3
2
1

The case statement.

In [56]:

for FILE in $(ls *)
do
    case "${FILE}" in
        *.ipynb) echo "Motebook  " ${FILE} ;;
        *.sh   ) echo "Script    " ${FILE} ;;
        *      ) echo  "Unknown  " ${FILE};;
    esac
done

Unknown   Chinook_Sqlite.sqlite
Motebook   Python01.ipynb
Motebook   Python02.ipynb
Motebook   Python03.ipynb
Motebook   Python04A.ipynb
Motebook   Python04B.ipynb
Motebook   Python05.ipynb
Motebook   Python06.ipynb
Motebook   Python06A.ipynb
Motebook   Python06B.ipynb
Motebook   Python06C.ipynb
Motebook   Python07A.ipynb
Motebook   Python07B.ipynb
Motebook   Python08A.ipynb
Motebook   Python09A.ipynb
Motebook   Python09B.ipynb
Motebook   Python9A_Exercise.ipynb
Motebook   Python9B_Exercise.ipynb
Motebook   UnixShell01.ipynb
Motebook   UnixShell02.ipynb
Motebook   UnixShell03.ipynb
Motebook   UnixShell04.ipynb
Unknown   bar.txt
Unknown   big.txt
Unknown   humpty.txt
Unknown   mystery.py
Script     s4.sh
Script     s5.sh
Script     s6.sh
Script     script2.sh
Unknown   tips.xlsx
Unknown   __pycache__:
Unknown   mystery.cpython-36.pyc
Unknown   figs:
Unknown   face.png

Array variables.

In [57]:

names=(Ann Bob Charles)

In [58]:

for (( i=0 ; i < ${#names[@]} ; i++ ))
do
    echo ${names[$i]}
done

Ann
Bob
Charles

In [59]:

cat > s4.sh <<'EOF'
#!/bin/bash
while read -r LINE
do
    echo $LINE | tr a-z A-Z
done
EOF

In [60]:

cat s4.sh | bash s4.sh

#!/BIN/BASH
WHILE READ -R LINE
DO
ECHO $LINE | TR A-Z A-Z
DONE
  • Process management
    • Foregrround and background processes
    • jobs
    • ps
    • Signals
    • kill
    • Coroutines and wait

Check for your own locally running processes with ps.

In [61]:

ps | head -5

  PID TTY           TIME CMD
 8180 ttys000    0:00.25 -bash
15485 ttys000    0:04.69 /usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python /usr/local/bin/jupyter-notebook
15405 ttys001    0:00.02 -bash
15600 ttys002    0:00.07 -bash

Check for all locally running processes with ps -a.

In [62]:

ps -a | head -5

  PID TTY           TIME CMD
 8179 ttys000    0:00.04 login -pf cliburn
 8180 ttys000    0:00.25 -bash
15485 ttys000    0:04.69 /usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python /usr/local/bin/jupyter-notebook
15402 ttys001    0:00.03 login -pfl cliburn /bin/bash -c exec -la bash /bin/bash

Check for all running processes with ps -e.

In [63]:

ps -e | head -5

  PID TTY           TIME CMD
    1 ??        15:32.63 /sbin/launchd
   50 ??         0:35.96 /usr/libexec/UserEventAgent (System)
   51 ??         0:13.69 /usr/sbin/syslogd
   53 ??         0:09.96 /System/Library/PrivateFrameworks/Uninstall.framework/Resources/uninstalld

Create to do-nothing scripts.

In [64]:

cat > s5.sh <<'EOF'
for ((;;))
do
    sleep 1
done
EOF

In [65]:

cat > s6.sh <<'EOF'
for ((;;))
do
    sleep 1
done
EOF

Set them both running as background processes.

This is a simple way to achieve parallelism in bashh.

In [66]:

bash s5.sh &

[1] 21530

In [67]:

bash s6.sh &

[2] 21532

Check what jobs are running.

In [68]:

jobs

[1]-  Running                 bash s5.sh &
[2]+  Running                 bash s6.sh &

Kill using job ID.

In [69]:

kill %1



In [70]:

jobs

[2]+  Running                 bash s6.sh &

Show running jobs and process ID.

In [71]:

jobs -l

[2]+ 21532 Running                 bash s6.sh &

Kill using the process ID.

In [72]:

kill <PID>

bash: syntax error near unexpected token `newline'

Confirm that we have no more running jobs.

In [73]:

jobs -l

[2]+ 21532 Running                 bash s6.sh &