{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Shell Scripts" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Variables\n", " - `${var}`\n", " - `${var/pattern/replacement}`\n", " - Quoting\n", " - Command substitution" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "echo $PWD" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Match and discard the longest pattern from the front." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "echo ${PWD##/*/}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Match and discard the shortest pattern from the front." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "echo ${PWD#/*/}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Match and discard the longest pattern from the back." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "echo ${PWD%%/*}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Match and discard the shortest pattern from the back." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "echo ${PWD%/*}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Another way to get `basename`" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "echo ${PWD##/*/}" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "basename $PWD" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Another way to get `dirname`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "echo ${PWD%/*}" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "dirname $PWD" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Simple substitution." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "echo ${PWD/2017/2018}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Single quote does not expand variables." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "echo '${PWD}/foo.txt'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Double quotes expand variables." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "echo \"${PWD}/foo.txt\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Command substitution with `S(command)`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "echo $(printf \"%5.2f%%\" 3.14)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Another command substitution." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "echo $(ls -l)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Functions\n", " - Special variables\n", " - `$1, $2, $*, $@, $#`\n", " - `$?`\n", " - Writing a function" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are two equivalent ways to write a function.\n", "\n", "We show one way below. \n", "\n", "Note the use of positional arguments `$1` and $2` to capture function inputs." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "add_int () {\n", " echo $(($1 + $2))\n", "}" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "add_int 1 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here's the other way.\n", "\n", "Note also that `$` is optional within `$((arithmetic expression))`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "function add_int {\n", " echo $((1 + 2))\n", "}" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "add_int 1 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In addition to the positional arguments `$1`, `$2`, ..., you can also show all arguments with `$@`" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "show_args () {\n", " echo $@\n", "}" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "show_args one two three four" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `shift` command lets you skip arguments. \n", "\n", "This is often useful when the function has some arguments that set up the context, and others that are then passed on as real command arguments." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "show_args_with_shift () {\n", " ARG1=$1\n", " ARG2=$2\n", " printf \"There are %d arguments before shirt\\n\" $#\n", " shift 2\n", " echo $@ $ARG1 $ARG2\n", " printf \"There are %d arguments after shift\\n\" $#\n", "}" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "show_args_with_shift one two three four" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Shell scripts\n", " - Writing a script\n", " - Sourcing a scirpt\n", " - `#!` magic\n", " - `chmod +x`\n", " - Where does Unix look for executable programs?\n", " - The `PATH` environment variable" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Writing a shell script typically involves the following steps.\n", "\n", "- Write the script\n", "- Add a `#!/path_to_interpreter`\n", "- Give execute permission with `chmod +x`\n", "- Put in path in $PATH variable\n", "\n", "Note: Steps 2-4 are optional if you don't mind using `bash `." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "cat > script1.sh < script2.sh <<'EOF'\n", "#!/bin/bash\n", "\n", "echo $ANSWER\n", "EOF" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Regular variables are not visible within a script. Only environment variables are." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "ANSWER=42" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "bash script2.sh" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can make a variable visible to a script by prefixing it on the same line as the call." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "ANSWER=42 bash script2.sh" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or you can create an environment variable with the `export` statement." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "export ANSWER=42\n", "bash script2.sh" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Control flow\n", " - Return values\n", " - Conditional expressions\n", " - Logical operators `&&`, `||`, `!`, `-a`, `-o`\n", " - Arithmetic tests wiht `(( ))`\n", " \n", " - `if`\n", " - `for`\n", " - `while`\n", " - `until`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The last evaluation outcome is stored in `$_`. A successful outcome is indicated with 0. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "echo $_" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "ls non_existent_direcotry" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "echo $_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Conditionals test for a successful outcome, that is, 0." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "if [[ 0 ]]\n", "then\n", " echo YES\n", "fi" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For example, a common test is to check for file existence." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "if [[ -e UnixShell01.ipynb ]]\n", "then\n", " echo YES\n", "fi" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "if [[ -e foo.txt ]]\n", "then\n", " echo YES\n", "fi" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "touch foo.txt" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "if [[ -e foo.txt ]]\n", "then\n", " echo YES\n", "fi" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "rm foo.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Some common file test operators\n", "\n", "| Operator | Meaning |\n", "| --- | --- |\n", "| -e | exists |\n", "| -s | exists and has non-zero size |\n", "| -f | is a regular file |\n", "| -d | is a directory |\n", "| -b | is a block device | \n", "| -r | has read permission |\n", "| -w | has write permissions |\n", "| -x | has execute permissions |\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Use `(( arithmetic expression ))` to evaluate arithmetic expressions." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "if (( (2 + 3 > 6) && (2 + 3 > 4) ))\n", "then\n", " echo YES\n", "fi" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "if (( (2 + 3 > 6) || (2 + 3 > 4) ))\n", "then\n", " echo YES\n", "fi" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Date formatting." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "date +\"%d%b%y_%H:%M:%S\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check out \n", "\n", "```bash\n", "man strftime\n", "```\n", "\n", "to see the full list of formatting options." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using a **for** loop." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "for FILE in $(ls *ipynb)\n", "do\n", " echo \"Old name\" $FILE\n", " NAME=${FILE/UnixShell/Bash}\n", " echo \"New name\" $NAME\n", " echo \"EXAMPLE: mv ${FILE} ${NAME}\"\n", " echo\n", "done" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "for IDX in $(seq 5) \n", "do\n", " TIMESTAMP=$(date +\"%d%b%y_%H:%M:%S\")\n", " sleep 1\n", " FILE=$(printf \"Experiment_%03d_%s.txt\" $IDX $TIMESTAMP)\n", " echo $FILE\n", "done" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This form of the **for** loop should be familiar to C, C++ and Java developers." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "for (( IDX=1 ; IDX <= 5; IDX++))\n", "do\n", " echo $IDX\n", "done" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The **while** loop. \n", "\n", "While and until loops are prone to infinite looping. It is usually a good idea to have a loop counter and exit after it reaches a maximum count during development." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "COUNT=1\n", "while (( $COUNT <= 5))\n", "do\n", " echo $COUNT\n", " ((COUNT++))\n", "done" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "COUNT=5\n", "until (( $COUNT == 0))\n", "do\n", " echo $COUNT\n", " ((COUNT--))\n", "done" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The **case** statement." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "for FILE in $(ls *)\n", "do\n", " case \"${FILE}\" in\n", " *.ipynb) echo \"Motebook \" ${FILE} ;;\n", " *.sh ) echo \"Script \" ${FILE} ;;\n", " * ) echo \"Unknown \" ${FILE};;\n", " esac\n", "done" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Array variables." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "names=(Ann Bob Charles)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "for (( i=0 ; i < ${#names[@]} ; i++ ))\n", "do\n", " echo ${names[$i]}\n", "done" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "cat > s4.sh <<'EOF'\n", "#!/bin/bash\n", "while read -r LINE\n", "do\n", " echo $LINE | tr a-z A-Z \n", "done\n", "EOF" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "cat s4.sh | bash s4.sh " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Process management\n", " - Foregrround and background processes\n", " - `jobs`\n", " - `ps`\n", " - Signals\n", " - `kill`\n", " - Coroutines and `wait`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check for your own locally running processes with `ps`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "ps | head -5" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check for all locally running processes with `ps -a`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "ps -a | head -5" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check for all running processes with `ps -e`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true, "scrolled": false }, "outputs": [], "source": [ "ps -e | head -5" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create to do-nothing scripts." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "cat > s5.sh <<'EOF'\n", "for ((;;))\n", "do\n", " sleep 1\n", "done\n", "EOF" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "cat > s6.sh <<'EOF'\n", "for ((;;))\n", "do\n", " sleep 1\n", "done\n", "EOF" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Set them both running as background processes. \n", "\n", "This is a simple way to achieve parallelism in `bashh`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "bash s5.sh &" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "bash s6.sh &" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check what jobs are running." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "jobs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Kill using job ID." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "kill %1" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "jobs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Show running jobs and process ID." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "jobs -l" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Kill using the process ID." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "kill " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Confirm that we have no more running jobs." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "jobs -l" ] } ], "metadata": { "kernelspec": { "display_name": "Bash", "language": "bash", "name": "bash" }, "language_info": { "codemirror_mode": "shell", "file_extension": ".sh", "mimetype": "text/x-sh", "name": "bash" } }, "nbformat": 4, "nbformat_minor": 2 }