Text and arithmetic in bash

  • Working with text streams
    • stdin, stdout, sterr
    • >, >>, <
    • |
    • echo
    • cat
    • HEREDOCS
    • printf
    • cut
    • sort
    • uniq
    • brace expansions
In [1]:

echo 'Hello' > foo.txt

In [2]:

echo 'Goodbye' >> foo.txt

In [3]:

cat foo.txt

Hello
Goodbye

In [4]:

cat foo.txt

Hello
Goodbye

In [5]:

cat < foo.txt

Hello
Goodbye

In [6]:

cat > bar.txt <<EOF
One
Two
Three
EOF

In [7]:

cat bar.txt

One
Two
Three

In [ ]:

cat > bar.txt <<EOF
One
Two
Three
EOF |

In [9]:

head -2 bar.txt | tail -1

Two

In [10]:

printf 'Hello %s again' world

Hello world again

In [11]:

printf '%d inch is %8.2f centimeters' 1 2.54

1 inch is     2.54 centimeters

In [12]:

printf '%d inch is %-8.2f centimeters' 1 2.54

1 inch is 2.54     centimeters

In [13]:

FEET=5
INCH=1
CM_IN_FEET=30
CM_IN_INCH=2.54
RESULT=$(echo "$FEET * $CM_IN_FEET + $INCH * $CM_IN_INCH"  | bc)
printf '%d feet %d inches is %.1f centimeters' $FEET $INCH $RESULT

5 feet 1 inches is 152.5 centimeters

In [14]:

cat /etc/passwd | head -3

##
# User Database
#

In [15]:

cut -f1 -d: /etc/passwd | head -3

##
# User Database
#

In [16]:

cut -f2 -d/ /etc/passwd | head -3

##
# User Database
#

In [17]:

cut -c3-5 /etc/passwd | head -3


Use


In [18]:

cut -f1 -d: /etc/passwd | sort | head -5

#
#
#
# Note that this file is consulted directly only when the system is running
# Open Directory.

In [19]:

cut -f1 -d: /etc/passwd | sort -r | head -5

root
nobody
daemon
_xgridcontroller
_xgridagent

In [20]:

cut -f3 -d: /etc/passwd | head -5

##
# User Database
#
# Note that this file is consulted directly only when the system is running
# in single-user mode.  At other times this information is provided by

In [21]:

cut -f3 -d: /etc/passwd | sort -r | head -5

99
98
97
96
95

In [22]:

cut -f3 -d: /etc/passwd | sort -nr | head -5

214
213
212
211
210

In [23]:

ls -l * | sort -rn -k 5

-rw-r--r--  1 cliburn  staff  1207075 Oct 17 10:28 face.png
-rw-r--r--  1 cliburn  staff  1067008 Oct 30 13:44 Chinook_Sqlite.sqlite
-rw-r--r--  1 cliburn  staff   470003 Oct 17 10:31 Python06C.ipynb
-rw-r--r--  1 cliburn  staff   435109 Oct 29 11:50 Python08A.ipynb
-rw-r--r--  1 cliburn  staff   279122 Nov  2 09:31 Python09B.ipynb
-rw-r--r--  1 cliburn  staff   173446 Oct 30 13:44 Python09A.ipynb
-rw-r--r--  1 cliburn  staff   161864 Oct  7 10:18 Python07B.ipynb
-rw-r--r--@ 1 cliburn  staff    41276 Sep 19 08:29 Python01.ipynb
-rw-r--r--  1 cliburn  staff    40464 Sep 21 19:45 Python03.ipynb
-rw-r--r--  1 cliburn  staff    34613 Oct  6 17:57 Python06A.ipynb
-rw-r--r--  1 cliburn  staff    27355 Sep 27 08:54 Python05.ipynb
-rw-r--r--@ 1 cliburn  staff    27153 Sep 21 08:09 Python02.ipynb
-rw-r--r--  1 cliburn  staff    24515 Oct  6 19:29 Python07A.ipynb
-rw-r--r--  1 cliburn  staff    22925 Sep 29 09:01 Python04B.ipynb
-rw-r--r--@ 1 cliburn  staff    22784 Sep 12 09:33 UnixShell03.ipynb
-rw-r--r--  1 cliburn  staff    22266 Oct  6 18:05 Python06B.ipynb
-rw-r--r--  1 cliburn  staff    14210 Oct 30 13:44 tips.xlsx
-rw-r--r--  1 cliburn  staff    13814 Sep 29 09:01 Python04A.ipynb
-rw-r--r--@ 1 cliburn  staff    12974 Sep  5 09:03 UnixShell02.ipynb
-rw-r--r--  1 cliburn  staff    11642 Oct 31 07:16 Python9A_Exercise.ipynb
-rw-r--r--  1 cliburn  staff    11068 Oct  3 10:49 Python06.ipynb
-rw-r--r--  1 cliburn  staff     5929 Oct 31 07:21 Python9B_Exercise.ipynb
-rw-r--r--  1 cliburn  staff     5117 Sep 12 09:37 UnixShell04.ipynb
-rw-r--r--  1 cliburn  staff     3203 Aug 31 19:19 UnixShell01.ipynb
-rw-r--r--  1 cliburn  staff  361 Oct 23 16:23 mystery.cpython-36.pyc
-rw-r--r--  1 cliburn  staff      135 Oct 23 16:23 mystery.py
-rw-r--r--  1 cliburn  staff       14 Nov  2 11:25 foo.txt
-rw-r--r--  1 cliburn  staff       14 Nov  2 11:25 bar.txt
total 8
total 2360
figs:
__pycache__:



In [24]:

echo '3 2 1 2 3 1 4 5 5' | tr ' ' '\n'  | sort -n | uniq

1
2
3
4
5

In [25]:

echo ba{r,z}.txt

bar.txt baz.txt

In [26]:

ls -l ba{r,z}.txt

ls: baz.txt: No such file or directory
-rw-r--r--  1 cliburn  staff  14 Nov  2 11:25 bar.txt
  • Regular expressions
    • patterns
    • direct match
    • alternatives
    • character sets
    • named character sets
    • word boundaries
    • numeric qualifiers
    • capture groups
In [27]:

[[ 'abc' =~ 'abc' ]] && echo 'match' || echo 'no match'

match

In [28]:

[[ 'abc' =~ 'cba' ]] && echo 'match' || echo 'no match'

no match

In [29]:

TEXT1="123-ABC-123-xyz"
TEXT2="123-ABC-321-xyz"

PATTERN="([0-9]+)-[A-Z]+-\1-[a-z]+"

[[ ${TEXT1} =~ ${PATTERN} ]] && [[ ! ${TEXT2} =~ ${PATTERN} ]] && echo 'match'
  • Finding stuff
    • find
    • grep
    • Combining find and grep
In [30]:

find . -name "*.txt"

./bar.txt
./foo.txt

In [31]:

cat bar.txt baz.txt foo.txt

One
Two
Three
cat: baz.txt: No such file or directory
Hello
Goodbye



In [32]:

cat noise.py

cat: noise.py: No such file or directory



In [33]:

grep 'while' *py



In [34]:

grep -r 'sleep' .

./UnixShell02.ipynb:    "grep -r 'sleep' ."
./UnixShell02.ipynb:    "grep -r -n --exclude=\"*.ipynb\" 'sleep' ."
./UnixShell03.ipynb:    "    sleep 1\n",
./UnixShell03.ipynb:    "    sleep 1\n",
./UnixShell03.ipynb:    "    sleep 1\n",

In [35]:

grep -r -n --exclude="*.ipynb" 'sleep' .



In [36]:

find . -name "*.txt" | xargs grep -i "one"

./bar.txt:One
  • Modifying stuff
    • tr
    • sed
In [37]:

echo 'GATTACA' | tr 'ACTG' 'actg'

gattaca

In [38]:

echo 'GATTACA' | tr 'ACTG' 'ACUG'

GAUUACA

In [39]:

echo 'GATTACA' | tr 'ACTG' 'CAGT' | rev

CACGGCT

In [40]:

echo 'The quick brown fox' | tr 'A-Za-z' 'a-zA-Z'

tHE QUICK BROWN FOX

In [41]:

echo 'The quick brown fox' | tr 'A-Za-z' 'c-zabC-ZAB'

vJG SWKEM DTQYP HQZ

In [42]:

echo 'The quick brown fox' | tr 'A-Za-z' 'c-zabC-ZAB' | tr 'c-zabC-ZAB' 'A-Za-z'

The quick brown fox

In [43]:

cat bar.txt baz.txt foo.txt > big.txt

cat: baz.txt: No such file or directory



In [44]:

cat big.txt

One
Two
Three
Hello
Goodbye

In [45]:

cat big.txt | sed 4,6d

One
Two
Three

In [46]:

cat big.txt | sed /'^ '/d

One
Two
Three
Hello
Goodbye

In [47]:

cat big.txt | sed -n /'o'/Ip

sed: 1: "/o/Ip": invalid command code I



In [48]:

cat > humpty.txt <<EOF
Humpty Dumpty sat on a wall,
Humpty Dumpty had a big fall.
All the king's horses and all the king's men
Couldn't put Humpty together again
EOF

In [49]:

sed s/'u.p'/'[XXX]'/ humpty.txt

H[XXX]ty Dumpty sat on a wall,
H[XXX]ty Dumpty had a big fall.
All the king's horses and all the king's men
Couldn't put H[XXX]ty together again

In [50]:

sed s/'u.p'/'[XXX]'/g humpty.txt

H[XXX]ty D[XXX]ty sat on a wall,
H[XXX]ty D[XXX]ty had a big fall.
All the king's horses and all the king's men
Couldn't put H[XXX]ty together again

In [51]:

sed /fall/s/'u.p'/'[XXX]'/g humpty.txt

Humpty Dumpty sat on a wall,
H[XXX]ty D[XXX]ty had a big fall.
All the king's horses and all the king's men
Couldn't put Humpty together again

In [52]:

sed /fall/s/'u.p'/'[XXX]'/2 humpty.txt

Humpty Dumpty sat on a wall,
Humpty D[XXX]ty had a big fall.
All the king's horses and all the king's men
Couldn't put Humpty together again
  • Working with arithmetic expressions
    • seq
    • Integer arithmetic
    • Floating point arithmetic
In [53]:

seq 5 | tr '\n' ', '

1,2,3,4,5,

In [54]:

seq 3 7  | tr '\n' ', '

3,4,5,6,7,

In [55]:

seq 3 2 7  | tr '\n' ', '

3,5,7,

In [56]:

seq 1 0.2 2  | tr '\n' ', '

1,1.2,1.4,1.6,1.8,2,

In [57]:

echo $((2 + 3))

5

In [58]:

for IDX in $(seq 5); do
    FILENAME=$(printf "Experiment_%03d.txt" ${IDX})
    echo $FILENAME
done

Experiment_001.txt
Experiment_002.txt
Experiment_003.txt
Experiment_004.txt
Experiment_005.txt