The Unix Shell: File and Directory Management¶
Listing files¶
In [1]:
ls
0
01_The_Unix_Shell.ipynb
01_The_Unix_Shell_Solutions.ipynb
The Unix Shell - File and Directory Management.ipynb
The Unix Shell - Finding Stuff.ipynb
The Unix Shell - Getting Help.ipynb
The Unix Shell - Shell Scripts.ipynb
The Unix Shell - Working with Text.ipynb
data
hello.txt
one two three
scripts
stderr.txt
stdout.txt
test1.txt
In [2]:
ls data
example.fna iris.csv iris24.csv
In [3]:
ls scripts
avg.sh cat_if_exists.sh extract_headers.sh
Exclude current and parent directory¶
In [5]:
ls -A
.DS_Store
.gitignore
.ipynb_checkpoints
0
01_The_Unix_Shell.ipynb
01_The_Unix_Shell_Solutions.ipynb
The Unix Shell - File and Directory Management.ipynb
The Unix Shell - Finding Stuff.ipynb
The Unix Shell - Getting Help.ipynb
The Unix Shell - Shell Scripts.ipynb
The Unix Shell - Working with Text.ipynb
data
hello.txt
one two three
scripts
stderr.txt
stdout.txt
test1.txt
Show details¶
In [6]:
ls -l
total 280
-rw-r--r-- 1 cliburn staff 0 Jun 9 08:23 0
-rw-r--r-- 1 cliburn staff 7378 May 2 17:21 01_The_Unix_Shell.ipynb
-rw-r--r-- 1 cliburn staff 11613 Jun 7 18:22 01_The_Unix_Shell_Solutions.ipynb
-rw-r--r-- 1 cliburn staff 27356 Jun 9 09:01 The Unix Shell - File and Directory Management.ipynb
-rw-r--r-- 1 cliburn staff 23128 Jun 8 18:25 The Unix Shell - Finding Stuff.ipynb
-rw-r--r-- 1 cliburn staff 18214 Jun 8 11:29 The Unix Shell - Getting Help.ipynb
-rw-r--r-- 1 cliburn staff 23346 Jun 9 08:50 The Unix Shell - Shell Scripts.ipynb
-rw-r--r-- 1 cliburn staff 8386 Jun 8 14:04 The Unix Shell - Working with Text.ipynb
drwxr-xr-x 6 cliburn staff 204 Jun 9 08:38 data
-rw-r--r-- 1 cliburn staff 45 Jun 8 11:58 hello.txt
-rw-r--r-- 1 cliburn staff 0 Jun 8 12:03 one two three
drwxr-xr-x 6 cliburn staff 204 Jun 9 08:46 scripts
-rw-r--r-- 1 cliburn staff 42 Jun 8 11:58 stderr.txt
-rw-r--r-- 1 cliburn staff 0 Jun 8 11:58 stdout.txt
-rw-r--r-- 1 cliburn staff 50 Jun 8 12:06 test1.txt
Sort by last modified time¶
In [7]:
ls -lt
total 280
-rw-r--r-- 1 cliburn staff 27356 Jun 9 09:01 The Unix Shell - File and Directory Management.ipynb
-rw-r--r-- 1 cliburn staff 23346 Jun 9 08:50 The Unix Shell - Shell Scripts.ipynb
drwxr-xr-x 6 cliburn staff 204 Jun 9 08:46 scripts
drwxr-xr-x 6 cliburn staff 204 Jun 9 08:38 data
-rw-r--r-- 1 cliburn staff 0 Jun 9 08:23 0
-rw-r--r-- 1 cliburn staff 23128 Jun 8 18:25 The Unix Shell - Finding Stuff.ipynb
-rw-r--r-- 1 cliburn staff 8386 Jun 8 14:04 The Unix Shell - Working with Text.ipynb
-rw-r--r-- 1 cliburn staff 50 Jun 8 12:06 test1.txt
-rw-r--r-- 1 cliburn staff 0 Jun 8 12:03 one two three
-rw-r--r-- 1 cliburn staff 42 Jun 8 11:58 stderr.txt
-rw-r--r-- 1 cliburn staff 0 Jun 8 11:58 stdout.txt
-rw-r--r-- 1 cliburn staff 45 Jun 8 11:58 hello.txt
-rw-r--r-- 1 cliburn staff 18214 Jun 8 11:29 The Unix Shell - Getting Help.ipynb
-rw-r--r-- 1 cliburn staff 11613 Jun 7 18:22 01_The_Unix_Shell_Solutions.ipynb
-rw-r--r-- 1 cliburn staff 7378 May 2 17:21 01_The_Unix_Shell.ipynb
Human readable output¶
In [8]:
ls -lth
total 280
-rw-r--r-- 1 cliburn staff 27K Jun 9 09:01 The Unix Shell - File and Directory Management.ipynb
-rw-r--r-- 1 cliburn staff 23K Jun 9 08:50 The Unix Shell - Shell Scripts.ipynb
drwxr-xr-x 6 cliburn staff 204B Jun 9 08:46 scripts
drwxr-xr-x 6 cliburn staff 204B Jun 9 08:38 data
-rw-r--r-- 1 cliburn staff 0B Jun 9 08:23 0
-rw-r--r-- 1 cliburn staff 23K Jun 8 18:25 The Unix Shell - Finding Stuff.ipynb
-rw-r--r-- 1 cliburn staff 8.2K Jun 8 14:04 The Unix Shell - Working with Text.ipynb
-rw-r--r-- 1 cliburn staff 50B Jun 8 12:06 test1.txt
-rw-r--r-- 1 cliburn staff 0B Jun 8 12:03 one two three
-rw-r--r-- 1 cliburn staff 42B Jun 8 11:58 stderr.txt
-rw-r--r-- 1 cliburn staff 0B Jun 8 11:58 stdout.txt
-rw-r--r-- 1 cliburn staff 45B Jun 8 11:58 hello.txt
-rw-r--r-- 1 cliburn staff 18K Jun 8 11:29 The Unix Shell - Getting Help.ipynb
-rw-r--r-- 1 cliburn staff 11K Jun 7 18:22 01_The_Unix_Shell_Solutions.ipynb
-rw-r--r-- 1 cliburn staff 7.2K May 2 17:21 01_The_Unix_Shell.ipynb
Recursive listing¶
In [9]:
ls -R
0
01_The_Unix_Shell.ipynb
01_The_Unix_Shell_Solutions.ipynb
The Unix Shell - File and Directory Management.ipynb
The Unix Shell - Finding Stuff.ipynb
The Unix Shell - Getting Help.ipynb
The Unix Shell - Shell Scripts.ipynb
The Unix Shell - Working with Text.ipynb
data
hello.txt
one two three
scripts
stderr.txt
stdout.txt
test1.txt
./data:
example.fna iris.csv iris24.csv
./scripts:
avg.sh cat_if_exists.sh extract_headers.sh
Globbing¶
The use of wild cards to specify Unix paths is known as globbing.
* represets any number of characters¶
In [10]:
ls *ipynb
01_The_Unix_Shell.ipynb
01_The_Unix_Shell_Solutions.ipynb
The Unix Shell - File and Directory Management.ipynb
The Unix Shell - Finding Stuff.ipynb
The Unix Shell - Getting Help.ipynb
The Unix Shell - Shell Scripts.ipynb
The Unix Shell - Working with Text.ipynb
In [11]:
ls data/*
data/example.fna data/iris.csv data/iris24.csv
Character sets¶
[abc]represents a or b or c- [a-z] represents any lower case character
!negates
In [13]:
ls [0-9]*
0 01_The_Unix_Shell_Solutions.ipynb
01_The_Unix_Shell.ipynb
In [14]:
ls [!0-9]*
The Unix Shell - File and Directory Management.ipynb
The Unix Shell - Finding Stuff.ipynb
The Unix Shell - Getting Help.ipynb
The Unix Shell - Shell Scripts.ipynb
The Unix Shell - Working with Text.ipynb
hello.txt
one two three
stderr.txt
stdout.txt
test1.txt
data:
example.fna iris.csv iris24.csv
scripts:
avg.sh cat_if_exists.sh extract_headers.sh
Spitting paths¶
In [2]:
pwd
/Users/cliburn/_teach/bios-821-2017/lessons
In [5]:
dirname $(pwd)
/Users/cliburn/_teach/bios-821-2017
In [4]:
basename $(pwd)
lessons
In [16]:
pwd | xargs dirname
/Users/cliburn/_teach/bios-821
In [17]:
pwd | xargs basename
lessons
Making and removing new directories¶
In [26]:
mkdir foo
In [27]:
ls
0
01_The_Unix_Shell.ipynb
01_The_Unix_Shell_Solutions.ipynb
The Unix Shell - File and Directory Management.ipynb
The Unix Shell - Finding Stuff.ipynb
The Unix Shell - Getting Help.ipynb
The Unix Shell - Shell Scripts.ipynb
The Unix Shell - Working with Text.ipynb
data
foo
hello.txt
one two three
scripts
stderr.txt
stdout.txt
test1.txt
Making intermediate directories automatically¶
In [28]:
mkdir -p a/b/c/d
In [29]:
ls -R a
b
a/b:
c
a/b/c:
d
a/b/c/d:
Deleting directories¶
In [30]:
rmdir foo
Only works if directory is empty¶
The | cat part is not necessary on the command line, but is only
used here for convenience of Run All Cells as Jupyter stops on non-zero
exit codes. The | cat syntax “pipes” the output of rmdir data to
a the cat program.
In [31]:
rmdir data | cat
rmdir: data: Directory not empty
Recursive intermediate directories as well¶
In [32]:
rmdir -p a/b/c/d
In [33]:
ls
0
01_The_Unix_Shell.ipynb
01_The_Unix_Shell_Solutions.ipynb
The Unix Shell - File and Directory Management.ipynb
The Unix Shell - Finding Stuff.ipynb
The Unix Shell - Getting Help.ipynb
The Unix Shell - Shell Scripts.ipynb
The Unix Shell - Working with Text.ipynb
data
hello.txt
one two three
scripts
stderr.txt
stdout.txt
test1.txt
Working with files¶
Making an empty file¶
In [34]:
touch foo.txt
In [35]:
ls
0
01_The_Unix_Shell.ipynb
01_The_Unix_Shell_Solutions.ipynb
The Unix Shell - File and Directory Management.ipynb
The Unix Shell - Finding Stuff.ipynb
The Unix Shell - Getting Help.ipynb
The Unix Shell - Shell Scripts.ipynb
The Unix Shell - Working with Text.ipynb
data
foo.txt
hello.txt
one two three
scripts
stderr.txt
stdout.txt
test1.txt
Deleting a file¶
In [36]:
rm foo.txt
In [37]:
ls
0
01_The_Unix_Shell.ipynb
01_The_Unix_Shell_Solutions.ipynb
The Unix Shell - File and Directory Management.ipynb
The Unix Shell - Finding Stuff.ipynb
The Unix Shell - Getting Help.ipynb
The Unix Shell - Shell Scripts.ipynb
The Unix Shell - Working with Text.ipynb
data
hello.txt
one two three
scripts
stderr.txt
stdout.txt
test1.txt
Viewing a file¶
In [38]:
cat scripts/avg.sh
#!/bin/bash
sum=0
n=0
while read line
do
if [ ! -z "$line" ]
then
n=$((n+1))
sum=$(echo $sum + $line | bc)
fi
done
echo $sum / $n | bc -l
In [39]:
head -n 3 scripts/avg.sh
#!/bin/bash
sum=0
n=0
In [40]:
tail -n 3 scripts/avg.sh
fi
done
echo $sum / $n | bc -l
Can start tail form a specified line number with (+)¶
In [41]:
tail -n +4 scripts/avg.sh
while read line
do
if [ ! -z "$line" ]
then
n=$((n+1))
sum=$(echo $sum + $line | bc)
fi
done
echo $sum / $n | bc -l
Copying and moving files¶
In [42]:
ls
0
01_The_Unix_Shell.ipynb
01_The_Unix_Shell_Solutions.ipynb
The Unix Shell - File and Directory Management.ipynb
The Unix Shell - Finding Stuff.ipynb
The Unix Shell - Getting Help.ipynb
The Unix Shell - Shell Scripts.ipynb
The Unix Shell - Working with Text.ipynb
data
hello.txt
one two three
scripts
stderr.txt
stdout.txt
test1.txt
Copying files¶
In [43]:
cp "The Unix Shell - File and Directory Management.ipynb" The_Unix_Shell.ipynb
In [44]:
ls
0
01_The_Unix_Shell.ipynb
01_The_Unix_Shell_Solutions.ipynb
The Unix Shell - File and Directory Management.ipynb
The Unix Shell - Finding Stuff.ipynb
The Unix Shell - Getting Help.ipynb
The Unix Shell - Shell Scripts.ipynb
The Unix Shell - Working with Text.ipynb
The_Unix_Shell.ipynb
data
hello.txt
one two three
scripts
stderr.txt
stdout.txt
test1.txt
Copying directories (Recursive copy)¶
In [45]:
cp -R data data2
In [46]:
ls -R
0
01_The_Unix_Shell.ipynb
01_The_Unix_Shell_Solutions.ipynb
The Unix Shell - File and Directory Management.ipynb
The Unix Shell - Finding Stuff.ipynb
The Unix Shell - Getting Help.ipynb
The Unix Shell - Shell Scripts.ipynb
The Unix Shell - Working with Text.ipynb
The_Unix_Shell.ipynb
data
data2
hello.txt
one two three
scripts
stderr.txt
stdout.txt
test1.txt
./data:
example.fna iris.csv iris24.csv
./data2:
example.fna iris.csv iris24.csv
./scripts:
avg.sh cat_if_exists.sh extract_headers.sh
Renaming a file¶
In [47]:
mv The_Unix_Shell.ipynb The_Unix_Shell_Copy.ipynb
In [48]:
ls
0
01_The_Unix_Shell.ipynb
01_The_Unix_Shell_Solutions.ipynb
The Unix Shell - File and Directory Management.ipynb
The Unix Shell - Finding Stuff.ipynb
The Unix Shell - Getting Help.ipynb
The Unix Shell - Shell Scripts.ipynb
The Unix Shell - Working with Text.ipynb
The_Unix_Shell_Copy.ipynb
data
data2
hello.txt
one two three
scripts
stderr.txt
stdout.txt
test1.txt
Move a file to a new location¶
In [49]:
mv The_Unix_Shell_Copy.ipynb scripts
In [50]:
ls -R
0
01_The_Unix_Shell.ipynb
01_The_Unix_Shell_Solutions.ipynb
The Unix Shell - File and Directory Management.ipynb
The Unix Shell - Finding Stuff.ipynb
The Unix Shell - Getting Help.ipynb
The Unix Shell - Shell Scripts.ipynb
The Unix Shell - Working with Text.ipynb
data
data2
hello.txt
one two three
scripts
stderr.txt
stdout.txt
test1.txt
./data:
example.fna iris.csv iris24.csv
./data2:
example.fna iris.csv iris24.csv
./scripts:
The_Unix_Shell_Copy.ipynb cat_if_exists.sh
avg.sh extract_headers.sh
File compression and archival¶
Combine multiple files into single file¶
In [51]:
ls data
example.fna iris.csv iris24.csv
In [52]:
man tar | head -n 20
BSDTAR(1) BSD General Commands Manual BSDTAR(1)
NAME
tar -- manipulate tape archives
SYNOPSIS
tar [bundled-flags <args>] [<file> | <pattern> ...]
tar {-c} [options] [files | directories]
tar {-r | -u} -f archive-file [options] [files | directories]
tar {-t | -x} [options] [patterns]
DESCRIPTION
tar creates and manipulates streaming archive files. This implementation
can extract from tar, pax, cpio, zip, jar, ar, and ISO 9660 cdrom images
and can create tar, pax, cpio, ar, and shar archives.
The first synopsis form shows a ``bundled'' option word. This usage is
provided for compatibility with historical implementations. See COMPATI-
BILITY below for details.
In [53]:
tar -cvf data.tar data
a data
a data/.ipynb_checkpoints
a data/example.fna
a data/iris.csv
a data/iris24.csv
In [54]:
rm -rf data/
In [55]:
ls data*
data.tar
data2:
example.fna iris.csv iris24.csv
Compress concatenated file¶
In [56]:
gzip data.tar
In [57]:
ls data*
data.tar.gz
data2:
example.fna iris.csv iris24.csv
Uncompress¶
In [58]:
gunzip data.tar.gz
In [59]:
ls data*
data.tar
data2:
example.fna iris.csv iris24.csv
Recover original files¶
In [60]:
tar -xvf data.tar
x data/
x data/.ipynb_checkpoints/
x data/example.fna
x data/iris.csv
x data/iris24.csv
In [61]:
ls data*
data.tar
data:
example.fna iris.csv iris24.csv
data2:
example.fna iris.csv iris24.csv
In [62]:
rm data.tar
Concatenate and compress¶
In [63]:
tar -cvzf data.tar.gz data
a data
a data/.ipynb_checkpoints
a data/example.fna
a data/iris.csv
a data/iris24.csv
In [64]:
rm -rf data/
In [65]:
ls data*
data.tar.gz
data2:
example.fna iris.csv iris24.csv
Uncompress and recover¶
In [66]:
tar -xvzf data.tar.gz
x data/
x data/.ipynb_checkpoints/
x data/example.fna
x data/iris.csv
x data/iris24.csv
In [67]:
ls data*
data.tar.gz
data:
example.fna iris.csv iris24.csv
data2:
example.fna iris.csv iris24.csv
In [68]:
rm data.tar.gz
Clean up¶
In [69]:
rm scripts/The_Unix_Shell_Copy.ipynb
In [70]:
rm -r data2
In [71]:
ls -R
0
01_The_Unix_Shell.ipynb
01_The_Unix_Shell_Solutions.ipynb
The Unix Shell - File and Directory Management.ipynb
The Unix Shell - Finding Stuff.ipynb
The Unix Shell - Getting Help.ipynb
The Unix Shell - Shell Scripts.ipynb
The Unix Shell - Working with Text.ipynb
data
hello.txt
one two three
scripts
stderr.txt
stdout.txt
test1.txt
./data:
example.fna iris.csv iris24.csv
./scripts:
avg.sh cat_if_exists.sh extract_headers.sh