{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bash Exercise  (Solutions)\n",
    "\n",
    "Do the following exercises. Feel free to use `man`, or `<command> --help` or even Google to find solutions. Avoid copy and pasting solutins though - we want you to learn how to do these operations fluently, and typing the commands will help."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Clean up\n",
    "\n",
    "This will remove the docs folder in your home direcotry if you have one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "rm -rf ~/docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Creating text files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a file with the following contents in a new `docs` folder in your home directory\n",
    "\n",
    "```\n",
    "first,last,middle,age,sex\n",
    "Jane,Frost,G,23,F\n",
    "John,Mundy,F,25,M\n",
    "Bob,Evans,H,57,M\n",
    "John,Smith,M<4,M\n",
    "```\n",
    "\n",
    "- First make the `docs` folder in your home directory and make that your working directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd\n",
    "mkdir docs\n",
    "cd docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the following ways to create the document\n",
    "\n",
    "- Using `echo` and redirection (save as `echo.txt`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "echo 'first,last,middle,age,sex' > echo.txt\n",
    "echo 'Jane,Frost,G,23,F' >> echo.txt\n",
    "echo 'John,Mundy,F,25,M' >> echo.txt\n",
    "echo 'Bob,Evans,H,57,M' >> echo.txt\n",
    "echo 'John,Smith,M,4,M' >> echo.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first,last,middle,age,sex\n",
      "Jane,Frost,G,23,F\n",
      "John,Mundy,F,25,M\n",
      "Bob,Evans,H,57,M\n",
      "John,Smith,M,4,M\n"
     ]
    }
   ],
   "source": [
    "cat echo.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Using `cat` and a `heredoc` (save as `cat.txt`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat <<EOF > cat.txt\n",
    "first,last,middle,age,sex\n",
    "Jane,Frost,G,23,F\n",
    "John,Mundy,F,25,M\n",
    "Bob,Evans,H,57,M\n",
    "John,Smith,M,4,M\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first,last,middle,age,sex\n",
      "Jane,Frost,G,23,F\n",
      "John,Mundy,F,25,M\n",
      "Bob,Evans,H,57,M\n",
      "John,Smith,M,4,M\n"
     ]
    }
   ],
   "source": [
    "cat cat.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Opening a terminal window and using a `nano` text editor (save as `editor.txt`). You can also use `emacs` or `vi` if you are familiar with these traditional editors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simulate using editor\n",
    "# You should not do this!\n",
    "\n",
    "cp cat.txt editor.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. File and directory management."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Display your current working directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/cliburn/docs\n"
     ]
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Change your working directory to your hoem directory if not already there"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- List the contents of the `docs` folder in your home directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cat.txt  echo.txt  editor.txt\n"
     ]
    }
   ],
   "source": [
    "ls docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Make a new directory `stuff` within the `docs` folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir docs/stuff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Move the files `cat.txt` and `editor.txt` to the `stuff` folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd docs\n",
    "mv cat.txt editor.txt stuff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Copy the file `echo.txt` to the `stuff` folder as `echoecho.txt`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "cp echo.txt stuff/echoecho.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Show the contents of the docs directory recursively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/cliburn/docs:\n",
      "echo.txt  stuff\n",
      "\n",
      "/home/cliburn/docs/stuff:\n",
      "cat.txt  echoecho.txt  editor.txt\n"
     ]
    }
   ],
   "source": [
    "ls -R ~/docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Remove the `stuff` directory and everything in it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "docs:\n",
      "echo.txt\n"
     ]
    }
   ],
   "source": [
    "rm -rf stuff\n",
    "cd\n",
    "ls -R docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Working with text files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Display the contents of `echo.txt` without the header (first) line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Jane,Frost,G,23,F\n",
      "John,Mundy,F,25,M\n",
      "Bob,Evans,H,57,M\n",
      "John,Smith,M,4,M\n"
     ]
    }
   ],
   "source": [
    "tail +2 docs/echo.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Display the contents of `echo.txt` sorted by first name (do not display the first row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bob,Evans,H,57,M\n",
      "Jane,Frost,G,23,F\n",
      "John,Mundy,F,25,M\n",
      "John,Smith,M,4,M\n"
     ]
    }
   ],
   "source": [
    "tail +2 docs/echo.txt | sort -t',' -k 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Display the contents of `echo.txt` sorted by age in descending order (do not display the first row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bob,Evans,H,57,M\n",
      "John,Mundy,F,25,M\n",
      "Jane,Frost,G,23,F\n",
      "John,Smith,M,4,M\n"
     ]
    }
   ],
   "source": [
    "tail +2 docs/echo.txt | sort -t',' -rn -k 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Display only lines that contain `John`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "John,Mundy,F,25,M\n",
      "John,Smith,M,4,M\n"
     ]
    }
   ],
   "source": [
    "grep 'John' docs/echo.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Display only lines that do not contain `John`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first,last,middle,age,sex\n",
      "Jane,Frost,G,23,F\n",
      "Bob,Evans,H,57,M\n"
     ]
    }
   ],
   "source": [
    "grep -v 'John' docs/echo.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Display only lines that contain 4-letter words starting with `J`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Jane,Frost,G,23,F\n",
      "John,Mundy,F,25,M\n",
      "John,Smith,M,4,M\n"
     ]
    }
   ],
   "source": [
    "grep 'J...' docs/echo.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Save only the first three columns of data sorted by last name to a new file `names.txt` in the `docs` folder (do not display the first row))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat docs/echo.txt | cut -d ',' -f 1-3 | tail +2 | sort -t ',' -k 2 > docs/names.txt "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bob,Evans,H\n",
      "Jane,Frost,G\n",
      "John,Mundy,F\n",
      "John,Smith,M\n"
     ]
    }
   ],
   "source": [
    "cat docs/names.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional challenging exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Display the contents of `echo.txt` but changing all occurrences of `John` to `Tom`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first,last,middle,age,sex\n",
      "Jane,Frost,G,23,F\n",
      "Tom,Mundy,F,25,M\n",
      "Bob,Evans,H,57,M\n",
      "Tom,Smith,M,4,M\n"
     ]
    }
   ],
   "source": [
    "cat docs/echo.txt | sed 's/John/Tom/g'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Save only the rows corresponding to males to a new file `male.txt` in the `docs` folder. (do not display the first row))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat docs/echo.txt | awk 'BEGIN {FS = \",\"} ; $5 == \"M\"' > docs/male.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "John,Mundy,F,25,M\n",
      "Bob,Evans,H,57,M\n",
      "John,Smith,M,4,M\n"
     ]
    }
   ],
   "source": [
    "cat docs/male.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "echo.txt  male.txt  names.txt\n"
     ]
    }
   ],
   "source": [
    "ls docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Using a for loop, save lines 2-3 of each file in the docs folder as a new file with a name that looks like `<originaal name>-copy.txt` in a folder with a name like `<originaal name>`. \n",
    "\n",
    "- Upon completiton, `ls -R ~/docs` should have this structure\n",
    "\n",
    "```\n",
    "/home/cliburn/docs:\n",
    "echo  echo.txt  male  male.txt  names  names.txt\n",
    "\n",
    "/home/cliburn/docs/echo:\n",
    "echo-copy.txt\n",
    "\n",
    "/home/cliburn/docs/male:\n",
    "male-copy.txt\n",
    "\n",
    "/home/cliburn/docs/names:\n",
    "names-copy.txt\n",
    "```\n",
    "\n",
    "- Upon completioin, `wc ~/docs/*/*` shouod show this\n",
    "\n",
    "```\n",
    "2  2 36 docs/echo/echo-copy.txt\n",
    "2  2 34 docs/male/male-copy.txt\n",
    "2  2 26 docs/names/names-copy.txt\n",
    "6  6 96 total\n",
    "```\n",
    "\n",
    "Hints:\n",
    "\n",
    "- Use variables to store path and file names (convetnion for variable names is ALL_CAPS_WITH_UNDERSCORES)\n",
    "- Use `globbig` with `ls` to show the full path to a file\n",
    "- The direcotry part of a path can be extracted with `dirname`\n",
    "- The filename part of a path can be extracted with `basename`\n",
    "- You can remove the file extension of a filename stored in `X` with `Y=${X%.*}`\n",
    "- Experiment using the above hints with simple one line commands to understand what they do\n",
    "- Use `echo` statements in your for loop to see what is going on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "for FILE in $(ls ~/docs/*)\n",
    "do\n",
    "    DIR_NAME=$(dirname $FILE)\n",
    "    FILE_NAME=$(basename $FILE)\n",
    "    NAME=${FILE_NAME%.*}\n",
    "    NEW_DIR=$DIR_NAME/$NAME\n",
    "    NEW_FILE=${NAME}-copy.txt\n",
    "    mkdir -p $NEW_DIR\n",
    "    cat $FILE | head -3 | tail -2  > $NEW_DIR/$NEW_FILE\n",
    "done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/cliburn/docs:\n",
      "echo  echo.txt  male  male.txt  names  names.txt\n",
      "\n",
      "/home/cliburn/docs/echo:\n",
      "echo-copy.txt\n",
      "\n",
      "/home/cliburn/docs/male:\n",
      "male-copy.txt\n",
      "\n",
      "/home/cliburn/docs/names:\n",
      "names-copy.txt\n"
     ]
    }
   ],
   "source": [
    "ls -R ~/docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 2  2 36 docs/echo/echo-copy.txt\n",
      " 2  2 34 docs/male/male-copy.txt\n",
      " 2  2 26 docs/names/names-copy.txt\n",
      " 6  6 96 total\n"
     ]
    }
   ],
   "source": [
    "wc docs/*/*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}