Unix/Unix (1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Basic Unix Tutorial\n",
    "\n",
    "This is a tutorial on basic unix operations for the [KIPAC computing boot camp](http://kipac.github.io/BootCamp).\n",
    "\n",
    "Authors: [Yao-Yuan Mao](http://yymao.github.io), [Joe DeRose](https://github.com/j-dr), [Justin Myles](https://github.com/jtmyles)\n",
    "\n",
    "Before getting started we will need to install a bash kernel for jupyter notebooks:\n",
    "\n",
    "```\n",
    "pip install bash_kernel  \n",
    "python -m bash_kernel.install\n",
    "```\n",
    "\n",
    "## Outline\n",
    "* [Introduction to Unix](#intro)\n",
    "* [Navigating the file system](#filesystem)\n",
    "* [Manipulating the file system and files within it](#file_manip)\n",
    "* [Executing commands](#exec_commands)\n",
    "* [Useful utilities](#useful_utils)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Unix<a name=\"intro\"></a>\n",
    "\n",
    "Unix refers to an operating system paradigm that has become the *de facto* standard in scientific computing, HPC, and software development. Unix was originally a single propriety operating system, but is now most prevalent in the form of distributions of operating systems based on the open source Linux kernel. In this tutorial, we focus on two central concepts of Unix: the filesystem and the shell. The Unix filesystem is summarized by the maxim [“Everything is a file”](https://en.wikipedia.org/wiki/Everything_is_a_file). In this filesystem everything from external hardware components to text files are organized in a single [tree](https://en.wikipedia.org/wiki/Tree_%28data_structure%29).  The Unix shell is an interactive program (e.g. bash, csh, zsh, tcsh) that takes as input text commands written by a user, sends them to the operating system, and shows text output. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Navigating the file system<a name=\"filesystem\"></a>\n",
    "\n",
    "Let’s take a look at `/`, the root of the filesystem. Open a shell and enter `ls /`, then press `return`. An example output is below.\n",
    "\n",
    "Here `bin` stands for binaries. This is a directory that contains very basic programs like `ls`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```  bash\n",
    "-bash-4.1$ ls /\n",
    "bin   cern    dev    lib    lost+found  media  mnt  nfs  proc  sbin     scs      selinux  sys  u    var\n",
    "afs   boot  cgroup  etc  home  lib64  lustre      misc   net  opt  root  scratch  scswork  srv      tmp  usr\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To learn more about these files, we can pass some flags to the `ls` program. Now try `ls -la /`. (to learn what flags a program allows, see its manual page with e.g. `man ls`). This tells `ls` that you want to see the output in *l*ong listing format and that you want to see *a*ll files, including files with names that begin with `.`\n",
    "\n",
    "In this long output, you see that in the Unix filesystem, each file has a set of rules governing who can see and use the file. \n",
    "\n",
    "The first char specifies the file type (e.g. `-` for normal file, `d` for directory, `l` for link).\n",
    "\n",
    "The next nine chars specify the permission to read (`r`), write (`w`), and/or execute (`x`) for the Owner, Group, and Other users, respectively. To learn more about the output, see the full `man` page with `info coreutils 'ls invocation’`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "-bash-4.1$ ls -la /\n",
    "total 1170\n",
    "dr-xr-xr-x.   33 root root   4096 Sep 11 17:21 .\n",
    "dr-xr-xr-x.   33 root root   4096 Sep 11 17:21 ..\n",
    "-rw-r--r--     1 root root      0 Sep 11 17:21 .autofsck\n",
    "-rw-r--r--     1 root root      0 Sep 13  2013 .autorelabel\n",
    "drwx------     2 root root   4096 Sep 13  2013 .elinks\n",
    "-rw-r--r--     1 root bin       0 Nov 12  2013 .readahead_collect\n",
    "-rw-r--r--.    1 root root      0 Sep 13  2013 Kickstart_end\n",
    "drwxr-xr-x   401 root root  18432 Dec 31  1969 afs\n",
    "dr-xr-xr-x.    2 root root   4096 Jun 21 02:23 bin\n",
    "dr-xr-xr-x.    5 root root   4096 Aug 16 02:05 boot\n",
    "lrwxrwxrwx     1 root root     48 Sep 13  2013 cern -> /afs/slac.stanford.edu/package/cernlib/@sys/cern\n",
    "drwxr-xr-x.    2 root root   4096 Nov 28  2017 cgroup\n",
    "drwxr-xr-x    17 root root   3720 Sep 11 17:22 dev\n",
    "drwxr-xr-x.  153 root root  12288 Oct  9 08:28 etc\n",
    "drwxr-xr-x     4 root bin    4096 May  1  2015 gpfs\n",
    "drwxr-xr-x.    2 root root   4096 Jun 28  2011 home\n",
    "dr-xr-xr-x.   13 root root   4096 Jul  5 22:58 lib\n",
    "dr-xr-xr-x.   11 root root  12288 Jun 21 02:24 lib64\n",
    "drwx------.    2 root root  16384 Sep 13  2013 lost+found\n",
    "drwxr-xr-x     3 root bin    4096 Sep 18  2013 lustre\n",
    "drwxr-xr-x.    2 root root   4096 Jun 28  2011 media\n",
    "drwxr-xr-x.    2 root root   4096 May 16 17:36 misc\n",
    "drwxr-xr-x.    2 root root   4096 Jun 28  2011 mnt\n",
    "drwxr-xr-x.    2 root root   4096 May 16 17:36 net\n",
    "dr-xr-xr-x   112 root bin    4096 Mar 26  2018 nfs\n",
    "drwxr-xr-x.    8 root root   4096 Jun 24  2015 opt\n",
    "dr-xr-xr-x   492 root root      0 Sep 11 17:21 proc\n",
    "drwx------.    4 root root   4096 Jan 13  2017 root\n",
    "dr-xr-xr-x.    2 root root  12288 Sep 13 01:58 sbin\n",
    "drwxrwxrwt. 2320 root root  90112 Sep 26 00:28 scratch\n",
    "drwx------     3 scs  bin    4096 Oct  3 02:00 scs\n",
    "drwxr-xr-x.    6 root root   4096 Sep 17  2014 scswork\n",
    "drwxr-xr-x.    2 root root   4096 Sep 13  2013 selinux\n",
    "drwxr-xr-x.    2 root root   4096 Jun 28  2011 srv\n",
    "drwxr-xr-x    13 root root      0 Sep 11 17:21 sys\n",
    "drwxrwxrwt. 2089 root root 933888 Oct  9 08:51 tmp\n",
    "lrwxrwxrwx     1 root root     24 Sep 13  2013 u -> /afs/slac.stanford.edu/u\n",
    "drwxr-xr-x.   18 root root   4096 May  1  2015 usr\n",
    "drwxr-xr-x.   27 root root   4096 May  1  2015 var\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike using a desktop enviornment, when you are working with a command line interface, you always \"exist\" in a specfic place in that system. We call this place the **Working Directory** (WD).\n",
    "\n",
    "- To print out the working directory, use *command* `pwd`. \n",
    "- To change the WD, use command `cd`, followed by an *argument*, the path of the new WD.\n",
    "    - If the path does NOT start with the file tree root (/), then it's RELATIVE to current WD\n",
    "    - To change the WD to the parent directory of current WD, usd `cd ..`\n",
    "    - To go back to the last WD, use `cd -`\n",
    "    - To go to your home directory, use `cd ~`\n",
    "    \n",
    "Try the following commands out. Use `pwd` to check if you get it right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd /Users/Joe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd Downloads"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd ~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see the files and directories inside a directory, use `ls`, followed by that directory. If no argument is followed, it lists the files and directories inside current WD. \n",
    "\n",
    "Try the following commands out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls ~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note that when you `ls` a directory, you did not automatically `cd` into that directory. You still stay in your original WD. Use `pwd` to be sure.)\n",
    "\n",
    "Keep trying some more:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -a ~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -l ~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -t ~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that by adding *optional arguments* such as `-a` or `-l` after `ls`, we obtain slightly different outputs. For `ls`, `-a` prints all files including the hidden ones, `-l` prints a long, detailed list, `-t` prints the list ordered by time.\n",
    "\n",
    "You can also combine options, like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -alt ~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You might have noticed**:\n",
    "\n",
    "- Operations are specificed by `<command> <arguments>`\n",
    "- Optional arguments *usually* start with a dash (-)\n",
    "- The names of hidden files/directories always start with a dot (.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When in doubt use ```man```, which will show the manual for any command with documentation on the linux man pages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Manipulating the file system<a name=\"file_manip\"></a>\n",
    "\n",
    "Now we are going to download some files for this session, so `cd` into a directory of your chioce, we will store the files there.\n",
    "\n",
    "Want to create a new empty directory? `cd` into the directory where you want to create the new directory, and use `mkdir` followed by the name of new directory. Then `cd` into it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir new_dir\n",
    "cd new_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Also noticed that when you make a new directory, you do not automatically `cd` into it.)\n",
    "\n",
    "Now we can download the files. I have compressed the files as a signle tarball. Type the following commands to download and decompress the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wget https://raw.githubusercontent.com/KIPAC/BootCamp/master/Unix/files_for_practice.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: If you don't have `wget` on your machine, just download [this file](https://github.com/KIPAC/BootCamp/raw/master/Unix/files_for_practice.tar.gz) and then move it to the new directory you just created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tar -xzf files_for_practice.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1\n",
    "\n",
    "- Create a directory call `mp3` under `files_for_practice/random_files`\n",
    "- Move all the mp3 files under `files_for_practice/random_files` into `files_for_practice/random_files/mp3`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd files_for_practice/random_files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir mp3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mv *.mp3 mp3/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here `mv` is the command to move or rename files/directories. It should be called as \n",
    "\n",
    "    mv [source file] [target file]\n",
    "    ### OR ###\n",
    "    mv [files to be moved [...]] [target directory]\n",
    "    \n",
    "The star symbol (\\*), or asterisk, is a **wildcard character** that matches any charcters. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2\n",
    "\n",
    "- Create a directory call `est` under `files_for_practice/random_files`\n",
    "- **Copy** all files whose filenames _are_ `est`, **regardless of their file extensions**, under `files_for_practice/random_files`, into `files_for_practice/random_files/est`\n",
    "\n",
    "(What's the command for copy? You guessed it --- `cp`.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir est"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cp est.* est/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3\n",
    "\n",
    "- In `files_for_practice/random_files`, rename the file `architecto.mp3` to `consequuntur.html`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mv architecto.mp3 consequuntur.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4\n",
    "\n",
    "- In `files_for_practice/random_files`, remove the file `molestias.css`\n",
    "\n",
    "(Yes, command for remove is `rm`. It is a good idea to add `-i` after it, so that you won't accidental delete files you don't want to delete.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rm -i molestias.css"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Useful utilities <a name=\"useful_utils\"></a>\n",
    "\n",
    "### What we have learned so far\n",
    "\n",
    "- `pwd`: print current working directory\n",
    "- `cd`: change working directory\n",
    "- `ls`: list files and directories\n",
    "- `mkdir`: make new directories\n",
    "- `mv`: move files and directories\n",
    "- `cp`: copy files\n",
    "- `rm`: delete files\n",
    "- `wget [URL]`: download file\n",
    "- `tar -xzf [FILE]`: uncompress file\n",
    "\n",
    "Linux utilities like these are written in highly optimized code and can accomplish complex tasks. It's worth developing familiarity with these programs so you can take advantage of their speed when they apply to the task you need to accomplish."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read text files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Use `more` or `less` to read a text file.\n",
    "   - Use Page Up/Down to scroll\n",
    "   - /keyword to search for a keyword, n/N to navigate to next/previous result\n",
    "   - Type `q` to quit\n",
    "- Use `cat` to print out the content of a file.\n",
    "- Use `head` to print out the first few lines of a file.\n",
    "- Use `tail` to print out the last few lines of a file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 5\n",
    "\n",
    "- what's the second to the last line in file `users.txt`?\n",
    "- what's the third line in file `clients.txt`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tail -n 2 users.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head -n 3 clients.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Execute files <a name=\"exec_commands\"></a>\n",
    "\n",
    "We have already used many *programs* already, like `wget` and `tar` --- they *are* programs with command-line interfaces. \n",
    "\n",
    "Just on like all other systems, these programs live *somewhere* in the system. You can find out where with the `which` command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "which tar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "which wget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "which which"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The system knows where to find these programs by looking into some pre-defined paths, which are stored in an enviroment variable called `$PATH`. We'll talk more about variables later. For now you can check out what's in your pre-defined paths by printing out this variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "echo $PATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what if I have a program also called `wget` in my current WD and I want to run it? \n",
    "\n",
    "You then need to specify it's path (i.e. at least one slash '/' needs to appear). It the program is in your current WD, you can call it with `./<program name>`, where the dot (.) stands for current WD."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 6\n",
    "\n",
    "- change WD to files_for_practice/executables\n",
    "- try running the fake programs `wget` and `tar`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd ../executables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "./wget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "./tar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice that you cannot execute `tar`. The reason is that you don't have the permission to execute it. \n",
    "\n",
    "You can **change the permission** (read/write/execute) by the command `chmod`. Try:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chmod u+x tar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 7\n",
    "\n",
    "- remove user's read permisson from the file `alphabet.txt`\n",
    "- See if it works by reading the file\n",
    "- If it works, reinstate user's read permisson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chmod u-r alphabet.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat alphabet.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chmod u+r alphabet.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I/O redirection and piping\n",
    "\n",
    "**Redirection**:\n",
    "To read `input_file` as standard input and print standard output to a new file `output_file`:\n",
    "\n",
    "    command < input_file > output_file\n",
    "    \n",
    "You can use only one part of it:\n",
    "\n",
    "    command < input_file\n",
    "    \n",
    "    command > output_file\n",
    "    \n",
    "You can also redirect the output and **append** to a file with two arrows:\n",
    "\n",
    "    command >> output_file\n",
    "\n",
    "\n",
    "**Piping**:\n",
    "To use the standard output from `command1` as the standard input of `command2`\n",
    "\n",
    "    command1 | command2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 8\n",
    "\n",
    "- add execute permission to user for the file `reverse`\n",
    "- run `reverse` with the content of `alphabet.txt` as stanford input (try both redirection and piping)\n",
    "\n",
    "**Hints**:\n",
    "recall the command to change permission, and the command to print out the whole file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chmod u+x reverse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "./reverse < alphabet.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 9\n",
    "\n",
    "- go to the directory `files_for_practice/random_files`\n",
    "- how many files in total are in this directory?\n",
    "- how many \"png\" files are in this directory?\n",
    "\n",
    "\n",
    "**Hints**:\n",
    "\n",
    "- `ls` has an option `-1`. With this option, each file would be printed as one single line.\n",
    "- Command `wc` can count the words and lines from standard input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd ../random_files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -1 | wc -l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -1 *.png | wc -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## grep\n",
    "\n",
    "grep is a powerful program that finds patterns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grep ^d users.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grep ^d *.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -1 | grep -E i.\\.mp3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 10\n",
    "Search grepdata.txt for all lines that do not begin with a capital letter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grep '^[^A-Z]' grepdata.txt "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## sed\n",
    "sed is a stream editor whose strength is altering input streams (such as text files).\n",
    "\n",
    "To replace a phrase in a file the syntax is:\n",
    "\n",
    "```\n",
    "sed 's/phrase/replacement/' filename\n",
    "```\n",
    "\n",
    "To learn more about using `sed`, see the extensive tutorial at [link](http://www.grymoire.com/Unix/Sed.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sed 's/\\./SPAM/' users.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sed -i.bak 's/\\./SPAM/' users.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### awk\n",
    "awk is another useful program for selecting parts of text files.\n",
    "\n",
    "To learn more about using `awk`, see the extensive tutorial at [link](http://www.grymoire.com/Unix/Awk.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -l | awk '{print $1}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monitoring and killing processes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can view all processes currently running in the terminal using ```ps```, and kill them using ```kill```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 11\n",
    "Kill all anaconda processes using ```ps``` and ```kill```. \n",
    "\n",
    "*Hint: use ```xargs```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ps | grep anaconda | awk '{print $1}' | xargs kill"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}