unix-conda-pip.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "399f6d9e",
   "metadata": {},
   "source": [
    "# Unix - Conda - Pip"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef16b651",
   "metadata": {},
   "source": [
    "We run commands in CLI, e.g., Terminal, but here we are using Jupyter Notebook to demonstrate:\n",
    "\n",
    "1. Output from commands\n",
    "2. That when experiementing or developing programs in the notebook, you can avoid switching tools.\n",
    "3. That you can install pacakges in the beginning of the notebook (e.g., Sagemaker) so it's clear what packages are used."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "944d1a51",
   "metadata": {},
   "source": [
    "## Unix commands"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f70f6301",
   "metadata": {},
   "source": [
    "### Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a32b0652",
   "metadata": {},
   "source": [
    "For some of the basic commands, you can run use the cell to run it directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "001e8c71",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/skirui/Desktop/mle-8-projects/fourthbrain-8/assignments/mle-basictools/nb'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pwd # print working directory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00bf36c2",
   "metadata": {},
   "source": [
    "You always want to know \"where you are\" and how to look for help. Use `man` to display the user manual of any command that we can run on the terminal. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0e1d0b83",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PWD(1)                           User Commands                          PWD(1)\n",
       "\n",
       "NAME\n",
       "       pwd - print name of current/working directory\n",
       "\n",
       "SYNOPSIS\n",
       "       pwd [OPTION]...\n",
       "\n",
       "DESCRIPTION\n",
       "       Print the full filename of the current working directory.\n",
       "\n",
       "       -L, --logical\n",
       "              use PWD from environment, even if it contains symlinks\n",
       "\n",
       "       -P, --physical\n",
       "              avoid all symlinks\n",
       "\n",
       "       --help display this help and exit\n",
       "\n",
       "       --version\n",
       "              output version information and exit\n",
       "\n",
       "       If no option is specified, -P is assumed.\n",
       "\n",
       "       NOTE:  your shell may have its own version of pwd, which usually super‐\n",
       "       sedes the version described here.  Please refer to your  shell's  docu‐\n",
       "       mentation for details about the options it supports.\n",
       "\n",
       "AUTHOR\n",
       "       Written by Jim Meyering.\n",
       "\n",
       "REPORTING BUGS\n",
       "       GNU coreutils online help: <http://www.gnu.org/software/coreutils/>\n",
       "       Report pwd translation bugs to <http://translationproject.org/team/>\n",
       "\n",
       "COPYRIGHT\n",
       "       Copyright  ©  2017  Free Software Foundation, Inc.  License GPLv3+: GNU\n",
       "       GPL version 3 or later <http://gnu.org/licenses/gpl.html>.\n",
       "       This is free software: you are free  to  change  and  redistribute  it.\n",
       "       There is NO WARRANTY, to the extent permitted by law.\n",
       "\n",
       "SEE ALSO\n",
       "       getcwd(3)\n",
       "\n",
       "       Full documentation at: <http://www.gnu.org/software/coreutils/pwd>\n",
       "       or available locally via: info '(coreutils) pwd invocation'\n",
       "\n",
       "GNU coreutils 8.28               January 2018                           PWD(1)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "man pwd # press ESC to exit"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a2b554f",
   "metadata": {},
   "source": [
    "Further, cell magic `%%bash` turns a cell into a bash script where you can run multiple lines (See more cell/line magics [here](https://ipython.readthedocs.io/en/stable/interactive/magics.html)). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a0641721",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/skirui/Desktop/mle-8-projects/fourthbrain-8/assignments/mle-basictools/nb\n",
      "total 56K\n",
      "drwxrwxr-x 3 skirui skirui 4.0K Jun 23 22:59 .\n",
      "drwxrwxr-x 5 skirui skirui 4.0K Jun 23 22:47 ..\n",
      "drwxrwxr-x 2 skirui skirui 4.0K Jun 21 18:35 .ipynb_checkpoints\n",
      "-rw-rw-r-- 1 skirui skirui  41K Jun 23 22:59 unix-conda-pip.ipynb\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "pwd\n",
    "ls -lah # WE DO: What are those flags for?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2a67922f-bbfa-47f3-9758-573858c1d59f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The flag above is used to list the contents of a directory, including hidden files\n"
     ]
    }
   ],
   "source": [
    "print(\"The flag above is used to list the contents of a directory, including hidden files\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d5a8316",
   "metadata": {},
   "source": [
    "### Download a data file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71675d92",
   "metadata": {},
   "source": [
    "Let's download some data from [Census Income Data Set](https://archive.ics.uci.edu/ml/datasets/Census+Income); but first we would like to make a new directory to store the data at the same level as `nb` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3b75c995",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "# WE DO: make a new diretory called `dat` under the project root directory\n",
    "cd ..\n",
    "rm -rf dat\n",
    "mkdir dat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e4f5671",
   "metadata": {},
   "source": [
    "Let's confirm that we just created an empty directory `dat` using command `tree`. Not all command can be run in the cell directly, adding ! in front of command can do the trick:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0a31b61f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[01;34m..\u001b[00m\n",
      "├── \u001b[01;34mdat\u001b[00m\n",
      "├── LICENSE\n",
      "├── \u001b[01;34mmd\u001b[00m\n",
      "│   └── git-more.md\n",
      "├── \u001b[01;34mnb\u001b[00m\n",
      "│   └── unix-conda-pip.ipynb\n",
      "└── README.md\n",
      "\n",
      "3 directories, 4 files\n"
     ]
    }
   ],
   "source": [
    "!tree .. -L 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "273eed2c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2022-06-23 22:59:26--  https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data\n",
      "Resolving archive.ics.uci.edu (archive.ics.uci.edu)... 128.195.10.252\n",
      "Connecting to archive.ics.uci.edu (archive.ics.uci.edu)|128.195.10.252|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 3974305 (3.8M) [application/x-httpd-php]\n",
      "Saving to: ‘../dat/adult.csv’\n",
      "\n",
      "../dat/adult.csv    100%[===================>]   3.79M  16.4MB/s    in 0.2s    \n",
      "\n",
      "2022-06-23 22:59:26 (16.4 MB/s) - ‘../dat/adult.csv’ saved [3974305/3974305]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# download a csv file\n",
    "!wget https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data -O ../dat/adult.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d45bd2fd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[01;34m..\u001b[00m\n",
      "├── \u001b[01;34mdat\u001b[00m\n",
      "│   └── adult.csv\n",
      "├── LICENSE\n",
      "├── \u001b[01;34mmd\u001b[00m\n",
      "│   └── git-more.md\n",
      "├── \u001b[01;34mnb\u001b[00m\n",
      "│   └── unix-conda-pip.ipynb\n",
      "└── README.md\n",
      "\n",
      "3 directories, 5 files\n"
     ]
    }
   ],
   "source": [
    "!tree .. -L 2 # check again"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1c3ad4c",
   "metadata": {},
   "source": [
    "### Inspect data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a17d9ad5",
   "metadata": {},
   "source": [
    "Let's inspect first 10 lines of data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4b397608",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "39, State-gov, 77516, Bachelors, 13, Never-married, Adm-clerical, Not-in-family, White, Male, 2174, 0, 40, United-States, <=50K\n",
      "50, Self-emp-not-inc, 83311, Bachelors, 13, Married-civ-spouse, Exec-managerial, Husband, White, Male, 0, 0, 13, United-States, <=50K\n",
      "38, Private, 215646, HS-grad, 9, Divorced, Handlers-cleaners, Not-in-family, White, Male, 0, 0, 40, United-States, <=50K\n",
      "53, Private, 234721, 11th, 7, Married-civ-spouse, Handlers-cleaners, Husband, Black, Male, 0, 0, 40, United-States, <=50K\n",
      "28, Private, 338409, Bachelors, 13, Married-civ-spouse, Prof-specialty, Wife, Black, Female, 0, 0, 40, Cuba, <=50K\n",
      "37, Private, 284582, Masters, 14, Married-civ-spouse, Exec-managerial, Wife, White, Female, 0, 0, 40, United-States, <=50K\n",
      "49, Private, 160187, 9th, 5, Married-spouse-absent, Other-service, Not-in-family, Black, Female, 0, 0, 16, Jamaica, <=50K\n",
      "52, Self-emp-not-inc, 209642, HS-grad, 9, Married-civ-spouse, Exec-managerial, Husband, White, Male, 0, 0, 45, United-States, >50K\n",
      "31, Private, 45781, Masters, 14, Never-married, Prof-specialty, Not-in-family, White, Female, 14084, 0, 50, United-States, >50K\n",
      "42, Private, 159449, Bachelors, 13, Married-civ-spouse, Exec-managerial, Husband, White, Male, 5178, 0, 40, United-States, >50K\n"
     ]
    }
   ],
   "source": [
    "!head ../dat/adult.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17f5a9f0",
   "metadata": {},
   "source": [
    "How about the last 3 lines? ( wait, I only see 2 lines. what's wrong? ) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c59382eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22, Private, 201490, HS-grad, 9, Never-married, Adm-clerical, Own-child, White, Male, 0, 0, 20, United-States, <=50K\n",
      "52, Self-emp-inc, 287927, HS-grad, 9, Married-civ-spouse, Exec-managerial, Wife, White, Female, 15024, 0, 40, United-States, >50K\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!tail -n 3 ../dat/adult.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93d8f299",
   "metadata": {},
   "source": [
    "How many records are there?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ff9852aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "32562 ../dat/adult.csv\n"
     ]
    }
   ],
   "source": [
    "!wc -l ../dat/adult.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e36c22bf-bcfc-417b-8974-447324a0bf41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 32,562 records in the adult.csv file\n"
     ]
    }
   ],
   "source": [
    "print(\"There are 32,562 records in the adult.csv file\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "422009f8",
   "metadata": {},
   "source": [
    "Challenge: how many columns are there?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "195b5b57",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15\n"
     ]
    }
   ],
   "source": [
    "!head -1 ../dat/adult.csv | sed 's/[^,]//g' | wc -c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9105e3ba-0ac8-4906-9a3a-def7e959c863",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 15 columns in the adult.csv file\n"
     ]
    }
   ],
   "source": [
    "print(\"There are 15 columns in the adult.csv file\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e52d18e0",
   "metadata": {},
   "source": [
    "## Conda: environment manager + package manager"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fec4699",
   "metadata": {},
   "source": [
    "You might have heard about Anaconda / Miniconda / Miniforge:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70b40550",
   "metadata": {},
   "source": [
    "- What is the difference between `conda` and `miniconda`? [An answer](https://stackoverflow.com/questions/45421163/anaconda-vs-miniconda).\n",
    "- What is the difference between `miniconda` and `miniforge`? [An answer](https://stackoverflow.com/questions/60532678/what-is-the-difference-between-miniconda-and-miniforge).\n",
    "\n",
    "Though Conda is considered both a package manager and an environment manager, we  focus on using it as the environment manager, and use `pip` as the package manager."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa2f0973",
   "metadata": {},
   "source": [
    "### Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aef39f0a",
   "metadata": {},
   "source": [
    "First, install `miniconda` (Feel free to skip if you have already installed it, or any flavor of conda).\n",
    "\n",
    "Run the following commands in terminal to download the lastest Miniconda distribution and install it (Mac Intel Book). \n",
    "\n",
    "    ```\n",
    "    cd ~\n",
    "    wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh\n",
    "    bash Miniconda3-latest-MacOSX-x86_64.sh\n",
    "    ```\n",
    "\n",
    "For M1 chip, install `miniforge` instead. If you are new to `brew`, following instructions [here](https://brew.sh) to install Homebrew first:\n",
    "\n",
    "    ```\n",
    "    brew install miniforge\n",
    "    ```\n",
    "\n",
    "Outputs from these installations are rather long, so we ask you to run it in a terminal window instead."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ac65956",
   "metadata": {},
   "source": [
    "Confirm that the installation is successful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f8a10e9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/skirui/anaconda3/condabin/conda\n",
      "conda 4.10.3\n",
      "/home/skirui/anaconda3/envs/week-0/bin/python\n",
      "Python 3.8.13\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "which conda  # where the executable conda is in my path?\n",
    "conda -V     # which version of conda?\n",
    "which python # where the executable python is in my path?\n",
    "python -V    # what version of python?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6331b4b",
   "metadata": {},
   "source": [
    "### Create a new env"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b25f1463",
   "metadata": {},
   "source": [
    "Create a new conda environment named `py39_12` where you specify a python version to be 3.9.12. Note the last flag `--yes` to skip the confirmation prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "212568f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting package metadata (current_repodata.json): done\n",
      "Solving environment: done\n",
      "\n",
      "\n",
      "==> WARNING: A newer version of conda exists. <==\n",
      "  current version: 4.10.3\n",
      "  latest version: 4.13.0\n",
      "\n",
      "Please update conda by running\n",
      "\n",
      "    $ conda update -n base -c defaults conda\n",
      "\n",
      "\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /home/skirui/anaconda3/envs/py39_12\n",
      "\n",
      "  added / updated specs:\n",
      "    - python=3.9.12\n",
      "\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main\n",
      "  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu\n",
      "  ca-certificates    pkgs/main/linux-64::ca-certificates-2022.4.26-h06a4308_0\n",
      "  certifi            pkgs/main/linux-64::certifi-2022.6.15-py39h06a4308_0\n",
      "  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1\n",
      "  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2\n",
      "  libgcc-ng          pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1\n",
      "  libgomp            pkgs/main/linux-64::libgomp-11.2.0-h1234567_1\n",
      "  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1\n",
      "  ncurses            pkgs/main/linux-64::ncurses-6.3-h7f8727e_2\n",
      "  openssl            pkgs/main/linux-64::openssl-1.1.1o-h7f8727e_0\n",
      "  pip                pkgs/main/linux-64::pip-21.2.4-py39h06a4308_0\n",
      "  python             pkgs/main/linux-64::python-3.9.12-h12debd9_1\n",
      "  readline           pkgs/main/linux-64::readline-8.1.2-h7f8727e_1\n",
      "  setuptools         pkgs/main/linux-64::setuptools-61.2.0-py39h06a4308_0\n",
      "  sqlite             pkgs/main/linux-64::sqlite-3.38.3-hc218d9a_0\n",
      "  tk                 pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0\n",
      "  tzdata             pkgs/main/noarch::tzdata-2022a-hda174b7_0\n",
      "  wheel              pkgs/main/noarch::wheel-0.37.1-pyhd3eb1b0_0\n",
      "  xz                 pkgs/main/linux-64::xz-5.2.5-h7f8727e_1\n",
      "  zlib               pkgs/main/linux-64::zlib-1.2.12-h7f8727e_2\n",
      "\n",
      "\n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n",
      "#\n",
      "# To activate this environment, use\n",
      "#\n",
      "#     $ conda activate py39_12\n",
      "#\n",
      "# To deactivate an active environment, use\n",
      "#\n",
      "#     $ conda deactivate\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!conda create --name py39_12 python=3.9.12 --yes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d0bd2f20",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# conda environments:\n",
      "#\n",
      "base                     /home/skirui/anaconda3\n",
      "py39_12                  /home/skirui/anaconda3/envs/py39_12\n",
      "week-0                *  /home/skirui/anaconda3/envs/week-0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# YOU CODE HERE \n",
    "# list all environments in conda\n",
    "!conda env list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1efee8c",
   "metadata": {},
   "source": [
    "You shall see something like this:\n",
    "```\n",
    "# conda environments:\n",
    "#\n",
    "                         /Users/flora/miniforge3\n",
    "                         /Users/flora/miniforge3/envs/tf38\n",
    "base                  *  /usr/local/Caskroom/miniforge/base\n",
    "py39                     /usr/local/Caskroom/miniforge/base/envs/py39\n",
    "py39_12                  /usr/local/Caskroom/miniforge/base/envs/py39_12\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a669545e",
   "metadata": {},
   "source": [
    "### Activate an env"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46aefe99",
   "metadata": {},
   "source": [
    "By default, you are under the `base` environment, to activate the new environement, run `conda activate py39_12` in terminal.\n",
    "\n",
    "If you are using VS code, click on the top right button to switch python kernel. You will be prompted to install `ipykernel`, follow the instructions to install the package. Or you can run `source activate`, followed by \n",
    "`conda activate /usr/local/Caskroom/miniforge/base/envs/py39_12` in a cell. \n",
    "\n",
    "If you are using jupyter notebook on a localhost (by defaut 8888), you can restart the jupyter notebook after activating the new environment.\n",
    "\n",
    "Now verify the python version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "265f7506",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python 3.8.13\n",
      "/home/skirui/anaconda3/envs/week-0/bin/python\n",
      "# conda environments:\n",
      "#\n",
      "base                     /home/skirui/anaconda3\n",
      "py39_12                  /home/skirui/anaconda3/envs/py39_12\n",
      "week-0                *  /home/skirui/anaconda3/envs/week-0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "python --version\n",
    "which python\n",
    "conda env list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08985441",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>Click to see how to add the Conda env to you jupyter notebook kernels</summary>\n",
    "\n",
    "```\n",
    "pip install ipykernel\n",
    "python -m ipykernel install --user --name=py39_12\n",
    "```\n",
    "\n",
    "Next time you launch jupyter notebook, you will see `py39_12` as an option under Kernel/Change kernel.\n",
    "</details>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfc622f5",
   "metadata": {},
   "source": [
    "### Delete an env"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99ecb107",
   "metadata": {},
   "source": [
    "To keep a lean list of environments, we want to prune unused environements from time to time. Simply do the following\n",
    "```\n",
    "conda remove --name old_env --all --yes\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f3e11d0",
   "metadata": {},
   "source": [
    "## Pip: python package manager"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e2dc9e0",
   "metadata": {},
   "source": [
    "We recommend to use pip as your python package installer/manager [(fun read on pip vs conda.)](https://stackoverflow.com/questions/20994716/what-is-the-difference-between-pip-and-conda) Note you don't have install pip explictly since it was done during the installation of miniconda/miniforge. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "ac8585e9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/skirui/anaconda3/envs/week-0/bin/pip\n"
     ]
    }
   ],
   "source": [
    "!which pip"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df4b9751",
   "metadata": {},
   "source": [
    "### Install package"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8913690d",
   "metadata": {},
   "source": [
    "We can certainly run `pip` commands the same way as earlier. Just for fun, let's try Jupyter line magic `%pip`. \n",
    "\n",
    "First make sure that we would like to have the most recent version of `pip` installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f10cab48",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pip in /home/skirui/anaconda3/envs/week-0/lib/python3.8/site-packages (22.1.2)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install --upgrade pip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "6f5a19b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting numpy\n",
      "  Downloading numpy-1.23.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)\n",
      "\u001b[2K     \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m17.1/17.1 MB\u001b[0m \u001b[31m35.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting pandas\n",
      "  Downloading pandas-1.4.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)\n",
      "\u001b[2K     \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.7/11.7 MB\u001b[0m \u001b[31m28.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m0:01\u001b[0m01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: pytz>=2020.1 in /home/skirui/anaconda3/envs/week-0/lib/python3.8/site-packages (from pandas) (2022.1)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /home/skirui/anaconda3/envs/week-0/lib/python3.8/site-packages (from pandas) (2.8.2)\n",
      "Requirement already satisfied: six>=1.5 in /home/skirui/anaconda3/envs/week-0/lib/python3.8/site-packages (from python-dateutil>=2.8.1->pandas) (1.16.0)\n",
      "Installing collected packages: numpy, pandas\n",
      "Successfully installed numpy-1.23.0 pandas-1.4.3\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install numpy pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b23b1338",
   "metadata": {},
   "source": [
    "Now you can use these packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "5e863b17",
   "metadata": {},
   "outputs": [],
   "source": [
    "# allow multiple outputs in a single cell\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "88f533d8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10,)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "array([[0.26406018, 0.94896433],\n",
       "       [0.34587905, 0.96877618],\n",
       "       [0.36774945, 0.41784615],\n",
       "       [0.12685561, 0.78575172],\n",
       "       [0.48829278, 0.35593762]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# create a numpy array with random numbers\n",
    "x = np.random.rand(10)\n",
    "x.shape\n",
    "x.reshape((5,-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "77844449",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'%.3f'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "array([[0.264, 0.949],\n",
       "       [0.346, 0.969],\n",
       "       [0.368, 0.418],\n",
       "       [0.127, 0.786],\n",
       "       [0.488, 0.356]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%precision 3\n",
    "x.reshape((5,-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "c3777c14",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>77516</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>2174</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>83311</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>215646</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>Private</td>\n",
       "      <td>234721</td>\n",
       "      <td>11th</td>\n",
       "      <td>7</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>Private</td>\n",
       "      <td>338409</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Wife</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>Cuba</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0                  1       2           3   4                    5   \\\n",
       "0  39          State-gov   77516   Bachelors  13        Never-married   \n",
       "1  50   Self-emp-not-inc   83311   Bachelors  13   Married-civ-spouse   \n",
       "2  38            Private  215646     HS-grad   9             Divorced   \n",
       "3  53            Private  234721        11th   7   Married-civ-spouse   \n",
       "4  28            Private  338409   Bachelors  13   Married-civ-spouse   \n",
       "\n",
       "                   6               7       8        9     10  11  12  \\\n",
       "0        Adm-clerical   Not-in-family   White     Male  2174   0  40   \n",
       "1     Exec-managerial         Husband   White     Male     0   0  13   \n",
       "2   Handlers-cleaners   Not-in-family   White     Male     0   0  40   \n",
       "3   Handlers-cleaners         Husband   Black     Male     0   0  40   \n",
       "4      Prof-specialty            Wife   Black   Female     0   0  40   \n",
       "\n",
       "               13      14  \n",
       "0   United-States   <=50K  \n",
       "1   United-States   <=50K  \n",
       "2   United-States   <=50K  \n",
       "3   United-States   <=50K  \n",
       "4            Cuba   <=50K  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>32556</th>\n",
       "      <td>27</td>\n",
       "      <td>Private</td>\n",
       "      <td>257302</td>\n",
       "      <td>Assoc-acdm</td>\n",
       "      <td>12</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Tech-support</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>38</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32557</th>\n",
       "      <td>40</td>\n",
       "      <td>Private</td>\n",
       "      <td>154374</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32558</th>\n",
       "      <td>58</td>\n",
       "      <td>Private</td>\n",
       "      <td>151910</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Unmarried</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32559</th>\n",
       "      <td>22</td>\n",
       "      <td>Private</td>\n",
       "      <td>201490</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32560</th>\n",
       "      <td>52</td>\n",
       "      <td>Self-emp-inc</td>\n",
       "      <td>287927</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>15024</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       0              1       2            3   4                    5   \\\n",
       "32556  27        Private  257302   Assoc-acdm  12   Married-civ-spouse   \n",
       "32557  40        Private  154374      HS-grad   9   Married-civ-spouse   \n",
       "32558  58        Private  151910      HS-grad   9              Widowed   \n",
       "32559  22        Private  201490      HS-grad   9        Never-married   \n",
       "32560  52   Self-emp-inc  287927      HS-grad   9   Married-civ-spouse   \n",
       "\n",
       "                       6           7       8        9      10  11  12  \\\n",
       "32556        Tech-support        Wife   White   Female      0   0  38   \n",
       "32557   Machine-op-inspct     Husband   White     Male      0   0  40   \n",
       "32558        Adm-clerical   Unmarried   White   Female      0   0  40   \n",
       "32559        Adm-clerical   Own-child   White     Male      0   0  20   \n",
       "32560     Exec-managerial        Wife   White   Female  15024   0  40   \n",
       "\n",
       "                   13      14  \n",
       "32556   United-States   <=50K  \n",
       "32557   United-States    >50K  \n",
       "32558   United-States   <=50K  \n",
       "32559   United-States   <=50K  \n",
       "32560   United-States    >50K  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(32561, 15)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "census_df = pd.read_csv(filepath_or_buffer='../dat/adult.csv', header = None)\n",
    "census_df.head()\n",
    "census_df.tail()\n",
    "census_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d99563f1",
   "metadata": {},
   "source": [
    "### Uninstall packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "cd13cba4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found existing installation: numpy 1.23.0\n",
      "Uninstalling numpy-1.23.0:\n",
      "  Successfully uninstalled numpy-1.23.0\n",
      "Found existing installation: pandas 1.4.3\n",
      "Uninstalling pandas-1.4.3:\n",
      "  Successfully uninstalled pandas-1.4.3\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "# Uninstall a package\n",
    "%pip uninstall numpy pandas --yes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "3dba9166",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files removed: 5\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip cache purge"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "5927eeb448d79922bc8ff2e0b29f33a28322850e36c32320028d81390f797ebb"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}