Accessing-Creating-Data-from-Files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ed522008",
   "metadata": {},
   "source": [
    "## How to read various files format with Python\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf026cb0",
   "metadata": {},
   "source": [
    "* Data is acquired in various formats, mainly depending on its content. Therefore, to succeed in the data science industry one needs to read, analyze and interpret data as efficiently as possible. Herein, we summarise how different file formats can be accessed via Python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6590f4a",
   "metadata": {},
   "source": [
    "### Reading CSV Files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00ed12f1",
   "metadata": {},
   "source": [
    "#### CSV.READER"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "aa3d0bf1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ef4523e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Name', '     \"Sex\"', ' \"Age\"', ' \"Height (in)\"', ' \"Weight (lbs)\"']\n",
      "[['Alex', '       \"M\"', '   41', '       74', '      170'], ['Bert', '       \"M\"', '   42', '       68', '      166'], ['Carl', '       \"M\"', '   32', '       70', '      155'], ['Dave', '       \"M\"', '   39', '       72', '      167'], ['Elly', '       \"F\"', '   30', '       66', '      124'], ['Fran', '       \"F\"', '   33', '       66', '      115'], ['Gwen', '       \"F\"', '   26', '       64', '      121'], ['Hank', '       \"M\"', '   30', '       71', '      158'], ['Ivan', '       \"M\"', '   53', '       72', '      175'], ['Jake', '       \"M\"', '   32', '       69', '      143'], ['Kate', '       \"F\"', '   47', '       69', '      139'], ['Luke', '       \"M\"', '   34', '       72', '      163'], ['Myra', '       \"F\"', '   23', '       62', '       98'], ['Neil', '       \"M\"', '   36', '       75', '      160'], ['Omar', '       \"M\"', '   38', '       70', '      145'], ['Page', '       \"F\"', '   31', '       67', '      135'], ['Quin', '       \"M\"', '   29', '       71', '      176'], ['Ruth', '       \"F\"', '   28', '       65', '      131'], []]\n"
     ]
    }
   ],
   "source": [
    "file = open('biostats.csv')\n",
    "csvreader = csv.reader(file)\n",
    "\n",
    "# Extract headers\n",
    "header = []\n",
    "header = next(csvreader)\n",
    "header\n",
    "\n",
    "# Extral actual data points\n",
    "rows = []\n",
    "for row in csvreader:\n",
    "        rows.append(row)\n",
    "rows\n",
    "\n",
    "print(header)\n",
    "print(rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4160ca70",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\"Name\",     \"Sex\", \"Age\", \"Height (in)\", \"Weight (lbs)\"\\n']\n",
      "['\"Alex\",       \"M\",   41,       74,      170\\n', '\"Bert\",       \"M\",   42,       68,      166\\n', '\"Carl\",       \"M\",   32,       70,      155\\n', '\"Dave\",       \"M\",   39,       72,      167\\n', '\"Elly\",       \"F\",   30,       66,      124\\n', '\"Fran\",       \"F\",   33,       66,      115\\n', '\"Gwen\",       \"F\",   26,       64,      121\\n', '\"Hank\",       \"M\",   30,       71,      158\\n', '\"Ivan\",       \"M\",   53,       72,      175\\n', '\"Jake\",       \"M\",   32,       69,      143\\n', '\"Kate\",       \"F\",   47,       69,      139\\n', '\"Luke\",       \"M\",   34,       72,      163\\n', '\"Myra\",       \"F\",   23,       62,       98\\n', '\"Neil\",       \"M\",   36,       75,      160\\n', '\"Omar\",       \"M\",   38,       70,      145\\n', '\"Page\",       \"F\",   31,       67,      135\\n', '\"Quin\",       \"M\",   29,       71,      176\\n', '\"Ruth\",       \"F\",   28,       65,      131\\n', '\\n']\n"
     ]
    }
   ],
   "source": [
    "with open('biostats.csv') as file:\n",
    "    content = file.readlines()\n",
    "header = content[:1] \n",
    "rows = content[1:]\n",
    "print(header)\n",
    "print(rows)\n",
    "#where 'n' denotes the end of a row"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "739bbc75",
   "metadata": {},
   "source": [
    "#### Pandas Library"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccc67d2b",
   "metadata": {},
   "source": [
    "Although the previous method seems to work fine, we need to consider the complexity of huge datasets. When analysing big datasets it is essential to be able to access the data easily and efficiently, in that regard the most straight-forward way to achieve this, is via the \"pandas\" library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "33acbaca",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/8k/hzrqv57s5l9cczbhnbv3bq800000gn/T/ipykernel_3924/3898047633.py:2: FutureWarning: The default value of regex will change from True to False in a future version.\n",
      "  dataset1.columns = dataset1.columns.str.replace('[#,@,&,\"\",'', ]', '')\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>Height(in)</th>\n",
       "      <th>Weight(lbs)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alex</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>41</td>\n",
       "      <td>74</td>\n",
       "      <td>170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Bert</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>42</td>\n",
       "      <td>68</td>\n",
       "      <td>166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Carl</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>32</td>\n",
       "      <td>70</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Dave</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>39</td>\n",
       "      <td>72</td>\n",
       "      <td>167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Elly</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>30</td>\n",
       "      <td>66</td>\n",
       "      <td>124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Fran</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>33</td>\n",
       "      <td>66</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Gwen</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>26</td>\n",
       "      <td>64</td>\n",
       "      <td>121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Hank</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>30</td>\n",
       "      <td>71</td>\n",
       "      <td>158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Ivan</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>53</td>\n",
       "      <td>72</td>\n",
       "      <td>175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Jake</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>32</td>\n",
       "      <td>69</td>\n",
       "      <td>143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Kate</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>47</td>\n",
       "      <td>69</td>\n",
       "      <td>139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Luke</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>34</td>\n",
       "      <td>72</td>\n",
       "      <td>163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Myra</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>23</td>\n",
       "      <td>62</td>\n",
       "      <td>98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Neil</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>36</td>\n",
       "      <td>75</td>\n",
       "      <td>160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Omar</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>38</td>\n",
       "      <td>70</td>\n",
       "      <td>145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Page</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>31</td>\n",
       "      <td>67</td>\n",
       "      <td>135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Quin</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>29</td>\n",
       "      <td>71</td>\n",
       "      <td>176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Ruth</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>28</td>\n",
       "      <td>65</td>\n",
       "      <td>131</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Name         Sex  Age  Height(in)  Weight(lbs)\n",
       "0   Alex         \"M\"   41          74          170\n",
       "1   Bert         \"M\"   42          68          166\n",
       "2   Carl         \"M\"   32          70          155\n",
       "3   Dave         \"M\"   39          72          167\n",
       "4   Elly         \"F\"   30          66          124\n",
       "5   Fran         \"F\"   33          66          115\n",
       "6   Gwen         \"F\"   26          64          121\n",
       "7   Hank         \"M\"   30          71          158\n",
       "8   Ivan         \"M\"   53          72          175\n",
       "9   Jake         \"M\"   32          69          143\n",
       "10  Kate         \"F\"   47          69          139\n",
       "11  Luke         \"M\"   34          72          163\n",
       "12  Myra         \"F\"   23          62           98\n",
       "13  Neil         \"M\"   36          75          160\n",
       "14  Omar         \"M\"   38          70          145\n",
       "15  Page         \"F\"   31          67          135\n",
       "16  Quin         \"M\"   29          71          176\n",
       "17  Ruth         \"F\"   28          65          131"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset1 = pd.read_csv('biostats.csv', delimiter = ',')\n",
    "dataset1.columns = dataset1.columns.str.replace('[#,@,&,\"\",'', ]', '')\n",
    "dataset1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "59704b70-c927-47d0-8946-58828cf90f42",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "41"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset1.Age[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "789e3a37",
   "metadata": {},
   "source": [
    "The pandas library also allows the user to access specific columns and therefore permits him to focus with any variable of interest. For example we could compare the age of people within our dataset as follows. In this case, we can notice that majority of participants are young adults aged between 30-35 years old."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "bd0faac3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'Age')"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlwAAAHlCAYAAAAgMSKQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAdzUlEQVR4nO3dfbRld1kf8O/DhEDIQEReLhiUCdIiSlpsroiidAZRo7BEaEFT5K10ja2CUVMh2hewloosgli0KCoFBZmFvNkSxcCqw4tCcCaCAwaEQkATCCISGIxg4Okf54QO40wyZ+757ZN78vmsddfcs8+++/fMk31XvrP37/x2dXcAABjnFqsuAABg3QlcAACDCVwAAIMJXAAAgwlcAACDCVwAAIOdsuoCbswd73jH3rVr16rLmMRnPvOZnH766asuY23o5/Lp6XLp5/Lp6XLp5+IOHjz48e6+09Hbb/KBa9euXTlw4MCqy5jE/v37s3v37lWXsTb0c/n0dLn0c/n0dLn0c3FV9aFjbXdLEQBgMIELAGAwgQsAYDCBCwBgMIELAGAwgQsAYDCBCwBgMIELAGAwgQsAYDCBCwBgMIELAGAwgQsAYDCBCwBgMIELAGAwgQsAYLDJA1dV/VhVvbuq3lVVL6uqW09dAwDAlCYNXFV1ZpIfSbLZ3fdJsiPJ909ZAwDA1FZxS/GUJKdV1SlJbpPkqhXUAAAwmUkDV3dfmeTZST6c5CNJrunuS6asAQBgatXd0w1Wdfskr0zyfUk+meS3k7yiu19y1H57k+xNko2NjXP27ds3WY2rdPjw4ezcuXPVZdwkHLrymi0fY+O05Oprl1AMX3TWGTuco0vkd3759HS59HNxe/bsOdjdm0dvnzpwPTLJud39xPnrxya5f3f/0PF+ZnNzsw8cODBViSu1f//+7N69e9Vl3CTsuvDiLR/jgrOvy0WHTllCNVzvReee7hxdIr/zy6eny6Wfi6uqYwauqedwfTjJ/avqNlVVSb4tyeUT1wAAMKmp53BdmuQVSS5Lcmg+/gumrAEAYGqT32/p7qcledrU4wIArIqV5gEABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABps0cFXVvarqHUd8faqqfnTKGgAApnbKlIN193uT3DdJqmpHkiuTvHrKGgAAprbKW4rfluT/dveHVlgDAMBw1d2rGbjqhUku6+5fPMZ7e5PsTZKNjY1z9u3bN3V5K3H48OHs3Llz1WXcJBy68potH2PjtOTqa5dQDF901hk7nKNL5Hd++fR0ufRzcXv27DnY3ZtHb19J4KqqU5NcleTruvvqG9p3c3OzDxw4ME1hK7Z///7s3r171WXcJOy68OItH+OCs6/LRYcmvWu+9l507unO0SXyO798erpc+rm4qjpm4FrVLcXvyuzq1g2GLQCAdbCqwHVekpetaGwAgElNHriq6jZJvj3Jq6YeGwBgFSaf4NLdf5vkDlOPCwCwKlaaBwAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGGzywFVVX1ZVr6iq91TV5VX1TVPXAAAwpVNWMOYvJHldd//Lqjo1yW1WUAMAwGQmDVxVdbskD0zy+CTp7s8l+dyUNQAATG3qW4r3SPJXSf5nVf1JVf1aVZ0+cQ0AAJOq7p5usKrNJG9L8oDuvrSqfiHJp7r7Px21394ke5NkY2PjnH379k1W4yodPnw4O3fuXHUZNwmHrrxmy8fYOC25+tolFMMXnXXGDufoEvmdXz49XS79XNyePXsOdvfm0dunDlx3SfK27t41f/2tSS7s7occ72c2Nzf7wIEDE1W4Wvv378/u3btXXcZNwq4LL97yMS44+7pcdGgV0xTX14vOPd05ukR+55dPT5dLPxdXVccMXJPeUuzujyb5i6q613zTtyX5sylrAACY2ir++f/kJC+df0LxA0mesIIaAAAmM3ng6u53JPkHl9oAANaVleYBAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAY7ZeoBq+qKJJ9O8vkk13X35tQ1AABMafLANbenuz++orEBACblliIAwGCrCFyd5JKqOlhVe1cwPgDApKq7px2w6iu6+6qqunOS1yd5cne/6ah99ibZmyQbGxvn7Nu3b9IaV+Xw4cPZuXPnqsu4STh05TVbPsbGacnV1y6hGL7orDN2OEeXyO/88unpcunn4vbs2XPwWPPTJw9cXzJ41dOTHO7uZx9vn83NzT5w4MB0Ra3Q/v37s3v37lWXcZOw68KLt3yMC86+LhcdWtU0xfX0onNPd44ukd/55dPT5dLPxVXVMQPXpLcUq+r0qrrt9d8n+Y4k75qyBgCAqS0UuKrqGVV19y2Mt5HkLVX1ziRvT3Jxd79uC8cDALjJW/R+y48keWpVXZLkl5O8tru/cKI/3N0fSPJPFxwTAGBbW/SW4l2S/HBmV6pek+RDVfW0qjpz2YUBAKyLhQJXd3+mu3+lu89J8o1JLknyE0k+WFWvrqpzRxQJALCdnfSk+e7+4+5+YpKzkvxRkoclubiqPlBVP1xVFlUFAMgWAldVfXVVPSvJu5N8c5JXJ3l0krcmeW5mc7wAAG72Fpo0X1U7kjw8yQ8m2ZPk6iTPT/Ir3X3VfLd9VfXmJD+X+eKlAAA3Z4t+SvHKJHdK8qYk5yV5dXdfd4z9/iTJbbdYGwDAWlg0cP12kv/R3Zff0E7dfWk8GBsAIMmCgau7nzyqEACAdbXoSvNPrarnHee9/15VP7GcsgAA1seit/2ekORPj/PeO+bvAwBwhEUD11cled9x3vtAkq08ZxEAYC0tGrj+NsnxHuNztySf3Vo5AADrZ9HA9eYkP1FVtzpy4/z1BfP3AQA4wqLLQjw9s8f4/HlVvSSzdbnOTPIDSe6Q5PHLLA4AYB0suizEO6tqT5JnJ3lqZlfIvpDkLUn+RXe/c/klAgBsb4te4Up3vz3JA6vqtCS3T/I33X3t0isDAFgTCweu681DlqAFAHAjFg5cVXWPJI/KbImIWx/1dnf3E5dRGADAulgocFXVwzJ7nuItknws/3AZiF5SXQAAa2PRK1z/Ncn+JI/u7r9afjkAAOtn0cB1jyQXCFsAACdu0YVP35PZelsAAJygRQPXU5L81HziPAAAJ+BkVpq/Q5LLq+p9ST5x1Pvd3f98GYUBAKyLRQPX55O8d0QhAADratFH++weVAcAwNpadA4XAAALWjhwVdWZVfWcqjpQVR+sqvvMt/9oVX3j8ksEANjeFgpcVfV1SQ4leUySqzJ7vM+p87fvnuT8pVYHALAGFr3CdVGSy5OcleQRSeqI9/4oyf2XVBcAwNpY9FOK35LkvO4+XFU7jnrv6iR3WU5ZAADrY9ErXF+4gffumOTaLdQCALCWFg1cb0/yhOO896gkf7i1cgAA1s+itxR/JskbquqSJL+VpJM8uKrOT/LwJA9ccn0AANveQle4uvuNSb43s0nzL8xs0vwzk3xrku/t7kuXXSAAwHa36BWudPfFSS6uqnsmuXOSv+5uj/sBADiOhQPX9br7/Unev8RaAADW0kKBq6oee2P7dPdvnHw5AADrZ9ErXC86zvY+4nuBCwDgCIsGrrOOse0OSR6a5F8l+YEtVwQAsGYWClzd/aFjbP5QksuqqpL8eGbBCwCAuUUXPr0hb07ykCUeDwBgLSwzcN0/yeElHg8AYC0s+inF/3yMzacmuU9mV7d+8QSPsyPJgSRXdvdDF6kBAGC7WXTS/NOPse2zmc3jekaSnz3B45yf5PIkt1twfACAbWfRSfNbvgVZVXfL7GrYMzKbZA8AsNaWOYfrRD03yVOSfGEFYwMATK66+8b3un7nqq9a5ODd/eGjfv6hSb67u3+oqnYn+ffHmsNVVXuT7E2SjY2Nc/bt27fIsNvW4cOHs3PnzlWXcZNw6MprtnyMjdOSq69dQjF80Vln7HCOLpHf+eXT0+XSz8Xt2bPnYHdvHr190cD1hXzpqvI3qLt3HPXzP5vkMUmuS3LrzOZwvaq7j7tg6ubmZh84cOCEa9zO9u/fn927d6+6jJuEXRdevOVjXHD2dbno0Ek/LpRjeNG5pztHl8jv/PLp6XLp5+Kq6piBa9H/G/27JP8hyaeSvDzJ1UnukuRRSXZmNi/rs8f74e7+ySQ/OS9od2ZXuKxODwCstUUD172TXJbk4X3EpbGq+i9JXpPk3t39Y8srDwBg+1t00vx5SX6lj7oPOX/9y1ngsT7dvd8aXADAzcGigWtnkjsd5707Jzl9a+UAAKyfRQPX/iT/raq+4ciNVXW/zOZv7V9OWQAA62PRwPWkzCbFv62qrqiqS6vqiiRvTfJ38/cBADjCoivNf7CqvibJ4zN7WPVdk7wrs8D14u7++6VXCACwzS28SNE8VP3q/AsAgBtxUqtCVtU/SfLAJHfI7FOLH62qeya5urs/vcwCAQC2u4UCV1XdKslLkjwiSWW26vz/TvLRJM9K8udJLlxyjQAA29qik+afkeTBmT2eZyOz0HW930vynUuqCwBgbSx6S/G8JP+xu3+rqnYc9d4Hk+xaSlUAAGtk0Stcd0hy+Q0c61ZbKwcAYP0sGrg+mOSbjvPe/ZK8d2vlAACsn0UD128kubCqHp3k1Pm2rqo9SX4syQuXWRwAwDpYNHA9K8nFSX4zySfm296S5A1JXtfdz1tibQAAa2HRleY/n+T7q+qXMvtE4p2T/HVmYeuNA+oDANj2TjhwVdWpSd6W5MLuviTJm4dVBQCwRk74lmJ3fy7JWUmuG1cOAMD6WXQO1+uTfMeIQgAA1tWiC58+L8lLquqUJK9J8pHMHu/zRd39geWUBgCwHhYNXNdPjP/xzJaBOJajV6AHALhZu9HAVVUPSvL27j6c5F/nqCtaAADcsBO5wvX6zFaXf3t3v6iqbpFkf5Indvf7RhYHALAOTmTSfB3j9bckue3yywEAWD+LfkoRAIAFCVwAAIOd6KcUz6yqe8y/33HEtk8evaNlIQAAvtSJBq5XHGPba46zr2UhAACOcCKB6wnDqwAAWGM3Gri6+8VTFAIAsK5MmgcAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhs0sBVVbeuqrdX1Tur6t1V9dNTjg8AsAqnTDzeZ5M8qLsPV9Utk7ylqn6vu982cR0AAJOZNHB1dyc5PH95y/lXT1kDAMDUapaBJhywakeSg0numeSXuvupx9hnb5K9SbKxsXHOvn37htZ06Mprhh7/RG2cllx97aqrWB/6uXx6ulzL6OfZZ56xnGLWxOHDh7Nz585Vl7E29HNxe/bsOdjdm0dvnzxwfXHgqi9L8uokT+7udx1vv83NzT5w4MDQWnZdePHQ45+oC86+Lhcdmvou7/rSz+XT0+VaRj+veOZDllTNeti/f39279696jLWhn4urqqOGbhW9inF7v5kkv1Jzl1VDQAAU5j6U4p3ml/ZSlWdluTBSd4zZQ0AAFOb+t7AXZO8eD6P6xZJXt7dr524BgCASU39KcU/TfL1U44JALBqVpoHABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhM4AIAGEzgAgAYTOACABhs0sBVVV9ZVX9QVZdX1bur6vwpxwcAWIVTJh7vuiQXdPdlVXXbJAer6vXd/WcT1wEAMJlJr3B190e6+7L5959OcnmSM6esAQBgatXdqxm4aleSNyW5T3d/6qj39ibZmyQbGxvn7Nu3b2gth668ZujxT9TGacnV1666ivWhn8unp8uln8unp8u1Tv08+8wzJhlnz549B7t78+jtKwlcVbUzyRuTPKO7X3VD+25ubvaBAweG1rPrwouHHv9EXXD2dbno0NR3edeXfi6fni6Xfi6fni7XOvXzimc+ZJJxquqYgWvyTylW1S2TvDLJS28sbAEArIOpP6VYSX49yeXd/ZwpxwYAWJWpr3A9IMljkjyoqt4x//ruiWsAAJjUpDdmu/stSWrKMQEAVs1K8wAAgwlcAACDCVwAAIMJXAAAgwlcAACDCVwAAIMJXAAAgwlcAACDCVwAAIMJXAAAgwlcAACDCVwAAIMJXAAAgwlcAACDCVwAAIMJXAAAgwlcAACDCVwAAIMJXAAAgwlcAACDCVwAAIMJXAAAgwlcAACDCVwAAIMJXAAAgwlcAACDCVwAAIMJXAAAgwlcAACDCVwAAIMJXAAAgwlcAACDCVwAAIMJXAAAgwlcAACDCVwAAIMJXAAAgwlcAACDCVwAAIMJXAAAgwlcAACDCVwAAINNGriq6oVV9bGqeteU4wIArNLUV7helOTciccEAFipSQNXd78pySemHBMAYNXM4QIAGKy6e9oBq3YleW133+cG9tmbZG+SbGxsnLNv376hNR268pqhxz9RG6clV1+76irWh34un54ul34un54u1zr18+wzz5hknD179hzs7s2jt58yyegL6u4XJHlBkmxubvbu3buHjvf4Cy8eevwTdcHZ1+WiQzfJ/yTbkn4un54ul34un54u1zr184pH717p+G4pAgAMNvWyEC9L8tYk96qqv6yqJ045PgDAKkx6nbC7z5tyPACAmwK3FAEABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABhO4AAAGE7gAAAYTuAAABps8cFXVuVX13qp6f1VdOPX4AABTmzRwVdWOJL+U5LuSfG2S86rqa6esAQBgalNf4bpfkvd39we6+3NJ9iV52MQ1AABMaurAdWaSvzji9V/OtwEArK3q7ukGq3pkku/s7n8zf/2YJPfr7icftd/eJHvnL++V5L2TFblad0zy8VUXsUb0c/n0dLn0c/n0dLn0c3F37+47Hb3xlImL+MskX3nE67slueronbr7BUleMFVRNxVVdaC7N1ddx7rQz+XT0+XSz+XT0+XSz+WZ+pbiHyf5R1V1VlWdmuT7k/yviWsAAJjUpFe4uvu6qnpSkt9PsiPJC7v73VPWAAAwtalvKaa7fzfJ70497jZxs7uNOph+Lp+eLpd+Lp+eLpd+Lsmkk+YBAG6OPNoHAGAwgWsFquorq+oPquryqnp3VZ0/3/70qrqyqt4x//ruVde6XVTVravq7VX1znlPf3q+/cur6vVV9b75n7dfda3bwQ300zm6BVW1o6r+pKpeO3/t/NyiY/TUOboFVXVFVR2a9+7AfJvzdAncUlyBqrprkrt292VVddskB5N8b5JHJTnc3c9eZX3bUVVVktO7+3BV3TLJW5Kcn+QRST7R3c+cP7vz9t391FXWuh3cQD/PjXP0pFXVjyfZTHK77n5oVT0rzs8tOUZPnx7n6EmrqiuSbHb3x4/Y5jxdAle4VqC7P9Ldl82//3SSy2PF/S3pmcPzl7ecf3Vmj4568Xz7izMLttyIG+gnJ6mq7pbkIUl+7YjNzs8tOE5PWT7n6RIIXCtWVbuSfH2SS+ebnlRVf1pVL3TZdjHzWwvvSPKxJK/v7kuTbHT3R5JZ0E1y5xWWuK0cp5+Jc/RkPTfJU5J84Yhtzs+teW7+YU8T5+hWdJJLqurg/KkvifN0KQSuFaqqnUlemeRHu/tTSZ6f5KuT3DfJR5JctLrqtp/u/nx33zezJxjcr6rus+KStrXj9NM5ehKq6qFJPtbdB1ddy7q4gZ46R7fmAd39z5J8V5IfrqoHrrqgdSFwrch8Xswrk7y0u1+VJN199fx/cl9I8qtJ7rfKGrer7v5kkv2ZzTe6ej5n7vq5cx9bXWXb05H9dI6etAck+Z75/Jh9SR5UVS+J83MrjtlT5+jWdPdV8z8/luTVmfXPeboEAtcKzCck/3qSy7v7OUdsv+sRuz08ybumrm27qqo7VdWXzb8/LcmDk7wns0dHPW6+2+OS/M5KCtxmjtdP5+jJ6e6f7O67dfeuzB5p9n+6+wfi/Dxpx+upc/TkVdXp8w9ypapOT/IdmfXPeboEk680T5LZv8wek+TQfI5MkvxUkvOq6r6Z3UO/IskPrqK4bequSV5cVTsy+4fEy7v7tVX11iQvr6onJvlwkkeussht5Hj9/E3n6FI9M87PZXuWc/SkbSR59eyaQE5J8lvd/bqq+uM4T7fMshAAAIO5pQgAMJjABQAwmMAFADCYwAUAMJjABQAwmMAFrIWq+rWq6qp6zo3vDTAty0IA2958cdaPJrldZqtgn9nd1622KoD/zxUuYB08PLOw9buZPVj33NWWA/ClBC5gHTwuyd8keXySa5M89ugdquq8qnpPVf1dVR2qqu+pqv1Vtf+o/e5YVc+vqiur6rPzn9k7xV8CWF8e7QNsa1X1FZk96/EF3f1XVfWaJI+oqtt399/M9/n2JC/N7JlwFyS5Y5LnJrl1kj8/4li3S/KHSU5L8vQkH0zynUmeX1W36u7nTfTXAtaMwAVsd4/J7Gr9b8xfvzjJeUm+L8kvz7f9dJI/S/Lwnk9crapDSQ7miMCV5Pwkd09ydne/b77tDfMHeT+tqp5vbhhwMtxSBLa7xyZ5X3e/df76DUmumm/P/AHcm0le2Ud8Sqi7L8vsCtaRzk1yaZIPVtUp138l+f0kd0jytUP/JsDacoUL2Laq6hsyC0E/N78Kdb1XJXlSVf3jJNckuWVmn1482tVHvb5zknsm+fvjDHmHLRUM3GwJXMB29rj5n0+dfx3tsUmellmAuvMx3t9I8uEjXv91ZsHs/OOM996TKxO4ubMOF7AtVdWpmd06fH+SC4+xy88n+fIkuzKbCH+7zOZmXT+H65wkB5K8sbt3z7c9PcmTk9y7u491RQzgpAhcwLZUVY9I8sokj+/uFx/j/X+b5PlJHpTZ1fxLkvxOkhdk9inFp2f2acTLu/tB8585I8nbMpvf+vOZXdE6PcnXJPnW7n7Y2L8VsK5Mmge2q8cl+XSS3z7O+y/LbE2ux3X365M8Osm9k7w6s9uPF2S2Ov011/9Ad1+T5JszW0D1qZlNln9hkocl+YMhfwvgZsEVLuBmqarultntyGd098+suh5gvQlcwNqbP2vxOZktGfHxJPdI8pTMJs1/XXd/ZIXlATcDPqUI3Bx8PsldkvxiZks7fCbJm5M8UtgCpuAKFwDAYCbNAwAMJnABAAwmcAEADCZwAQAMJnABAAwmcAEADPb/ACn7XpJUHQNoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure ( figsize = (10,8))\n",
    "dataset1.Age\n",
    "dataset1.Age.hist(bins = 5)\n",
    "plt.ylabel('Frequency', fontsize = 16)\n",
    "plt.xlabel('Age', fontsize = 16)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6338e469",
   "metadata": {},
   "source": [
    "### Creating  CSV Files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c33fdfc",
   "metadata": {},
   "source": [
    "Let's assume we log daily average temperature and air quality data for the last week"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bb03341",
   "metadata": {},
   "source": [
    "#### CSV.WRITER"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "7bbf1874",
   "metadata": {},
   "outputs": [],
   "source": [
    "header = ['Day', 'Temperature (Celsius)', 'Air Quality Index']\n",
    "\n",
    "rows = [['Monday 01/10', 25.2, 150], ['Tuesday 02/10', 27.3, 160], ['Wendnesday 03/10', 26, 165],\n",
    "        ['Thursday 04/10', 24.6, 160], ['Friday 05/10', 22.8, 142], ['Saturday 06/10', 23, 155],\n",
    "        ['Sunday 07/10', 24.6, 145]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "47e09d48",
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = 'DailyMeteo.csv'\n",
    "\n",
    "with open(filename, 'w', newline=\"\") as file:\n",
    "    csvwriter = csv.writer(file) # create a csvwriter object\n",
    "    csvwriter.writerow(header)   # write the header\n",
    "    csvwriter.writerows(rows)    # write the rest of the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f227a16",
   "metadata": {},
   "source": [
    "#### Using Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "753923ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "df= pd.DataFrame(rows, columns=header)\n",
    "df.to_csv('Dailymeteo.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30b2d6f0",
   "metadata": {},
   "source": [
    "### Now let us focus a bit more on the Pandas library\n",
    "\n",
    "*Owing to their flexibility Pandas library is widely used in data science, machine learning, scientific computing, and many other data-intensive fields*\n",
    "\n",
    "* Assuming that our manager informed us that the biostats csv file, lacks some of the data we want to invgestigate. Hence he supplied it to use in a different CSV file and asked us to now carry out the data analysis for the complete dataset  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "37c99e79",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>Height(in)</th>\n",
       "      <th>Weight(lbs)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>John</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>29</td>\n",
       "      <td>75</td>\n",
       "      <td>164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Freddie</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>33</td>\n",
       "      <td>69</td>\n",
       "      <td>150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Jacob</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>46</td>\n",
       "      <td>84</td>\n",
       "      <td>172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Helen</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>52</td>\n",
       "      <td>62</td>\n",
       "      <td>134</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Ellie</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>30</td>\n",
       "      <td>66</td>\n",
       "      <td>124</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Name  Sex  Age  Height(in)  Weight(lbs)\n",
       "0     John  \"M\"   29          75          164\n",
       "1  Freddie  \"M\"   33          69          150\n",
       "2    Jacob  \"M\"   46          84          172\n",
       "3    Helen  \"F\"   52          62          134\n",
       "4    Ellie  \"F\"   30          66          124"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset2 = pd.read_csv('extrabiostats.csv', ',')\n",
    "dataset2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d64213c",
   "metadata": {},
   "source": [
    "We can combine datasets 1 and 2 as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "97c11a0d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>Height(in)</th>\n",
       "      <th>Weight(lbs)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alex</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>41</td>\n",
       "      <td>74</td>\n",
       "      <td>170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Bert</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>42</td>\n",
       "      <td>68</td>\n",
       "      <td>166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Carl</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>32</td>\n",
       "      <td>70</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Dave</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>39</td>\n",
       "      <td>72</td>\n",
       "      <td>167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Elly</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>30</td>\n",
       "      <td>66</td>\n",
       "      <td>124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Fran</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>33</td>\n",
       "      <td>66</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Gwen</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>26</td>\n",
       "      <td>64</td>\n",
       "      <td>121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Hank</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>30</td>\n",
       "      <td>71</td>\n",
       "      <td>158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Ivan</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>53</td>\n",
       "      <td>72</td>\n",
       "      <td>175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Jake</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>32</td>\n",
       "      <td>69</td>\n",
       "      <td>143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Kate</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>47</td>\n",
       "      <td>69</td>\n",
       "      <td>139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Luke</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>34</td>\n",
       "      <td>72</td>\n",
       "      <td>163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Myra</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>23</td>\n",
       "      <td>62</td>\n",
       "      <td>98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Neil</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>36</td>\n",
       "      <td>75</td>\n",
       "      <td>160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Omar</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>38</td>\n",
       "      <td>70</td>\n",
       "      <td>145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Page</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>31</td>\n",
       "      <td>67</td>\n",
       "      <td>135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Quin</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>29</td>\n",
       "      <td>71</td>\n",
       "      <td>176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Ruth</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>28</td>\n",
       "      <td>65</td>\n",
       "      <td>131</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>John</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>29</td>\n",
       "      <td>75</td>\n",
       "      <td>164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Freddie</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>33</td>\n",
       "      <td>69</td>\n",
       "      <td>150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Jacob</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>46</td>\n",
       "      <td>84</td>\n",
       "      <td>172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Helen</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>52</td>\n",
       "      <td>62</td>\n",
       "      <td>134</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Ellie</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>30</td>\n",
       "      <td>66</td>\n",
       "      <td>124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Zack</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>36</td>\n",
       "      <td>75</td>\n",
       "      <td>162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Carol</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>25</td>\n",
       "      <td>58</td>\n",
       "      <td>120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Omer</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>52</td>\n",
       "      <td>73</td>\n",
       "      <td>146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Elspeth</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>37</td>\n",
       "      <td>61</td>\n",
       "      <td>132</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Ignas</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>24</td>\n",
       "      <td>64</td>\n",
       "      <td>153</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Name         Sex  Age  Height(in)  Weight(lbs)\n",
       "0      Alex         \"M\"   41          74          170\n",
       "1      Bert         \"M\"   42          68          166\n",
       "2      Carl         \"M\"   32          70          155\n",
       "3      Dave         \"M\"   39          72          167\n",
       "4      Elly         \"F\"   30          66          124\n",
       "5      Fran         \"F\"   33          66          115\n",
       "6      Gwen         \"F\"   26          64          121\n",
       "7      Hank         \"M\"   30          71          158\n",
       "8      Ivan         \"M\"   53          72          175\n",
       "9      Jake         \"M\"   32          69          143\n",
       "10     Kate         \"F\"   47          69          139\n",
       "11     Luke         \"M\"   34          72          163\n",
       "12     Myra         \"F\"   23          62           98\n",
       "13     Neil         \"M\"   36          75          160\n",
       "14     Omar         \"M\"   38          70          145\n",
       "15     Page         \"F\"   31          67          135\n",
       "16     Quin         \"M\"   29          71          176\n",
       "17     Ruth         \"F\"   28          65          131\n",
       "18     John         \"M\"   29          75          164\n",
       "19  Freddie         \"M\"   33          69          150\n",
       "20    Jacob         \"M\"   46          84          172\n",
       "21    Helen         \"F\"   52          62          134\n",
       "22    Ellie         \"F\"   30          66          124\n",
       "23     Zack         \"M\"   36          75          162\n",
       "24    Carol         \"F\"   25          58          120\n",
       "25    Omer          \"M\"   52          73          146\n",
       "26  Elspeth         \"F\"   37          61          132\n",
       "27    Ignas         \"M\"   24          64          153"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datacombined = pd.concat([dataset1, dataset2], ignore_index = True ) # ingore_index allows us to label our indices from\n",
    "                                                                     # 0 to n-1, where n is the total sample number\n",
    "datacombined"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "273dd513",
   "metadata": {},
   "source": [
    "* We notice that here our manager provided us a dataset which increases the sample number. However, it is common for someone to require us to investigate other \"parameters\" which correspond to the same people. For example let us assume, that now our manager is not only interested in the age, weight, height of our participants but also in several health habits such as diet and smoking. This data is given to us in a new CSV file.\n",
    "\n",
    "\n",
    "<u> Please take into account that all, stats, names and diet habits presented herein are created for the purposes of this script only and do not conform with the reality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "8cd54774",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Diet</th>\n",
       "      <th>Smoker</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alex</td>\n",
       "      <td>\"Pescatarian\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Bert</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Carl</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Dave</td>\n",
       "      <td>\"Pescatarian\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Elly</td>\n",
       "      <td>\"Vegan\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Fran</td>\n",
       "      <td>\"Vegetarian\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Gwen</td>\n",
       "      <td>\"Vegan\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Hank</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Ivan</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Jake</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Kate</td>\n",
       "      <td>\"Vegan\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Luke</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Myra</td>\n",
       "      <td>\"Vegetarian\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Neil</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Omar</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Page</td>\n",
       "      <td>\"Pescatarian\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Quin</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Ruth</td>\n",
       "      <td>\"Vegetarian\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>John</td>\n",
       "      <td>\"Vegetarian\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Freddie</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Jacob</td>\n",
       "      <td>\"Vegan\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Helen</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Ellie</td>\n",
       "      <td>\"Vegan\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Zack</td>\n",
       "      <td>\"Pescatarian\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Carol</td>\n",
       "      <td>\"Vegan\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Omer</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Elspeth</td>\n",
       "      <td>\"Pescatarian\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Ignas</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Name           Diet Smoker\n",
       "0      Alex  \"Pescatarian\"  \"Yes\"\n",
       "1      Bert       \"Normal\"   \"No\"\n",
       "2      Carl       \"Normal\"  \"Yes\"\n",
       "3      Dave  \"Pescatarian\"   \"No\"\n",
       "4      Elly        \"Vegan\"   \"No\"\n",
       "5      Fran   \"Vegetarian\"   \"No\"\n",
       "6      Gwen        \"Vegan\"   \"No\"\n",
       "7      Hank       \"Normal\"  \"Yes\"\n",
       "8      Ivan       \"Normal\"  \"Yes\"\n",
       "9      Jake       \"Normal\"   \"No\"\n",
       "10     Kate        \"Vegan\"   \"No\"\n",
       "11     Luke       \"Normal\"   \"No\"\n",
       "12     Myra   \"Vegetarian\"   \"No\"\n",
       "13     Neil       \"Normal\"  \"Yes\"\n",
       "14     Omar       \"Normal\"   \"No\"\n",
       "15     Page  \"Pescatarian\"   \"No\"\n",
       "16     Quin       \"Normal\"   \"No\"\n",
       "17     Ruth   \"Vegetarian\"   \"No\"\n",
       "18     John   \"Vegetarian\"  \"Yes\"\n",
       "19  Freddie       \"Normal\"   \"No\"\n",
       "20    Jacob        \"Vegan\"  \"Yes\"\n",
       "21    Helen       \"Normal\"   \"No\"\n",
       "22    Ellie        \"Vegan\"   \"No\"\n",
       "23     Zack  \"Pescatarian\"   \"No\"\n",
       "24    Carol        \"Vegan\"   \"No\"\n",
       "25     Omer       \"Normal\"  \"Yes\"\n",
       "26  Elspeth  \"Pescatarian\"   \"No\"\n",
       "27    Ignas       \"Normal\"  \"Yes\""
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "habitdata = pd.read_csv('habitstats.csv', ',')\n",
    "habitdata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d87499ee",
   "metadata": {},
   "source": [
    "Contrastingly to the previous example, here we want to combine our habitdata dataframe with the combined data frame created above. However, in this case we want to combine vertically. This is achieved as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "e2919de0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>Height(in)</th>\n",
       "      <th>Weight(lbs)</th>\n",
       "      <th>Diet</th>\n",
       "      <th>Smoker</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alex</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>41.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>\"Pescatarian\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Bert</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>42.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>166.0</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Carl</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>32.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>155.0</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Dave</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>39.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>167.0</td>\n",
       "      <td>\"Pescatarian\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Elly</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>30.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>124.0</td>\n",
       "      <td>\"Vegan\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Fran</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>33.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>115.0</td>\n",
       "      <td>\"Vegetarian\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Gwen</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>26.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>121.0</td>\n",
       "      <td>\"Vegan\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Hank</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>30.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>158.0</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Ivan</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>53.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>175.0</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Jake</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>32.0</td>\n",
       "      <td>69.0</td>\n",
       "      <td>143.0</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Kate</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>47.0</td>\n",
       "      <td>69.0</td>\n",
       "      <td>139.0</td>\n",
       "      <td>\"Vegan\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Luke</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>34.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>163.0</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Myra</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>23.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>98.0</td>\n",
       "      <td>\"Vegetarian\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Neil</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>36.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>160.0</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Omar</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>38.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Page</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>31.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>135.0</td>\n",
       "      <td>\"Pescatarian\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Quin</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>29.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>176.0</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Ruth</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>28.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>131.0</td>\n",
       "      <td>\"Vegetarian\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>John</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>29.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>164.0</td>\n",
       "      <td>\"Vegetarian\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Freddie</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>33.0</td>\n",
       "      <td>69.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Jacob</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>46.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>172.0</td>\n",
       "      <td>\"Vegan\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Helen</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>52.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>134.0</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Ellie</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>30.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>124.0</td>\n",
       "      <td>\"Vegan\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Zack</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>36.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>\"Pescatarian\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Carol</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>25.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>\"Vegan\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Omer</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>52.0</td>\n",
       "      <td>73.0</td>\n",
       "      <td>146.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Elspeth</td>\n",
       "      <td>\"F\"</td>\n",
       "      <td>37.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>132.0</td>\n",
       "      <td>\"Pescatarian\"</td>\n",
       "      <td>\"No\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Ignas</td>\n",
       "      <td>\"M\"</td>\n",
       "      <td>24.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>153.0</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Omer</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>\"Normal\"</td>\n",
       "      <td>\"Yes\"</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Name         Sex   Age  Height(in)  Weight(lbs)           Diet Smoker\n",
       "0      Alex         \"M\"  41.0        74.0        170.0  \"Pescatarian\"  \"Yes\"\n",
       "1      Bert         \"M\"  42.0        68.0        166.0       \"Normal\"   \"No\"\n",
       "2      Carl         \"M\"  32.0        70.0        155.0       \"Normal\"  \"Yes\"\n",
       "3      Dave         \"M\"  39.0        72.0        167.0  \"Pescatarian\"   \"No\"\n",
       "4      Elly         \"F\"  30.0        66.0        124.0        \"Vegan\"   \"No\"\n",
       "5      Fran         \"F\"  33.0        66.0        115.0   \"Vegetarian\"   \"No\"\n",
       "6      Gwen         \"F\"  26.0        64.0        121.0        \"Vegan\"   \"No\"\n",
       "7      Hank         \"M\"  30.0        71.0        158.0       \"Normal\"  \"Yes\"\n",
       "8      Ivan         \"M\"  53.0        72.0        175.0       \"Normal\"  \"Yes\"\n",
       "9      Jake         \"M\"  32.0        69.0        143.0       \"Normal\"   \"No\"\n",
       "10     Kate         \"F\"  47.0        69.0        139.0        \"Vegan\"   \"No\"\n",
       "11     Luke         \"M\"  34.0        72.0        163.0       \"Normal\"   \"No\"\n",
       "12     Myra         \"F\"  23.0        62.0         98.0   \"Vegetarian\"   \"No\"\n",
       "13     Neil         \"M\"  36.0        75.0        160.0       \"Normal\"  \"Yes\"\n",
       "14     Omar         \"M\"  38.0        70.0        145.0       \"Normal\"   \"No\"\n",
       "15     Page         \"F\"  31.0        67.0        135.0  \"Pescatarian\"   \"No\"\n",
       "16     Quin         \"M\"  29.0        71.0        176.0       \"Normal\"   \"No\"\n",
       "17     Ruth         \"F\"  28.0        65.0        131.0   \"Vegetarian\"   \"No\"\n",
       "18     John         \"M\"  29.0        75.0        164.0   \"Vegetarian\"  \"Yes\"\n",
       "19  Freddie         \"M\"  33.0        69.0        150.0       \"Normal\"   \"No\"\n",
       "20    Jacob         \"M\"  46.0        84.0        172.0        \"Vegan\"  \"Yes\"\n",
       "21    Helen         \"F\"  52.0        62.0        134.0       \"Normal\"   \"No\"\n",
       "22    Ellie         \"F\"  30.0        66.0        124.0        \"Vegan\"   \"No\"\n",
       "23     Zack         \"M\"  36.0        75.0        162.0  \"Pescatarian\"   \"No\"\n",
       "24    Carol         \"F\"  25.0        58.0        120.0        \"Vegan\"   \"No\"\n",
       "25    Omer          \"M\"  52.0        73.0        146.0            NaN    NaN\n",
       "26  Elspeth         \"F\"  37.0        61.0        132.0  \"Pescatarian\"   \"No\"\n",
       "27    Ignas         \"M\"  24.0        64.0        153.0       \"Normal\"  \"Yes\"\n",
       "28     Omer         NaN   NaN         NaN          NaN       \"Normal\"  \"Yes\""
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "completeset = datacombined.merge(habitdata, how='outer')\n",
    "completeset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a28021b",
   "metadata": {},
   "source": [
    "### Create and arrange series with Pandas\n",
    "\n",
    "* Assuming we have a series of data for the final grades of a school student (maximum being 20). We want to create the final template to give to the parents at the end of the school year by listing alphabetically all courses with their respective grade by their side."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "1fa3994d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ancient Greek          5\n",
       "Chemistry             10\n",
       "German                19\n",
       "Grammar               12\n",
       "Latin                 14\n",
       "Literature            15\n",
       "Mathematics           20\n",
       "Physical Education    17\n",
       "Physics               20\n",
       "Religious Studies     12\n",
       "dtype: int64"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "series_data = {'Chemistry': 10, 'Ancient Greek': 5, 'Physics': 20, 'Literature': 15, 'Mathematics': 20,\n",
    "               'Grammar': 12, 'Latin': 14, 'Physical Education': 17, 'German': 19, 'Religious Studies': 12}\n",
    "grades = pd.Series(series_data, index=sorted(series_data.keys()))\n",
    "grades"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51afdc09",
   "metadata": {},
   "source": [
    "### Missing values and data frame re-arrangement\n",
    "* For this example we consider the data frame, created before, named as \"complete set\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "4fd9e160",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25 5\n",
      "25 6\n",
      "28 1\n",
      "28 2\n",
      "28 3\n",
      "28 4\n"
     ]
    }
   ],
   "source": [
    "missing_data_count=completeset.isnull()\n",
    "for i in range(len(missing_data_count)):\n",
    "    for j in range(7):\n",
    "        if missing_data_count.iloc[i,j] == True:\n",
    "            print(i,j)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43cd093a",
   "metadata": {},
   "source": [
    "* When interpreting the results of this analysis it is essential to bear in mind the fact that python starts counting from 0 instead of 1. Hence we can see for which people we are missing statistical data either due to a human or software error.\n",
    "\n",
    "\n",
    "* In this case until we identify the source of error that resulted in the absence of the aforementioned values, we will discard these values from our analysis. Alternatively we could substitute the missing values by 0 but this would alter our interpretation of this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "21274c27",
   "metadata": {},
   "outputs": [],
   "source": [
    "completeset.drop([25,28], axis = 0, inplace = True)\n",
    "missing_data_count=completeset.isnull()\n",
    "\n",
    "\n",
    "\n",
    "for i in range(len(missing_data_count)):\n",
    "    for j in range(7):\n",
    "        if missing_data_count.iloc[i,j] == True:\n",
    "            print(i,j)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5b18736",
   "metadata": {},
   "source": [
    "### Combining data frames that share similar information\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "* Lets assume that we create two data frame series which both contain values of two sensors which measure the temperature in the exact same beaker. However we are interested in seeing the temperatures that the first thermocouple measured but not our second one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "d80f1374",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1    24\n",
      "2    28\n",
      "3    30\n",
      "4    32\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "df1 = pd.Series([22, 24, 28, 30, 32])\n",
    "df2 = pd.Series([26, 22, 27, 25, 29])\n",
    "df1=df1[~df1.isin(df2)]\n",
    "print(df1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c17211d9",
   "metadata": {},
   "source": [
    "* Additionally, we could you get unique points from two separates data sets by finding their union."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "50ad8590",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0     2\n",
      "1     4\n",
      "5    13\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "df1 = pd.Series([2, 4, 5, 8, 10])\n",
    "df2 = pd.Series([5, 10, 13, 10, 8])\n",
    "p_union = pd.Series(np.union1d(df1, df2))          # union of series\n",
    "p_intersect = pd.Series(np.intersect1d(df1, df2))  # intersection of series\n",
    "unique_elements = p_union[~p_union.isin(p_intersect)]\n",
    "print(unique_elements)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5e271c0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}