Pyjournalists_zika.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center>\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/pyladies-bcn/pyladies_latex_template/master/pyladies.png\" WIDTH=600> \n",
    "\n",
    "<h1>\n",
    "WORKSHOP<br>\n",
    "\"Python for Journalists and Data Nerds\"<br>\n",
    "</h1>\n",
    "\n",
    "<dl>\n",
    "<dt><br></dt>\n",
    "<dt>Marta Alonso @malonfe</dt>\n",
    "<dt>Kathleen Burnett @kmb232</dt>\n",
    "</dl> \n",
    "Based in an idea from <a href=\"https://twitter.com/crisodisy\">@crisodisy</a> from <a href=\"http://www.meetup.com/PyLadies-BCN/\">PyLadiesBCN</a>\n",
    "\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1. INTRO TO JUPYTER NOTEBOODK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 1.1 Jupyter Kernel\n",
    "Be sure you are using the appropriate kernel.\n",
    "\n",
    "Go to the menu \"Kernel > Change Kernel\" and select the one that refers to your virtual environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 1.2 Text cells\n",
    "**Double click on this cell**, you will see the text without formatting. \n",
    "\n",
    "This allows you to edit this block of text which is written using [Markdown](http://daringfireball.net/projects/markdown/syntax). But you can also use <strong>html</strong> to edit it.\n",
    "\n",
    "Hit *shift* + *enter* or *shift* + *return* on your keyboard to show the formatted text again. This is called **running** the cell, and you can also do it using the run button in the toolbar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 1.3 Code cells\n",
    "One great advantage of Jupyter notebooks is that you can show your Python code alongside the results, add comments to the code, or even add blocks of text using Markdown. The following cell is a code cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Hit shift + enter or use the run button to run this cell and see the results\n",
    "\n",
    "print 'Hello PyLadies'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The last line of every code cell will be displayed by default, \n",
    "# even if you don't print it. Run this cell to see how this works.\n",
    "\n",
    "2 + 2 # The result of this line will not be displayed\n",
    "3 + 3 # The result of this line will be displayed, because it is the last line of the cell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 1.4 Creating cells \n",
    " \n",
    "To create a new **code cell**, click \"Insert > Insert Cell [Above or Below]\". A code cell will automatically be created.\n",
    "\n",
    "To create a new **markdown cell**, first follow the process above to create a code cell, then change the type from \"Code\" to \"Markdown\" using the dropdown next to the run, stop, and restart buttons.\n",
    "\n",
    "You can also use the **+** button to create a cell and the **scissors** buttom to cut it out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 1.5 Re-running cells\n",
    "\n",
    "If you find a bug in your code, you can always update the cell and re-run it. However, any cells that come afterward won't be automatically updated. Try it out below. First run each of the three cells. The first two don't have any output, but you will be able to tell they've run because a number will appear next to them, for example, \"In [5]\". The third cell should output the message \"Intro to Data Analysis is awesome!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "who = \"Python\"\n",
    "who"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "message = \"We love \" + who + \"!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "message"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you've run all three cells, try modifying the first one to set **`who`** to your anything else.  \n",
    "\n",
    "Rerun the first and third cells without rerunning the second and see what happens. It seems that you need to run the three cells to have the expected result.\n",
    "\n",
    "Often, after changing a cell, you'll want to rerun all the cells below it.  You can do that quickly by clicking **\"Cell > Run All Below\"**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.6 Autocompletion\n",
    "Use TAB to automplete python methods and variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print len(message)\n",
    "print len(message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.7 Final tips"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the output is too long click on the left part of the result: \n",
    "* One click will confine the result to a box\n",
    "* Double click will hide the result\n",
    "\n",
    "**One final thing to remember**: if you shut down the kernel after saving your notebook, the cells' output will still show up as you left it at the end of your session when you start the notebook back up. However, the state of the kernel will be reset. If you are actively working on a notebook, remember to re-run your cells to set up your working environment to really pick up where you last left off."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 2. WORKSHOP\n",
    "Handy documentation:\n",
    "\n",
    "[Pandas Docs](http://pandas.pydata.org/pandas-docs/version/0.18.1/)\n",
    "\n",
    "[Pandas API Reference](http://pandas.pydata.org/pandas-docs/version/0.18.1/api.html)\n",
    "\n",
    "[Pandas DataFrames](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# THE \"ZIKA REPORTS\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Obtaining and Preparing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this exercise we are going to use a real data set of Zika infection cases. The original data is provided by the  Centers for Desease Control and Prevention (CDC) in raw HTML. You can find it in the [CDC site](http://www.cdc.gov/zika/geo/united-states.html) from where it is scrapped and formatted as CSVs files. \n",
    "\n",
    "The CSVs are available from:\n",
    "* Epidemic Prediction Innitiative github: https://github.com/cdcepi/zika\n",
    "* Our github: https://github.com/PyLadiesDC/python-for-journalists\n",
    "\n",
    "You've downloaded already all the data so you should have all that you need to start right now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Check that we have all necessary files in current directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, we have to ensure that all files we downloded are in the working directory and all data files are available.\n",
    "\n",
    "In this exercice we are going to work with data stored in the **data** folder. Check that you have that folder running the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "print os.getcwd()\n",
    "print os.listdir('.')\n",
    "print os.listdir('data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Load .csv in a DataFrame with the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to import pandas library. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "#from now on we use pd as an abbreviation for pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a Pandas DataFrame from a csv file using the **read_csv** method. The name of the columns are going to be those of the CSVs columns. \n",
    "\n",
    "Check Pandas [Input/Output](http://pandas.pydata.org/pandas-docs/stable/io.html) commands to check other available read methods **read_json** or **read_excel**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"shape is an attribute (invoked without parenthesis)\n",
    "tells us how many rows and columns our DataFrame has\"\"\"\n",
    "august_df = pd.read_csv(\"data/CDC_Report-2016-08-31.csv\")\n",
    "august_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To explore the DataFrame we can use **head()** method that results by default in the first 5 lines of the DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"head() is a method, to call it you use parenthesis\"\"\"\n",
    "august_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The attribute **columns** returns a list of the column names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "august_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Concatenate several DataFrames\n",
    "If we want to have the historical data to see the evolution of the disease we need to create a new DataFrame containing the content of the different csv files. We use **pd.concat()** to concatenate one DataFrame per csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import glob \n",
    "\n",
    "#glob finds all the pathnames matching a specified pattern\n",
    "csv_list = glob.glob(\"data/*.csv\")\n",
    "\n",
    "df_list = []\n",
    "for f in csv_list:\n",
    "    df = pd.read_csv(f)\n",
    "    df_list.append(df)\n",
    "year_df = pd.concat(df_list, ignore_index=True)\n",
    "# NOTE: a more pythonic way of doing the last five lines would be:\n",
    "# year_df = pd.concat((pd.read_csv(f) for f in all_files), ignore_index=True)\n",
    "\n",
    "year_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "year_df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 Arrange columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.5.1 Processing dates\n",
    "We want to process the **report_date** column, being able to extract the year or the month or any other part of the date:\n",
    "* we need to import **datetime** library\n",
    "* using **strptime** to create a datetime object according to a format (see the [link](https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior) for understand the format directives)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import datetime as dt\n",
    "\n",
    "#creates a datetime object from a string representing a date and time and a corresponding format string.\n",
    "my_date = dt.datetime.strptime('2010-06-17', '%Y-%m-%d')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# from the datetime object we can extract the following attributes and functions\n",
    "print my_date.year\n",
    "print my_date.month\n",
    "print my_date.day\n",
    "print my_date.hour\n",
    "print my_date.minute\n",
    "print my_date.isocalendar() #returns a tuple (ISO year, ISO week number, ISO weekday)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.5.2 New column -- 'week'\n",
    "Now we are going to create a column **week** from the column **report_date** using the **apply** method:\n",
    "* We first define a function that extracts the week number for each of the dates.\n",
    "* Then we need to **apply** this function to all the elements in the **report_date** column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def get_week_number(any_date):\n",
    "     return dt.datetime.strptime(any_date,'%Y-%m-%d').isocalendar()[1]\n",
    "    \n",
    "# we apply the function to each of the elements of the column \"report_date\"\n",
    "year_df['week'] = year_df['report_date'].apply(get_week_number)\n",
    "\n",
    "year_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.5.3 New column -- 'state'\n",
    "Now we are going to create a column **state** from the column **location**:\n",
    "* We first define a function that extracts the week number for each of the dates.\n",
    "* Then we need to **apply** this function to all the elements in the **report_date** column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def get_state(location):\n",
    "    return location.split(\"-\")[1]\n",
    "\n",
    "year_df['state'] = year_df['location'].apply(get_state)\n",
    "year_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.5.4 Deleting columns\n",
    "We also want to delete the columns that we don't need. We use the **drop** method with:\n",
    "* **axis = 1** specifying that we are deleting a column (0 for deleting rows) \n",
    "* **inplace = True** specifies that we are deleting the column in our object, with inplace = False we are creating a new DataFrame without that column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "year_df.drop('time_period', axis=1, inplace=True)\n",
    "year_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE**: If you try to execute the drop twice you'll get an error because that column doesn't exist anymore. If you want to reset the kernel and start from scratch, you can go to the meny \"Kernel\" and select any of the \"Restart\" options and afterwards going to the menu \"Cell\" and Run the cells above or below the focused cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.5.5 Exercise - try it on your own!\n",
    "* Try to do this exercise without having a look to the solution in the next paragraph\n",
    "* Create a column **country** similarly as the one you created before\n",
    "* Delete the column **time_period_type**, **unit**, **data_field_code** and **location** that you are not using\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# your solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.5.6 Solution to the previous exercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Adding \"country\" column\n",
    "def get_country(location):\n",
    "    return location.split(\"-\")[0]    \n",
    "year_df['country'] = year_df['location'].apply(get_country)\n",
    "\n",
    "# Deleting extra columns\n",
    "year_df.drop('time_period_type', axis=1, inplace=True) \n",
    "year_df.drop('unit', axis=1, inplace=True) \n",
    "year_df.drop('data_field_code', axis=1, inplace=True) \n",
    "year_df.drop('location', axis=1, inplace=True) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "year_df.head() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.6 Select and filter data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.6.1 Selectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "year_df.ix[:3,'report_date':'data_field']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Creating a new DataFrame from a fragment of other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is a good practice to use the **copy()** method to avoid later problems when creating a new DataFrame out of a piece of another one. Pandas cannot assure the piece you are taking is a view or a copy of the original DataFrame and we want to be sure its a copy, so we do not modify the original one if we change the new one. Explanation: [here](http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "two_columns_df = year_df[[\"week\",\"value\"]].copy()\n",
    "two_columns_df.head()\n",
    "\n",
    "# Another way of copying columns selecting all the rows for certain columns, no need to add the copy() method\n",
    "# two_columns_df = year_df.loc[:,[\"week\",\"value\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.6.2 Applying conditions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which are the zika infection cases in the District of Columbia?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "year_df[(year_df[\"state\"] == \"District_of_Columbia\") & (year_df[\"week\"] == 30)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But we should store the value of the latest week, to know which are the latest results if we need them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "max_week = year_df[\"week\"].max()\n",
    "print max_week"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which are the **current** zika infection cases in the District of Columbia?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "year_df[(year_df[\"state\"] == \"District_of_Columbia\") & (year_df[\"week\"] == max_week)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which states or territories have zika cases due to local infection?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "year_df[(year_df[\"value\"] != 0) & (year_df[\"week\"] == max_week) & (year_df[\"data_field\"] == \"zika_reported_local\") ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.6.3 Exercise - try it on your own!\n",
    "* Try to do this exercise without having a look to the solution in the next paragraph\n",
    "* Use **year_df** dataframe and conditions to create a new df called **syear_df** where the **state** column contains only state names, not territory names \n",
    "* To ensure that we are making a copy use the **copy()** method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.6.4 Solution to exercise 2.6.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "syear_df = year_df[year_df['location_type'] == 'state'].copy()\n",
    "syear_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.7 Grouping and ordering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.7.1 Which states have more zika cases?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* First we selected the chunk that we want to use, the latest report data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "latest_df = syear_df[syear_df['week'] == max_week].copy()\n",
    "latest_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Now we group the data by state to sum both local and travel cases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sum_df = latest_df.groupby('state').sum()\n",
    "sum_df.head()\n",
    "# see how all the numerical columns ar added up, although the **week** column added doesn't make any sense\n",
    "# pay attention to how the resulted DF has states as indexes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We order the df by the value columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sorted_df = sum_df.sort_values(by='value', ascending=False)\n",
    "sorted_df[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.8 Plotting Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**matplotlib** is one of the most common libraries for making 2D plots of arrays in Python, Pandas library has a **plot()** method that wraps the plotting functionalities of *matplotlib*. We'll try to use this method when possible, for simplicity. \n",
    "\n",
    "Even so, in order to see the plots inmediately in our script we need to use the **matplotlib** *magic line* as we'll see in the next paragraph. More about magic lines [here](https://ipython.org/ipython-doc/3/interactive/magics.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.8.1 Bar graph with the aggregated cases of zika in the US"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"This is the matplotlib magic line, by default matplotlib defers drawing until the end \n",
    "of the script. But we need matplotlib to works interactively and draw the plots right away\"\"\"\n",
    "%matplotlib inline\n",
    "\n",
    "# Seaborn is a library that makes your plots prettier\n",
    "import seaborn as sns    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# we are drawing a plot bar with values in axis Y\n",
    "# by default the indexes of the DF (states) are used for axis X\n",
    "sorted_df[:10].plot.bar(y='value', figsize=(8,4), title='zika cases')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Let's draw it horizontally:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Remember how we sorted:\n",
    "#sorted_df = sum_df.sort_values(by='value', ascending=False)\n",
    "sorted_df[:10].plot.barh(y='value', figsize=(8,4), title='zika cases')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Wouldn't it be clearer if the highest value is at the top of the plot?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.8.2 Exercise - try it on your own!\n",
    "* Try to do this exercise without having a look to the solution in the next paragraph\n",
    "* Draw the same horizontal bar graph that we've done before but ordered from the top to the bottom\n",
    "* You need to re-write the code we've done for the graph, read the instructions in the following cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This is the code we used:\n",
    "# sorted_df = sum_df.sort_values(by='value', ascending=False)\n",
    "# sorted_df[:10].plot.barh(y='value', figsize=(8,4), title='zika cases')\n",
    "\n",
    "# 1. sort the dataframe in an ascending way\n",
    "# 2. get the last 10 positions of the dataframe instead of the first ten\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.8.3 Solution to 2.8.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sorted_df = sum_df.sort_values(by='value', ascending=True)\n",
    "sorted_df[-10:].plot.barh(y='value', figsize=(8,4), title='zika cases',color='sandybrown')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Hint**: color names for plotting:\n",
    "http://matplotlib.org/mpl_examples/color/named_colors.hires.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.8.4 Linegraph with the evolution of cases throught 2016 in the US"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Remember that syear_df is our dataframe with the anual data for the states\n",
    "weekly_df = syear_df.groupby('week').sum()\n",
    "weekly_df.head()\n",
    "# now the weeks are the new indexes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# again the X axis takes by default the indexes of the DF (the weeks)\n",
    "\n",
    "weekly_df.plot.line(y='value',figsize=(8,4), title='zika evolution in the US', color='orange')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.8.5 Add some labels!! (extra)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\"\"\"Our plotting function returns the axes object, which allow us to access\n",
    "the rectangles and add a text to them\"\"\"\n",
    "\n",
    "# first reorder the data descendingly\n",
    "sorted_df = sum_df.sort_values(by='value', ascending=False)\n",
    "# access the axes object when creating the bar plot\n",
    "axes = sorted_df[:10].plot.bar(y='value', figsize=(8,4), title='zika cases')\n",
    "# loop over the rectangles and add a tex\n",
    "for p in axes.patches:\n",
    "    axes.text(p.get_x() + p.get_width()/2, # x positions\n",
    "            p.get_height()-35,             # y position\n",
    "            int(p.get_height()),           # label text\n",
    "            ha='center', va='bottom', color='white', fontsize=9) # fontdict with font alignment and properties\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.8.6 Stacked bar plots (extra)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to show in the same bar plot which cases of zika were transmitted locally and which cases where due to travelling to affected areas.\n",
    "\n",
    "In order to plot two different variables **travel** and **local** we need to create two new columns for these variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# We'll use latest_df dataframe which contains the most recent values, already filtered only for states.\n",
    "\n",
    "latest_df['travel'] = latest_df.loc[latest_df['data_field'] == 'zika_reported_travel','value']\n",
    "latest_df['local'] = latest_df.loc[latest_df['data_field'] == 'zika_reported_local','value']\n",
    "latest_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "group_df = latest_df.groupby('state').sum()\n",
    "group_df.sort_values(by=\"value\", ascending = False, inplace = True)\n",
    "group_df[:10].plot.bar(y=['travel','local'], stacked = True, figsize=(8,4), title='zika cases')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.8.7 Arranging the axes for dates (extra)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example is a little bit more complex, in this case we have to go down to the original matplotlib library and use its **plot_data** method that allows as to configure their axes according to our necessities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.dates as dates\n",
    "\n",
    "# group by date returns a df with report_date as indexes\n",
    "bydate_df = syear_df.groupby('report_date').sum()\n",
    "\n",
    "# Series with the values to plot\n",
    "y_values = bydate_df['value']\n",
    "# Get the values of the indexes (our dates) and returns a list of Datetime objects\n",
    "my_date_index = pd.DatetimeIndex(bydate_df.index.values).to_pydatetime()\n",
    "\n",
    "f = plt.figure(figsize=(8, 4))\n",
    "ax = f.gca() # get current axes\n",
    "# set ticks location in the X axis, in the 14th day of each month (to see Sept)\n",
    "ax.xaxis.set_major_locator(dates.MonthLocator(bymonthday=14))\n",
    "# set ticks format in the X axis => (%b is abbreviated month)+new line+ year\n",
    "ax.xaxis.set_major_formatter(dates.DateFormatter('%b\\n%Y'))\n",
    "plt.plot_date(\n",
    "            my_date_index, # x values\n",
    "            y_values,      # y values\n",
    "            fmt='-',     # format of the line\n",
    "            xdate=True,  # x-axis will be labeled with dates\n",
    "            ydate=False, # y-axis don't\n",
    "            color='red')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.9 Store data in a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, consult Pandas [Input/Output commands](http://pandas.pydata.org/pandas-docs/stable/io.html) to check other file formats available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "group_df.to_csv(\"output.csv\")"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:myvenv]",
   "language": "python",
   "name": "conda-env-myvenv-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}