Lesson_08_Data_pipelines_and_Snakemake.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data pipeline and Snakemake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Today we discuss how to structure a data analysis pipeline, which is the logic of it, how to manage data and automate the analysis with **worflow management tools**\n",
    "\n",
    "in particular, the one we are going to use is called **snakemake**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Coming back to the previous lessons about analysis reliability and reproducibility:\n",
    "\n",
    "> anything you do by hand is to be consider already forgotten.\n",
    "> you have to automate it as soon as possible"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This automation can be done at several levels:\n",
    "\n",
    "* do not manually change the data\n",
    "* do not trow away the results of the analysis\n",
    "* do not check them just with your eyes\n",
    "* manually join the various steps of the analysis\n",
    "\n",
    "In an ideal world, all the analysis and preliminary reports should be done with a single push of the button"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data types\n",
    "\n",
    "data can be usually distinguished in two main types:\n",
    "\n",
    "* text data\n",
    "* binary data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Text data\n",
    "\n",
    "These data are nothing more than text files where the data are formatted in a way that makes them understandable by a computer as well as readable by a human being.\n",
    "\n",
    "These are usually expensive in terms of disk space and read/write time, but are also more robust to data corruption and can be easily inspected and audited by a human being.\n",
    "\n",
    "These formats are often compressed to save disk space and bandwidth, and should formally be considered binary formats, but given that many libraries are able to read them both compressed and uncompressed, I will pretend that there is no difference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### text data - csv (and tsv)\n",
    "\n",
    "Thise formats are used to store tabular data, storing the table line by line, where each line is separated by a comma (**Comma Separated Values**) or tab characters (**Tab Separated Values**).\n",
    "\n",
    "One might find several variations of this format, for example dividing data using `|` or `;`, and wheter text data contained in a cell si surrounded by `\"\"` or not.\n",
    "\n",
    "    name;age;residency\n",
    "    Antonio;32;Bologna\n",
    "    Maria;25;Torino\n",
    "    Francesco;47;Napoli\n",
    "        \n",
    "Many variants consider the `#` character at the beginning of the line to be a comment.\n",
    "\n",
    "They can be easily opened with programs wuch as Microsoft Excel or OpenOffice Calc, but they are limited in storing only tabular formats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Text data - JSON\n",
    "\n",
    "A standardized exchange format on the web, it is very useful to represent hierarchical data.\n",
    "It can roughly be thought as a text equivalent of a dictionary containing numbers, string, lists or other dictionaries.\n",
    "\n",
    "It is white space insensitive.\n",
    "\n",
    "    {\"people\":\n",
    "        {\"antonio\": \n",
    "            {\"age\": \"32\", \"residency\": \"Bologna\"},\n",
    "         \"maria\": \n",
    "            {\"age\": \"20\", \"residency\": \"Cesena\"},\n",
    "         }\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### text data - JSON lines\n",
    "\n",
    "[JSON lines](http://jsonlines.org/) is a recently proposed merge between **csv** and **json**, where each line is a valid json list, but the whole structure represents a tabular dataset.\n",
    "\n",
    "The advantage is that standard json parsers can be used to manage these files, simplifying the interoperability across systems (which is often a problem with the **csv** format, that does not have a formal specification)\n",
    "\n",
    "    [\"Name\", \"Session\", \"Score\", \"Completed\"]\n",
    "    [\"Gilbert\", \"2013\", 24, true]\n",
    "    [\"Alexa\", \"2013\", 29, true]\n",
    "    [\"May\", \"2012B\", 14, false]\n",
    "    [\"Deloise\", \"2012A\", 19, true] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### text data - INI\n",
    "\n",
    "INI were designed under microsoft windows as configuration files, and have become a very common format to include this kind of information (including in python).\n",
    "They a syntax and expressiveness similar to JSON, but several simplifications\n",
    "\n",
    "    [Section]\n",
    "    key_with_value = 2\n",
    "    boolean_toggle_key\n",
    "    \n",
    "        [subsection]\n",
    "        sub_detail = 3\n",
    "    \n",
    "    [New section]\n",
    "    another_parameter = \"hash code\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### text data - YAML\n",
    "\n",
    "YAML is another configuration format. \n",
    "It's quite rich and easy to write and parse for humans, but it's a format so rich and potentially complicated that there is no actually guarantee that any library can successfully import your data if the format is very complicated.\n",
    "\n",
    "It's quite comfortable to use if you don't need very complicated data.\n",
    "\n",
    "    people:\n",
    "        Antonio:\n",
    "            age: 30\n",
    "            residency: Ferrara\n",
    "            hobby: [fishing, football]\n",
    "        Maria:\n",
    "            age: 20\n",
    "            residency: Torino\n",
    "            hobby: [snorkling, skydiving]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### text data - XML\n",
    "\n",
    "This format is an evolution of the original XML, and is well suited to represent very complicated and interconnected data (could replace an entire database), but it is extremely verbose and parsing it can be non-trivial if there is no specification of its structure.\n",
    "\n",
    "    <peoplelist>\n",
    "        <person name='Antonio'>\n",
    "            <birthyear>1983</birthyear>\n",
    "            <townofresidency>Bologna</townofresidency>\n",
    "            <hobbies>\n",
    "                <fishing/>\n",
    "                <football/>\n",
    "            </hobbies>\n",
    "        </person>\n",
    "        <person name='Maria'>\n",
    "            <birthyear>1997</birthyear>\n",
    "            <townofresidency>Rimini</townofresidency>\n",
    "            <hobbies>\n",
    "                <snorkling/>\n",
    "                <skydiving/>\n",
    "            </hobbies>\n",
    "        </person>\n",
    "    </peoplelist>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### notes on XML\n",
    "\n",
    "a lot of \"regular\" formats are actually XML under the hood:\n",
    "\n",
    "* svg vector images\n",
    "* openoffice and office documents (compressed xml)\n",
    "* many html pages\n",
    "* several specialistic formats in various fields"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Binary formats\n",
    "\n",
    "Binary file formats codify the informations directly in a compressed way, exploiting the bit representation of numbers and text and their compression.\n",
    "\n",
    "Several of these formats are often specifics of a single application, but there are several well known formats that you might find in several fields.\n",
    "\n",
    "I'm going to list few of them that are common enough to be worth knowing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Binary formats - lossless images (bmp, png, gif, tiff)\n",
    "\n",
    "These formats contains images stored as individual pixels, each one representing the three fundamental colors (Red, Green and Blue), with different kind of compression, but without losing any kind of information (compared to lossy formats such as jpeg).\n",
    "\n",
    "If your image is a plot and not a generic image, consider storing it as an svg instead, as it allows modification of the elements of the plot after the generation.\n",
    "Or, even better, automatize so that you don't need that final edit!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Binary formats - DICOM\n",
    "\n",
    "Medical format for storing medical images (2D or 3D), for things such as mammographies, CAT scans, PET, MRI, etc...\n",
    "\n",
    "They contain a great amount of metadata about instrument configuration and the patient, and as such should be treated carefully to avoid problems with patient personal data treatment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Binary format - numpy format (NPY)\n",
    "\n",
    "Numpy defines its own binary format, useful to save single arrays in a very easy and simple (and open) format.\n",
    "\n",
    "Given the limitations to singles array and no compression, it's not a great format for long term storage, but they can be useful as a temporary file storage during analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Binary formats - HDF5\n",
    "\n",
    "This is a generic storage format, appropriate for storing multiple, complex numerical data (typically in some form of tabular structure).\n",
    "\n",
    "It is the default version of saving data for the recent versions of Matlab (even if they hide it using the .mat file extension).\n",
    "\n",
    "This file is basically an entire virtual filesystem that can store arbitrarily big data and related metadata.\n",
    "\n",
    "It is very good for long term storage but not great if one needs to repeatedly modify the existing data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Binary formats - databases (sqlite)\n",
    "\n",
    "Databases are the traditional solution for long term data storage, where data needs to be queried and modified often. Most databases requires a dedicated machine or a client-server architecture, but sqlite, a very common and open source database, store everything in a single file.\n",
    "\n",
    "The APIs to interact with it are included in most languages in a way or another. For example python comes with the module **sqlite3** in the standard library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Performances of various formats\n",
    "\n",
    "Performances of various formats to save a medium sized numerical matrix (500 columns x 150'000 rows)\n",
    "\n",
    "\n",
    "| format                | file size  | write time|read time| size after compression|\n",
    "|-----------------------|------------|-----------|---------|-----------------------|\n",
    "| extended csv          | 430 MB     | 1m 15s    | 9.78s   | 132 MB                |\n",
    "| csv with gzip         | 132 MB     | 6m 57s    | 16.2s   |                       |\n",
    "| hdf5 uncompressed     | 463 MB     | 155ms     | 299ms   | 154 MB                |\n",
    "| npy uncompressed      | 462 MB     | 638ms     | 393ms   | 154 MB                |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### File hashing\n",
    "\n",
    "A common and useful operation in data analysis is the so called **file hashing**.\n",
    "\n",
    "This involves generating an alfanumerical string (a mixture of digits and upppercase and lowercase letters) from the content of the file that identify almost perfectly.\n",
    "\n",
    "Good hashing algoritms guarantee that for small modifications of the file bits, there will be huge variations in the resulting hash, allowing to use these string to guarantee data integrity (but not security).\n",
    "\n",
    "There are several ways to calculate the hash (SHA1, MD5, SHA512, etc...) so the algorithm used should be explicitely told to allow to replicate the hash.\n",
    "\n",
    "Virtually all version control systems use hashing to check if data changed or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "md5: 24e0f1fe1d8407d2faff6ed758b18270\n",
      "md5: bcef484a5550fc09f0eb1acbcc3d9089\n",
      "md5: 1d9014f6be7aec56a8d0f4d10404546a\n"
     ]
    }
   ],
   "source": [
    "import hashlib\n",
    "\n",
    "stringa1 = \"1 am an innocent string\".encode('utf8')\n",
    "r = hashlib.md5(stringa1).hexdigest()\n",
    "print('md5: {}'.format(r))\n",
    "\n",
    "stringa1 = \"l am an innocent string\".encode('utf8')\n",
    "r = hashlib.md5(stringa1).hexdigest()\n",
    "print('md5: {}'.format(r))\n",
    "\n",
    "stringa1 = \"I am an innocent string\".encode('utf8')\n",
    "r = hashlib.md5(stringa1).hexdigest()\n",
    "print('md5: {}'.format(r))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2cef471efb258f70189297aa9992e6a4  ./Lesson_01_introduction.ipynb\r\n"
     ]
    }
   ],
   "source": [
    "!md5sum ./Lesson_01_introduction.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2cef471efb258f70189297aa9992e6a4\n"
     ]
    }
   ],
   "source": [
    "def md5(fname):\n",
    "    \"\"\"hash function appropriate for big data\"\"\"\n",
    "    hash_md5 = hashlib.md5()\n",
    "    with open(fname, \"rb\") as f:\n",
    "        for chunk in iter(lambda: f.read(4096), b\"\"):\n",
    "            hash_md5.update(chunk)\n",
    "    return hash_md5.hexdigest()\n",
    "\n",
    "hash_result = md5(\"Lesson_01_introduction.ipynb\")\n",
    "print(hash_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data Pipelines\n",
    "\n",
    "Now that we have an idea on how to recognize data, we can discuss on how this data can be connected together on an data analysis pipeline\n",
    "\n",
    "We can use a simple classification to discuss them.\n",
    "These are not formal, official name, just useful to discuss them and their role, and they reflect the tipical use in a pipeline\n",
    "\n",
    "1. metadata\n",
    "2. raw data\n",
    "3. source code\n",
    "4. source data\n",
    "5. usage data\n",
    "6. intermediate data\n",
    "7. temporary data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 1 - metadata\n",
    "\n",
    "Data related to the data that are going to be analyzed.\n",
    "\n",
    "Typically stored as a simple text file, it usually describe why the data have been collected, the experimental procedure, when, who did it, and so on.\n",
    "\n",
    "It might seem trivial, but after few years one might find themselves with a disk full of TBs of data that they have no idea what to do with because they don't know what data they are.\n",
    "\n",
    "Data of which one have no idea of the reason of existance if functionally as good as completely deleted data.\n",
    "\n",
    "Often it also include the hash values of raw data and source code to make sure of their integrity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 2 - raw data\n",
    "\n",
    "These are the original data that came out of your experiments.\n",
    "Directly from the machine, with no human intervention, **AT ALL**.\n",
    "One will not use these in the analysis, but will process to make them suitable for analsys.\n",
    "They might be in the most weird formats, and if they need some specific program to be read, it would be a good idea to keep track of that as well (if possible, store the program as well).\n",
    "\n",
    "These data **ARE SACRED**\n",
    "\n",
    "They should be preserved and never changed.\n",
    "\n",
    "If one happens to get new versions (for example a measure has been repeated and updated), do not overwrite them, but store them alongside the previous version."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Dal punto di vista di FOSSIL per il controllo di versione, questo tipo di dati può essere tenuto nel repository come **unversioned**, visto che non dovrebbe essere modificato."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 3 - source code\n",
    "\n",
    "It might seems trivial, but the source code to process and analyze the data contains informations about the data that might not be available elsewhere:\n",
    "\n",
    "* how should the data be read and processed\n",
    "* how were the errors corrected?\n",
    "* what are possible things to be aware of when using them?\n",
    "* which analysis have already been done?\n",
    "\n",
    "This are information about the data and should be treated as such, and are often as important as the RAW data.\n",
    "\n",
    "This is another reason to use the version control!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 4 - source data\n",
    "\n",
    "Once one has the RAW data, they can be processed in a useful format using the script generated by the source code.\n",
    "This format is the one that you will load with other program to preprocess them in the format used for the analysis.\n",
    "\n",
    "Using this script one can:\n",
    "* merge files broken down in various way (for example data divided by subjects), adding info about the origin\n",
    "* correct mistakes in the RAW files (don't edit them!)\n",
    "* restructure them in a data structure easy to maintain in the long term (my preference goes for text files such as csv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In this phase you should try to maintain the value of the data as close to the RAW as possible (but not necessarely their structure), so avoid any kind of preprocessing, such as detrending, normalizations and so on.\n",
    "\n",
    "If the RAW are already in a reasonable format, you could consider them also as your source data, but I never seen it happens.\n",
    "Most of the time, when this happens, those data are not really raw data, but preprocessed ones...try to understand what have been done to them!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 5 - usage data\n",
    "\n",
    "This is the actual starting point of our analysis.\n",
    "\n",
    "Starting from the **source data**, we can compose them in a format more comfortable for the analysis that we want, without worrying about database normalization, information duplication and so on.\n",
    "\n",
    "These data are typically going to be generated only once for each analysis, unless one finds problems with the underlying concept of the analysis (so one needs a different format) or in the assumptions of the data (so one might needs to go and reprocess the RAW data into the source data)\n",
    "\n",
    "different analysis might need different **usage data**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 6 - intermediate data\n",
    "\n",
    "Your analysis will probably be composed of several steps, such as data split, normalization, detrending and so on.\n",
    "\n",
    "\n",
    "After each one of these steps, is it good practice to keep track of the results with an intermediate dataset, so that you can recover your analysis at any point without having to run everything that led to that result.\n",
    "\n",
    "Usually the only issue that one might have losing this data is having to re-run the analysis, requiring more time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 7 - temporary data\n",
    "\n",
    "these are the intermediate results of analysis steps, similarly to the **intermediate data**, but with the explicit goal of being deleted after the end of each step.\n",
    "\n",
    "These might be generated for example by a parallel distributed algorithm, that analyze one patient at the time and then merge the results at the end of the run.\n",
    "Once the final table has been generated, there is no reason to keep the other files around consuming space, and thus they need to be removed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Snakemake\n",
    "\n",
    "**Snakemake** is a python reimplementation of the ideas behind the classing GNU make, traditionally used for code compilation.\n",
    "\n",
    "Snakemake allows to automatize complicated data processing pipelines in a very comfortable fashion, then to scale them up to parallel processing and distributed computing (included grids) with few lines of code.\n",
    "\n",
    "It allows to connect simple bash (or powershell) scripts, python and R programs and to mix raw python code in the pipeline code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Snakemake is only one of several system for worklow management.\n",
    "\n",
    "Is the one that in my personal opinion is closer to our needs, but I suggest you to check also the others if you need to.\n",
    "\n",
    "Sone famous other libraries (not all targeted to python) are:\n",
    "\n",
    "* Luigi\n",
    "* Apache Airflow\n",
    "* BigDataScript\n",
    "* Dask\n",
    "* Nextflow\n",
    "\n",
    "A more comprehensive list can be found at: [awesome-pipeline](https://github.com/pditommaso/awesome-pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The fundamental element of snakemake is the **rule**, that represent a program (typically a script), and the program input and output files.\n",
    "\n",
    "Each rule get executed as a separated process, orchestrated by the main snakemake execution.\n",
    "\n",
    "Each rule can have several subsections, of which the most important are:\n",
    "\n",
    "* **output**: the list of files that the list will generate (is a promise, you will actually have to create them inside the rule)\n",
    "* **input**: the list of files required by the rule to be executed to generate the promised output files\n",
    "* **run** or **shell**: execute one or more shell (or python) commands\n",
    "\n",
    "By default, if nothing else is specified, snakemake tries to execute the rule **all**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If the output files of a rule exists already, the rule is skipped (unless you force snakemake to).\n",
    "\n",
    "If the input required to generate those output do not exists, snakmake tries to find another rule that can have those input file as output of its executions.\n",
    "\n",
    "This structure is called **pull** (I specify my arrive point, and try to guess how to get there), compared to the **push** model, in which one specifies explicitely how the data flows from one step to the other.\n",
    "\n",
    "Working with a **pull** approach takes some time to get used to, but can provide several advantages for heavy computations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA\n",
      "/home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/snakemake_lesson\n",
      "/home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/snakemake_lesson\r\n"
     ]
    }
   ],
   "source": [
    "!cd ~/didattica/corso_programmazione_1819/programmingCourseDIFA\n",
    "!mkdir -p snakemake_lesson\n",
    "!cd snakemake_lesson\n",
    "\n",
    "!rm *.txt\n",
    "\n",
    "%cd ~/didattica/corso_programmazione_1819/programmingCourseDIFA\n",
    "%cd snakemake_lesson\n",
    "\n",
    "!pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing Snakefile\n"
     ]
    }
   ],
   "source": [
    "%%file Snakefile\n",
    "\n",
    "rule all:\n",
    "    shell:\n",
    "        \"echo 'hello world' > result.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mBuilding DAG of jobs...\u001b[0m\n",
      "\u001b[33mUsing shell: /bin/bash\u001b[0m\n",
      "\u001b[33mProvided cores: 1\u001b[0m\n",
      "\u001b[33mRules claiming more threads will be scaled down.\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tall\n",
      "\t1\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:35 2019]\u001b[0m\n",
      "\u001b[32mrule all:\n",
      "    jobid: 0\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:35 2019]\u001b[0m\n",
      "\u001b[32mFinished job 0.\u001b[0m\n",
      "\u001b[32m1 of 1 steps (100%) done\u001b[0m\n",
      "\u001b[33mComplete log: /home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/snakemake_lesson/.snakemake/log/2019-03-18T132835.224533.snakemake.log\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!snakemake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "result.txt  Snakefile\r\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If the output of a rule already exists, and is more recent than the input file, the rule will not be executed.\n",
    "\n",
    "This property is called **idempotency**, and makes the execution of the script easier to predict.\n",
    "\n",
    "It is still possible to force snakemake's hand if one needs to (study in the manual how)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting Snakefile\n"
     ]
    }
   ],
   "source": [
    "%%file Snakefile\n",
    "\n",
    "rule all:\n",
    "    output:\n",
    "        \"result.txt\"\n",
    "    shell:\n",
    "        \"echo 'hello world' > {output}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mBuilding DAG of jobs...\u001b[0m\r\n",
      "\u001b[33mNothing to be done.\u001b[0m\r\n",
      "\u001b[33mComplete log: /home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/snakemake_lesson/.snakemake/log/2019-03-18T132839.150574.snakemake.log\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!snakemake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "!rm result.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mBuilding DAG of jobs...\u001b[0m\n",
      "\u001b[33mUsing shell: /bin/bash\u001b[0m\n",
      "\u001b[33mProvided cores: 1\u001b[0m\n",
      "\u001b[33mRules claiming more threads will be scaled down.\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tall\n",
      "\t1\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:40 2019]\u001b[0m\n",
      "\u001b[32mrule all:\n",
      "    output: result.txt\n",
      "    jobid: 0\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:40 2019]\u001b[0m\n",
      "\u001b[32mFinished job 0.\u001b[0m\n",
      "\u001b[32m1 of 1 steps (100%) done\u001b[0m\n",
      "\u001b[33mComplete log: /home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/snakemake_lesson/.snakemake/log/2019-03-18T132840.589923.snakemake.log\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!snakemake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "if the required input files:\n",
    "    \n",
    "* do not exists\n",
    "* there is no rule that can generate them\n",
    "\n",
    "snakemake will crash with an error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting Snakefile\n"
     ]
    }
   ],
   "source": [
    "%%file Snakefile\n",
    "\n",
    "rule all:\n",
    "    input:\n",
    "        \"partial_1.txt\",\n",
    "        \"partial_2.txt\"\n",
    "    output:\n",
    "        \"result.txt\"\n",
    "    shell:\n",
    "        \"cat {input} > {output}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mBuilding DAG of jobs...\u001b[0m\r\n",
      "\u001b[31mMissingInputException in line 2 of /home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/snakemake_lesson/Snakefile:\r\n",
      "Missing input files for rule all:\r\n",
      "partial_2.txt\r\n",
      "partial_1.txt\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!snakemake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's see how would a script look using two rules, one to generate the partial files and one to process them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting Snakefile\n"
     ]
    }
   ],
   "source": [
    "%%file Snakefile\n",
    "\n",
    "rule all:\n",
    "    input:\n",
    "        \"partial_1.txt\",\n",
    "        \"partial_2.txt\"\n",
    "    output:\n",
    "        \"result.txt\"\n",
    "    shell:\n",
    "        \"cat {input} > {output}\"\n",
    "        \n",
    "rule create_partials:\n",
    "    output:\n",
    "        \"partial_1.txt\",\n",
    "        \"partial_2.txt\"\n",
    "    run:\n",
    "        for filename in output:\n",
    "            with open(filename, 'w') as file:\n",
    "                print(\"the result of {}\".format(filename), file=file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "!rm result.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mBuilding DAG of jobs...\u001b[0m\n",
      "\u001b[33mUsing shell: /bin/bash\u001b[0m\n",
      "\u001b[33mProvided cores: 1\u001b[0m\n",
      "\u001b[33mRules claiming more threads will be scaled down.\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tall\n",
      "\t1\tcreate_partials\n",
      "\t2\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:47 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_1.txt, partial_2.txt\n",
      "    jobid: 1\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:47 2019]\u001b[0m\n",
      "\u001b[32mFinished job 1.\u001b[0m\n",
      "\u001b[32m1 of 2 steps (50%) done\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:47 2019]\u001b[0m\n",
      "\u001b[32mrule all:\n",
      "    input: partial_1.txt, partial_2.txt\n",
      "    output: result.txt\n",
      "    jobid: 0\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:47 2019]\u001b[0m\n",
      "\u001b[32mFinished job 0.\u001b[0m\n",
      "\u001b[32m2 of 2 steps (100%) done\u001b[0m\n",
      "\u001b[33mComplete log: /home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/snakemake_lesson/.snakemake/log/2019-03-18T132847.277140.snakemake.log\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!snakemake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "partial_1.txt  partial_2.txt  result.txt  Snakefile\r\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the result of partial_1.txt\r\n"
     ]
    }
   ],
   "source": [
    "!cat partial_1.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the result of partial_2.txt\r\n"
     ]
    }
   ],
   "source": [
    "!cat partial_2.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the result of partial_1.txt\r\n",
      "the result of partial_2.txt\r\n"
     ]
    }
   ],
   "source": [
    "!cat result.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If the intermediate files already exists, the rule will not be executed again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "!rm result.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mBuilding DAG of jobs...\u001b[0m\n",
      "\u001b[33mUsing shell: /bin/bash\u001b[0m\n",
      "\u001b[33mProvided cores: 1\u001b[0m\n",
      "\u001b[33mRules claiming more threads will be scaled down.\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tall\n",
      "\t1\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:52 2019]\u001b[0m\n",
      "\u001b[32mrule all:\n",
      "    input: partial_1.txt, partial_2.txt\n",
      "    output: result.txt\n",
      "    jobid: 0\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:52 2019]\u001b[0m\n",
      "\u001b[32mFinished job 0.\u001b[0m\n",
      "\u001b[32m1 of 1 steps (100%) done\u001b[0m\n",
      "\u001b[33mComplete log: /home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/snakemake_lesson/.snakemake/log/2019-03-18T132852.354044.snakemake.log\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!snakemake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In this script the same python function create both files, one at the time, even if there is no need to wait.\n",
    "\n",
    "They could be generated concurrently, without waiting.\n",
    "To be able to do this I have to specify a rule that generate a **generic** partial file, using **wildcards**, and let snakemake do the rest.\n",
    "\n",
    "Wildcards might cause severe headaches, do not try to get too fancy with them!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm *.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting Snakefile\n"
     ]
    }
   ],
   "source": [
    "%%file Snakefile\n",
    "\n",
    "rule all:\n",
    "    input:\n",
    "        \"partial_1.txt\",\n",
    "        \"partial_2.txt\",\n",
    "    output:\n",
    "        \"result.txt\"\n",
    "    shell:\n",
    "        \"cat {input} > {output}\"\n",
    "        \n",
    "rule create_partials:\n",
    "    output:\n",
    "        out=\"partial_{number}.txt\"\n",
    "    run:\n",
    "        filename = output.out\n",
    "        with open(filename, 'w') as file:\n",
    "            print(\"the result of {}\".format(filename), file=file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mBuilding DAG of jobs...\u001b[0m\n",
      "\u001b[33mUsing shell: /bin/bash\u001b[0m\n",
      "\u001b[33mProvided cores: 1\u001b[0m\n",
      "\u001b[33mRules claiming more threads will be scaled down.\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tall\n",
      "\t2\tcreate_partials\n",
      "\t3\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 15:52:33 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_2.txt\n",
      "    jobid: 2\n",
      "    wildcards: number=2\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 15:52:34 2019]\u001b[0m\n",
      "\u001b[32mFinished job 2.\u001b[0m\n",
      "\u001b[32m1 of 3 steps (33%) done\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 15:52:34 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_1.txt\n",
      "    jobid: 1\n",
      "    wildcards: number=1\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 15:52:34 2019]\u001b[0m\n",
      "\u001b[32mFinished job 1.\u001b[0m\n",
      "\u001b[32m2 of 3 steps (67%) done\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 15:52:34 2019]\u001b[0m\n",
      "\u001b[32mrule all:\n",
      "    input: partial_1.txt, partial_2.txt\n",
      "    output: result.txt\n",
      "    jobid: 0\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 15:52:34 2019]\u001b[0m\n",
      "\u001b[32mFinished job 0.\u001b[0m\n",
      "\u001b[32m3 of 3 steps (100%) done\u001b[0m\n",
      "\u001b[33mComplete log: /home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/snakemake_lesson/.snakemake/log/2019-03-18T155233.983477.snakemake.log\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!snakemake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To be able to use wildcards I need somewhere an **expand** command, that tells snakemake exactly which files to search for.\n",
    "\n",
    "I can have several wildcards at the same time, the important thing is to give an example that allows to initialize all of them.\n",
    "\n",
    "There are methods for automatic inference of wildcards, but I suggest to start using the explicit ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting Snakefile\n"
     ]
    }
   ],
   "source": [
    "%%file Snakefile\n",
    "\n",
    "numbers = [1, 2, 3, 4]\n",
    "\n",
    "rule all:\n",
    "    input:\n",
    "        expand(\"partial_{number}.txt\", number=numbers)\n",
    "    output:\n",
    "        \"result.txt\"\n",
    "    shell:\n",
    "        \"cat {input} > {output}\"\n",
    "        \n",
    "rule create_partials:\n",
    "    output:\n",
    "        out = \"partial_{number}.txt\"\n",
    "    run:\n",
    "        filename = output.out\n",
    "        with open(filename, 'w') as file:\n",
    "            print(\"the result of {}\".format(filename), file=file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "!rm result.txt "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "partial_1.txt  partial_2.txt  Snakefile\r\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mBuilding DAG of jobs...\u001b[0m\n",
      "\u001b[33mUsing shell: /bin/bash\u001b[0m\n",
      "\u001b[33mProvided cores: 1\u001b[0m\n",
      "\u001b[33mRules claiming more threads will be scaled down.\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tall\n",
      "\t2\tcreate_partials\n",
      "\t3\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:58 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_4.txt\n",
      "    jobid: 4\n",
      "    wildcards: number=4\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:59 2019]\u001b[0m\n",
      "\u001b[32mFinished job 4.\u001b[0m\n",
      "\u001b[32m1 of 3 steps (33%) done\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:59 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_3.txt\n",
      "    jobid: 3\n",
      "    wildcards: number=3\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:59 2019]\u001b[0m\n",
      "\u001b[32mFinished job 3.\u001b[0m\n",
      "\u001b[32m2 of 3 steps (67%) done\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:59 2019]\u001b[0m\n",
      "\u001b[32mrule all:\n",
      "    input: partial_1.txt, partial_2.txt, partial_3.txt, partial_4.txt\n",
      "    output: result.txt\n",
      "    jobid: 0\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:28:59 2019]\u001b[0m\n",
      "\u001b[32mFinished job 0.\u001b[0m\n",
      "\u001b[32m3 of 3 steps (100%) done\u001b[0m\n",
      "\u001b[33mComplete log: /home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/snakemake_lesson/.snakemake/log/2019-03-18T132858.929586.snakemake.log\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!snakemake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the result of partial_1.txt\r\n",
      "the result of partial_2.txt\r\n",
      "the result of partial_3.txt\r\n",
      "the result of partial_4.txt\r\n"
     ]
    }
   ],
   "source": [
    "!cat result.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "!rm partial_1.txt\n",
    "!rm result.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Snakemake allows one to generate a flow chart of the execution plan, that allow to visualize everything that has been done or needs to be done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mBuilding DAG of jobs...\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!snakemake --dag | dot -Tsvg > dag.svg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "!cp ./dag.svg ../immagini/snakemake_dag.svg\n",
    "!convert ../immagini/snakemake_dag.svg ../immagini/snakemake_dag.png"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![visualizzazione pipeline](./immagini/snakemake_dag.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Another very useful function if the **provenance** registry, that keeps track of which files have been created by which rule, when and with which parameters\n",
    "\n",
    "This allows to trace the origin of each file in an easy way\n",
    "\n",
    "It can also the set to append the provenance to the complete log, giving the complete history of all files and how they have been changed over time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mBuilding DAG of jobs...\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!snakemake --detailed-summary > provenance.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>rule</th>\n",
       "      <th>version</th>\n",
       "      <th>log-file(s)</th>\n",
       "      <th>input-file(s)</th>\n",
       "      <th>shellcmd</th>\n",
       "      <th>status</th>\n",
       "      <th>plan</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>output_file</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>result.txt</th>\n",
       "      <td>-</td>\n",
       "      <td>all</td>\n",
       "      <td>-</td>\n",
       "      <td>NaN</td>\n",
       "      <td>partial_1.txt,partial_2.txt,partial_3.txt,part...</td>\n",
       "      <td>cat partial_1.txt partial_2.txt partial_3.txt ...</td>\n",
       "      <td>missing</td>\n",
       "      <td>update pending</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>partial_1.txt</th>\n",
       "      <td>-</td>\n",
       "      <td>create_partials</td>\n",
       "      <td>-</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-</td>\n",
       "      <td>missing</td>\n",
       "      <td>update pending</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>partial_2.txt</th>\n",
       "      <td>Mon Mar 18 13:28:47 2019</td>\n",
       "      <td>create_partials</td>\n",
       "      <td>-</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-</td>\n",
       "      <td>rule implementation changed</td>\n",
       "      <td>no update</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>partial_3.txt</th>\n",
       "      <td>Mon Mar 18 13:28:59 2019</td>\n",
       "      <td>create_partials</td>\n",
       "      <td>-</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-</td>\n",
       "      <td>ok</td>\n",
       "      <td>no update</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>partial_4.txt</th>\n",
       "      <td>Mon Mar 18 13:28:59 2019</td>\n",
       "      <td>create_partials</td>\n",
       "      <td>-</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-</td>\n",
       "      <td>ok</td>\n",
       "      <td>no update</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   date             rule version  log-file(s)  \\\n",
       "output_file                                                                     \n",
       "result.txt                            -              all       -          NaN   \n",
       "partial_1.txt                         -  create_partials       -          NaN   \n",
       "partial_2.txt  Mon Mar 18 13:28:47 2019  create_partials       -          NaN   \n",
       "partial_3.txt  Mon Mar 18 13:28:59 2019  create_partials       -          NaN   \n",
       "partial_4.txt  Mon Mar 18 13:28:59 2019  create_partials       -          NaN   \n",
       "\n",
       "                                                   input-file(s)  \\\n",
       "output_file                                                        \n",
       "result.txt     partial_1.txt,partial_2.txt,partial_3.txt,part...   \n",
       "partial_1.txt                                                NaN   \n",
       "partial_2.txt                                                NaN   \n",
       "partial_3.txt                                                NaN   \n",
       "partial_4.txt                                                NaN   \n",
       "\n",
       "                                                        shellcmd  \\\n",
       "output_file                                                        \n",
       "result.txt     cat partial_1.txt partial_2.txt partial_3.txt ...   \n",
       "partial_1.txt                                                  -   \n",
       "partial_2.txt                                                  -   \n",
       "partial_3.txt                                                  -   \n",
       "partial_4.txt                                                  -   \n",
       "\n",
       "                                    status            plan  \n",
       "output_file                                                 \n",
       "result.txt                         missing  update pending  \n",
       "partial_1.txt                      missing  update pending  \n",
       "partial_2.txt  rule implementation changed       no update  \n",
       "partial_3.txt                           ok       no update  \n",
       "partial_4.txt                           ok       no update  "
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.read_csv(\"provenance.tsv\", index_col=0, sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "!rm *.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If I want to execute several rules at the same time (obviously maintaining the execution oerder needed for the files to be generated) I just need to add the  `--cores <N>` and snakemake will execute automatically in parallel everything it can given the available number of processors defined\n",
    "\n",
    "There is an equivalent way to launch the execution on a cluster job queue, making distributed computing really easy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mBuilding DAG of jobs...\u001b[0m\n",
      "\u001b[33mUsing shell: /bin/bash\u001b[0m\n",
      "\u001b[33mProvided cores: 6\u001b[0m\n",
      "\u001b[33mRules claiming more threads will be scaled down.\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tall\n",
      "\t4\tcreate_partials\n",
      "\t5\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:13 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_3.txt\n",
      "    jobid: 3\n",
      "    wildcards: number=3\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:13 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_4.txt\n",
      "    jobid: 4\n",
      "    wildcards: number=4\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:13 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_1.txt\n",
      "    jobid: 1\n",
      "    wildcards: number=1\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:13 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_2.txt\n",
      "    jobid: 2\n",
      "    wildcards: number=2\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:13 2019]\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[32mFinished job 4.\u001b[0m\n",
      "\u001b[32m1 of 5 steps (20%) done\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:13 2019]\u001b[0m\n",
      "\u001b[32mFinished job 2.\u001b[0m\n",
      "\u001b[32m2 of 5 steps (40%) done\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:13 2019]\u001b[0m\n",
      "\u001b[32mFinished job 3.\u001b[0m\n",
      "\u001b[32m3 of 5 steps (60%) done\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:13 2019]\u001b[0m\n",
      "\u001b[32mFinished job 1.\u001b[0m\n",
      "\u001b[32m4 of 5 steps (80%) done\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:13 2019]\u001b[0m\n",
      "\u001b[32mrule all:\n",
      "    input: partial_1.txt, partial_2.txt, partial_3.txt, partial_4.txt\n",
      "    output: result.txt\n",
      "    jobid: 0\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:13 2019]\u001b[0m\n",
      "\u001b[32mFinished job 0.\u001b[0m\n",
      "\u001b[32m5 of 5 steps (100%) done\u001b[0m\n",
      "\u001b[33mComplete log: /home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/snakemake_lesson/.snakemake/log/2019-03-18T132913.290945.snakemake.log\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!snakemake --cores 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "!rm *.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "I can also specify limited resources (alongside the cpu cores) so that the pipeline does not exceed these limites.\n",
    "\n",
    "For example, if one has rules that require massive amount of memory, it is better to avoid launching them all the the same time.\n",
    "I can specify an expected mount of needed memory and a total available one, and snakemake will make sure to not overschedule the execution to never surpass those limits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting Snakefile\n"
     ]
    }
   ],
   "source": [
    "%%file Snakefile\n",
    "\n",
    "numbers = [1, 2, 3, 4]\n",
    "\n",
    "rule all:\n",
    "    input:\n",
    "        expand(\"partial_{number}.txt\", number=numbers)\n",
    "    output:\n",
    "        \"result.txt\"\n",
    "    shell:\n",
    "        \"cat {input} > {output}\"\n",
    "        \n",
    "rule create_partials:\n",
    "    output:\n",
    "        out = \"partial_{number}.txt\"\n",
    "    resources: \n",
    "        memory = 6\n",
    "    run:\n",
    "        filename = output.out\n",
    "        with open(filename, 'w') as file:\n",
    "            print(\"the result of {}\".format(filename), file=file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mBuilding DAG of jobs...\u001b[0m\n",
      "\u001b[33mUsing shell: /bin/bash\u001b[0m\n",
      "\u001b[33mProvided cores: 6\u001b[0m\n",
      "\u001b[33mRules claiming more threads will be scaled down.\u001b[0m\n",
      "\u001b[33mProvided resources: memory=12\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tall\n",
      "\t4\tcreate_partials\n",
      "\t5\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:18 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_2.txt\n",
      "    jobid: 2\n",
      "    wildcards: number=2\n",
      "    resources: memory=6\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:18 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_3.txt\n",
      "    jobid: 3\n",
      "    wildcards: number=3\n",
      "    resources: memory=6\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:18 2019]\u001b[0m\n",
      "\u001b[32mFinished job 2.\u001b[0m\n",
      "\u001b[32m1 of 5 steps (20%) done\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:18 2019]\u001b[0m\n",
      "\u001b[32mFinished job 3.\u001b[0m\n",
      "\u001b[32m2 of 5 steps (40%) done\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:18 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_1.txt\n",
      "    jobid: 1\n",
      "    wildcards: number=1\n",
      "    resources: memory=6\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:18 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_4.txt\n",
      "    jobid: 4\n",
      "    wildcards: number=4\n",
      "    resources: memory=6\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:19 2019]\u001b[0m\n",
      "\u001b[32mFinished job 1.\u001b[0m\n",
      "\u001b[32m3 of 5 steps (60%) done\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:19 2019]\u001b[0m\n",
      "\u001b[32mFinished job 4.\u001b[0m\n",
      "\u001b[32m4 of 5 steps (80%) done\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:19 2019]\u001b[0m\n",
      "\u001b[32mrule all:\n",
      "    input: partial_1.txt, partial_2.txt, partial_3.txt, partial_4.txt\n",
      "    output: result.txt\n",
      "    jobid: 0\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:19 2019]\u001b[0m\n",
      "\u001b[32mFinished job 0.\u001b[0m\n",
      "\u001b[32m5 of 5 steps (100%) done\u001b[0m\n",
      "\u001b[33mComplete log: /home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/snakemake_lesson/.snakemake/log/2019-03-18T132918.612842.snakemake.log\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!snakemake --cores 6 --resources memory=12"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Configurations\n",
    "\n",
    "If there is the need to pass some configuration parameters, these can be given from the command line or loaded as a configuration file in YAML or JSON format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting Snakefile\n"
     ]
    }
   ],
   "source": [
    "%%file Snakefile\n",
    "\n",
    "numbers = [i for i in range(int(config['number']))]\n",
    "\n",
    "rule all:\n",
    "    input:\n",
    "        expand(\"partial_{number}.txt\", number=numbers)\n",
    "    output:\n",
    "        \"result.txt\"\n",
    "    shell:\n",
    "        \"cat {input} > {output}\"\n",
    "        \n",
    "rule create_partials:\n",
    "    output:\n",
    "        out = \"partial_{number}.txt\"\n",
    "    resources: \n",
    "        memory = 6\n",
    "    run:\n",
    "        filename = output.out\n",
    "        with open(filename, 'w') as file:\n",
    "            print(\"the result of {}\".format(filename), file=file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "!rm *.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mBuilding DAG of jobs...\u001b[0m\n",
      "\u001b[33mUsing shell: /bin/bash\u001b[0m\n",
      "\u001b[33mProvided cores: 6\u001b[0m\n",
      "\u001b[33mRules claiming more threads will be scaled down.\u001b[0m\n",
      "\u001b[33mProvided resources: memory=12\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tall\n",
      "\t4\tcreate_partials\n",
      "\t5\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:23 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_3.txt\n",
      "    jobid: 4\n",
      "    wildcards: number=3\n",
      "    resources: memory=6\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:23 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_2.txt\n",
      "    jobid: 3\n",
      "    wildcards: number=2\n",
      "    resources: memory=6\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:23 2019]\u001b[0m\n",
      "\u001b[32mFinished job 4.\u001b[0m\n",
      "\u001b[32m1 of 5 steps (20%) done\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:23 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_0.txt\n",
      "    jobid: 1\n",
      "    wildcards: number=0\n",
      "    resources: memory=6\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:23 2019]\u001b[0m\n",
      "\u001b[32mFinished job 3.\u001b[0m\n",
      "\u001b[32m2 of 5 steps (40%) done\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:23 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_1.txt\n",
      "    jobid: 2\n",
      "    wildcards: number=1\n",
      "    resources: memory=6\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:23 2019]\u001b[0m\n",
      "\u001b[32mFinished job 1.\u001b[0m\n",
      "\u001b[32m3 of 5 steps (60%) done\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:23 2019]\u001b[0m\n",
      "\u001b[32mFinished job 2.\u001b[0m\n",
      "\u001b[32m4 of 5 steps (80%) done\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:23 2019]\u001b[0m\n",
      "\u001b[32mrule all:\n",
      "    input: partial_0.txt, partial_1.txt, partial_2.txt, partial_3.txt\n",
      "    output: result.txt\n",
      "    jobid: 0\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:23 2019]\u001b[0m\n",
      "\u001b[32mFinished job 0.\u001b[0m\n",
      "\u001b[32m5 of 5 steps (100%) done\u001b[0m\n",
      "\u001b[33mComplete log: /home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/snakemake_lesson/.snakemake/log/2019-03-18T132923.287973.snakemake.log\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!snakemake --cores 6 --resources memory=12 --config number=4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing config.yaml\n"
     ]
    }
   ],
   "source": [
    "%%file config.yaml\n",
    "number: 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting Snakefile\n"
     ]
    }
   ],
   "source": [
    "%%file Snakefile\n",
    "\n",
    "configfile: \"./config.yaml\"\n",
    "\n",
    "numbers = [i for i in range(int(config['number']))]\n",
    "\n",
    "rule all:\n",
    "    input:\n",
    "        expand(\"partial_{number}.txt\", number=numbers)\n",
    "    output:\n",
    "        \"result.txt\"\n",
    "    shell:\n",
    "        \"cat {input} > {output}\"\n",
    "        \n",
    "rule create_partials:\n",
    "    output:\n",
    "        out = \"partial_{number}.txt\"\n",
    "    resources: \n",
    "        memory = 6\n",
    "    run:\n",
    "        filename = output.out\n",
    "        with open(filename, 'w') as file:\n",
    "            print(\"the result of {}\".format(filename), file=file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "!rm *.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mBuilding DAG of jobs...\u001b[0m\n",
      "\u001b[33mUsing shell: /bin/bash\u001b[0m\n",
      "\u001b[33mProvided cores: 6\u001b[0m\n",
      "\u001b[33mRules claiming more threads will be scaled down.\u001b[0m\n",
      "\u001b[33mProvided resources: memory=12\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tall\n",
      "\t4\tcreate_partials\n",
      "\t5\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:28 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_2.txt\n",
      "    jobid: 3\n",
      "    wildcards: number=2\n",
      "    resources: memory=6\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:28 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_0.txt\n",
      "    jobid: 1\n",
      "    wildcards: number=0\n",
      "    resources: memory=6\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:28 2019]\u001b[0m\n",
      "\u001b[32mFinished job 3.\u001b[0m\n",
      "\u001b[32m1 of 5 steps (20%) done\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:28 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_3.txt\n",
      "    jobid: 4\n",
      "    wildcards: number=3\n",
      "    resources: memory=6\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:28 2019]\u001b[0m\n",
      "\u001b[32mFinished job 1.\u001b[0m\n",
      "\u001b[32m2 of 5 steps (40%) done\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:28 2019]\u001b[0m\n",
      "\u001b[32mrule create_partials:\n",
      "    output: partial_1.txt\n",
      "    jobid: 2\n",
      "    wildcards: number=1\n",
      "    resources: memory=6\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[33mJob counts:\n",
      "\tcount\tjobs\n",
      "\t1\tcreate_partials\n",
      "\t1\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:29 2019]\u001b[0m\n",
      "\u001b[32mFinished job 2.\u001b[0m\n",
      "\u001b[32m3 of 5 steps (60%) done\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:29 2019]\u001b[0m\n",
      "\u001b[32mFinished job 4.\u001b[0m\n",
      "\u001b[32m4 of 5 steps (80%) done\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:29 2019]\u001b[0m\n",
      "\u001b[32mrule all:\n",
      "    input: partial_0.txt, partial_1.txt, partial_2.txt, partial_3.txt\n",
      "    output: result.txt\n",
      "    jobid: 0\u001b[0m\n",
      "\u001b[32m\u001b[0m\n",
      "\u001b[32m[Mon Mar 18 13:29:29 2019]\u001b[0m\n",
      "\u001b[32mFinished job 0.\u001b[0m\n",
      "\u001b[32m5 of 5 steps (100%) done\u001b[0m\n",
      "\u001b[33mComplete log: /home/enrico/didattica/corso_programmazione_1819/programmingCourseDIFA/snakemake_lesson/.snakemake/log/2019-03-18T132928.691830.snakemake.log\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!snakemake --cores 6 --resources memory=12"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Exercise\n",
    "\n",
    "In the website (github) you can find some files that contains imaginary financial transactions of several people, stored as a `.tsv` files, where the fiirst column contains the name of the person (unique to that person) and the second column store the amount that that person earned for that day\n",
    "\n",
    "There will also be a file storing the md5sum for each one of these files\n",
    "\n",
    "the exercise is in two parts:\n",
    "\n",
    "1. Write a pipeline that download each of these files, check that the md5 has is the expected one\n",
    "2. load the data and merge them, summing the transaztion for each person in a single value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "notes:\n",
    "\n",
    "* the data can the found at the folder\n",
    "* `https://raw.githubusercontent.com/UniboDIFABiophysics/programmingCourseDIFA/master/snakemake_exercise/`\n",
    "* with filenames transazioni_{}.tsv with an index from 00 to 49\n",
    "* the file with the hash of each file is in `md5sums.tsv`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### suggestions\n",
    "\n",
    "* the has function can be implemented in python or done using the bash `md5sum` program\n",
    "* the download can be done using the `requests` library or the `wget` program from terminal\n",
    "* use the snakemake wildcards, or you will have to go crazy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "url_base = (\"https://raw.githubusercontent.com/UniboDIFABiophysics\"+\n",
    "            \"/programmingCourseDIFA/master/snakemake_exercise/\")\n",
    "filename = \"transazioni_00.tsv\"\n",
    "\n",
    "response = requests.get(url_base+filename)\n",
    "\n",
    "# Throw an error for bad status codes\n",
    "response.raise_for_status()\n",
    "\n",
    "with open(filename, 'wb') as handle:\n",
    "    handle.write(response.content)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}