Reproducibility.Rmd

---
title: "Reproducible Research Using RMarkdown and Git through RStudio"
subtitle: "Tutorial for NGS Workshop Week 3"
author: "Marian L. Schmidt, @micro_marian, marschmi@umich.edu"
date: "August 25, 2015"
output:
  html_document:
    theme: united
    toc: yes
---

## Information that is Relevant to this Tutorial  

This tutorial was constructed as a part of Titus Brown's Next Generation Sequencing Data Analysis Workshop **Week 3** that took place at Michigan State University's Kellogg Biological Station between August 24-28, 2015. 

- Titus Brown's <a href="https://github.com/ngs-docs/angus" target="_blank">NGS Course</a>  
    + <a href="http://angus.readthedocs.org/en/2015/week3.html" target="_blank">Schedule of the NGS Course Week 3</a>  
    + A <a href="http://ivory.idyll.org/blog/tag/ngs-course.html" target="_blank">blog post from Titus</a> synthesizing the third week of the course.  
- Github for <a href="https://github.com/marschmi/NGS2015_RMarkdown_Reproducibility" target="_blank">this website</a>  

## The Agenda and Learning Goals  
#### **Agenda**  

1. What is **robust and reproducible research**?  
2. How can you make **your research** more **robust and reproducible**?  
3. You can use RMarkdown!  Wait, what's RMarkdown?
4. How to use RMarkdown and why it may change your workflow.
5. Using **git** through RStudio.   
6. Initialize your a more reproducible github repo within RStudio!

#### **Learning Goals**  
1. Walk away knowing what **robust and reproducible research** are.  
2. To know the changes you can make to your research to make it more **robust and reproducible.**  
3. To understand what RMarkdown *is.*
4. To know how to create an RMarkdown document and understand it's uses for code sharing. 
5. To understand how to use **git** through the RStudio interface.
6. Know how to initialize a more reproducible github repo within RStudio!

***

## Key Resources Used to Build this lesson
- Vince Buffalo's <a href="http://www.amazon.com/Bioinformatics-Data-Skills-Reproducible-Research/dp/1449367372" target="_blank">Bioinformatics Data Skills</a> book and it's helpful <a href="https://github.com/vsbuffalo/bds-files" target="_blank">github page.</a>
    + Main source for this presentation.
- A course by Karl Broman at the University of Wisconsin-Madison on <a href="http://kbroman.org/Tools4RR/" target="_blank">Reproducible Research.</a>  
- <a href="http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001745" target="_blank">Best Practices for Scientific Computing</a> by Wilson et al., 2014.


***

## We scientists have a few problems  
- The <a href="http://fivethirtyeight.com/datalab/as-a-major-retraction-shows-were-all-vulnerable-to-faked-data/" target="_blank">LaCour Scandal</a> of fabricated data published in *Science*.  
- Scientists in the United States spend <a href="http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002165" target="_blank">$28 billion</a> each year on basic biomedical research that cannot be repeated successfully  
- A reproducibility study in psychology found that only <a href="http://www.nature.com/news/first-results-from-psychology-s-largest-reproducibility-test-1.17433" target="_blank">39 of 100 studies could be reproduced.</a>  
    + A quote: *“A lot of working scientists assume that if it’s published, it’s right... This makes it hard to dismiss that there are still a lot of false positives in the literature.”*  
- <a href="http://www.theatlantic.com/health/archive/2015/08/psychology-studies-reliability-reproducability-nosek/402466/" target="_blank">Recent Atlantic Article by Ed Yong</a>  
- The Journal *Nature* on the <a href="http://www.nature.com/news/reproducibility-1.17552#/" target="_blank">issue of reproducibility</a>  
    + A comment from the journal *Nature*: "Nature and the Nature research journals will introduce editorial measures to address the problem by improving the consistency and quality of reporting in life-sciences articles... **we will give more space to methods sections. We will examine statistics more closely and encourage authors to be transparent, for example by including their raw data.**"  
    + Nature also released a <a href="http://www.nature.com/authors/policies/checklist.pdf" target="_blank">checklist</a> with an albeit *wimpy* computational check *(see #18)*.  
        + This talk will hopefully help all of us improve this step.  

***

## How to replicate this Figure 1?
![](Figures/try.png)

#### To replicate this figure 1, we need...  
- Sequencing data!
- All other data (i.e. where in the water column it was sampled, particle-association, which lake, nutrient profile, mixing status)  
- The code!  Which should be:  
    + Easily read-able  
    + Very well commented and documented.  
          + Using set.seed! 
    + How to calculate:  
        + Observed richness?  
        + Simpson's Evenness?  
- The versions of the software and packages/libraries used  

### Could I replicate Figure 1 from your last publication?  
If not, what would **you and your co-authors** need to provide or do so I could replicate Figure 1 from your last publication?


***

## As scientists, it should be our goal to perform **robust** and **reproducible** research.

- "**Robust research** is about doing small things that stack the deck in your favor to prevent mistakes." *~Vince Buffalo*
- **Reproducible research** may be repeated by other researchers with the same results. 

### Reproducibility can be difficult with genomic data.  
1. Genomics data is too large and high dimensional to easily inspect or visualize.  Usually, workflows involve multiple steps and it's not feasible to inspect every step.  
2. Unlike in the wet lab, we don't always know what to expect of our genomics data analysis.  
3. It's difficult to distinguish *good* from *bad* results.  
4. Scientific code is usually only run once to generate results for a publication, and is therefore more likely to contain (silent) bugs.  
    + **Silent errors** arise from code that may produce unknowingly incorrect output (rather than stop with an error message).


### What are the ingredients to robust and reproducible research?  

- **Work must be well documented!  Methods, code, and data must be made available to others!**  
- Adopt a cautious attitude and *check everything*.    
    + Vince Buffalo's golden rule of bioinformatics:  "Never ever trust your tools (or data)"  
    + Remember, "garbage in, garbage out" - an analysis is only as good as the data going in.  
    + Let the data prove that it is high quality.
- **Take the time to develop fequently used scripts into tools.**  
    + Then have your lab mates or collaborators test them and try to break them.  
- **Collaborate!**  
    + Do paired-programming with your labmates and collaborators.  
    + Hack-a-thons! 



### What's the benefit for **_you?_**  
Yeah, it takes a lot of effort to be robust and reproducible.  However, *it will make your life (and science) easier!*  

- Most likely, you will have to re-run your analysis more than once.  
- In the future, you or a collaborator may have to re-visit part of the project.  
- You can make modularized parts of the project into re-useable tools for the future.  
- Reproducibility makes you easier to work and collaborate with.  

***
## Some Recommendations

### 5 Recommendations for Robust Research  

#### 1a. Write code for humans

**Code readability is very important.**  

- Code should be broken down into small chunks that may be re-used.  
    + Do not re-write code to do the same task over and over again.  
        + Do not repeat yourself!  (Who wants to read that?)  
            + If you need to, make it a tool/function.  
- Do not require readers to have to think of multiple facts at once.  
- Make names/variables consistent, distinctive and meaningful.  
- Adopt a <a href="https://github.com/google/styleguide" target="_blank">style</a> and format and keep it consistent.  
- Be a concise and clear commenter.  
    
If your code is more readable, then:  

- Your project is more reproducible.  
- It's easier to find and correct bugs.  
- You will be your friend in the future when you revisit the code.

#### 1b. Write data for computers

**Let your computer do the work for you**  
Format your data so its easily read by your computer, not by you or other humans.  

- Code written for people to read requires cleaning and tidying to be processed by a computer.  
- Name data files in a consistent way.  
    + Automating tasks will be easier, which will prevent you from making trivial mistakes.

#### 2. Make incremental changes.  
- Work in small steps with frequent feedback.  
    + Have a friend or labmate test your code and try to break it.  
    + Challenge your PI to test your code!
- Use version control!  
- Put all manual changes under version control, too! 


#### 3. Be a "Defensive Programmer" - Make Assertions  
Add tests within your code to make sure your code is doing what it is supposed to do.  

Assertions are statments that something holds true.  Assertions:  
1. Ensure that if something goes wrong, the program will stop.  
2. They also explain what the program is doing.

- In R you can use `stopifnot()`  
    + The `testthat` package is made for this!  Check it out the <a href="http://journal.r-project.org/archive/2011-1/RJournal_2011-1_Wickham.pdf" target="_blank">testthat package here</a>  
- In python you can use `assert()`

#### 4. Use existing libraries (packages) whenever possible  
- Do not try to re-invent the wheel while your performing your data anaylsis.  
- Use functions that have already been written and tested for you.

#### 5. Prevent catastrophe and help reproducibility by making your data **read-only**
Read-only is important because:  

- Modifying data can corrupt your results.    
- It's easy to lose track of how you have changed a file when you modify it in place.  



#### 5 Recommendations for Robust Research  
1. Write code for humans, write data for computers 
2. Make incremental changes.
3. Make assertions and be loud, in code and in your methods  
4. Use existing libraries (packages) whenever possible  
5. Prevent catastrophe and help reproducibility by making your data **read-only** 






***

### 6 Recommendations for reproducible research  


#### 1. Encapsulate the full project into one directory that is supported with version control.  
The **Reproducible-Science-Curriculum** <a href="https://github.com/Reproducible-Science-Curriculum/rr-init" target="_blank">Github repo for Reproducible Research Project Initialization</a> is a great place to start a reproducible research project.  

#### 2. Release your code and data
It is simple.  Without your code and data, your research is not reproducible.

#### 3. Document everything!   
**Bottom line:  Adopt a computing notebook that is as good as a wet-lab notebook**.

To fully reproduce a study, each step of analysis must be described in much more detail than can be included in a publication.

Include a record of your steps, where files are, where they came from, and what they contain.  

Include `session_info()` in your document, preferably at the bottom. Session info lists the version of R that you’re using plus all of the packages you’ve loaded. 

#### In your computing notebook:  

- Document your methods and workflows  
- Document the origin of all data in your project directory  
- Document **when** and **how** you downloaded the data  
- Record **data** version info   
- Record **software** version info with `session_info()`

For example, all the above information could be stored in a *README* file 


#### 4. Make figures, tables, and statistics the results of scripts.

Using `inline code` can make the creation of tables much easier if the data changes!

#### 5. Write code that uses relative paths.

Do not rely on hard-coded absolute paths (i.e. /Users/marschmi/Data/seq-data.csv or even ~/Data/seq-data.csv).  

Relative paths (i.e. Data/seq-data.csv) or command line arguments are better alternatives.

#### 6. Always Set your seed  

If there is any randomizations of data or simulations, use `set.seed()` in the first code chunk.

<a href="http://kbroman.org/knitr_knutshell/pages/reproducible.html" target="_blank">Karl Broman</a> suggests to open R and type runif(1, 0, 10^8) and then paste the resulting large number into `set.seed()` in the first code chunk. If you do this, then the random aspects of your analysis should be repeated the same way.


#### 6 Recommendations for reproducible research  
1. Encapsulate the full project into one directory that is supported with version control.  
2. Release your code and data.  
3. Document everything and use code as documentation!
4. Make figures, tables, and statistics the results of scripts.  
5. Write code that uses relative paths.  
6. Always Set your seed.  

***

#### How can you revise your work flow?  

- Where you can introduce **robust** steps?  
- Where can you add **reproducible** steps?


***

#### Do you have ...  
1. <a href="https://www.rstudio.com/products/rstudio/download/" target="_blank">RStudio?</a>  
2. <a href="http://www.inside-r.org/download" target="_blank">R?</a>  
    + Please install these packages:  
        + `install.packages("knitr")`  
        + `install.packages("rmarkdown")` 
3. A <a href="https://github.com/" target="_blank">Github</a> account?  
4. <a href="https://git-scm.com/book/en/v2/Getting-Started-Installing-Git" target="_blank">Git?</a>  
    + If you have not already, please generate an <a href="https://help.github.com/articles/generating-ssh-keys/" target="_blank">SSH key.</a>

You are good to go!

***

## Using Git through RStudio  

1.  Sign into <a href="https://github.com/" target="_blank">Github</a>.  
2. <a href="http://angus.readthedocs.org/en/2015/CTB-github.html" target="_blank">Initialize repo on the github page</a>.  
    + Name the Repo "NGS_Reproducibility"  
    + Down in the right-hand corner, copy the SSH clone URL, do not copy the https url!   
3. Open up RStudio  
4. File -> New Project -> Version Control -> Git -> Paste the **SSH clone URL**.  
    + Be sure to use the same repo name as on your github page!  
    + If you get the following error, you may have copied the **HTTPS** clone URL instead of the **SSH** clone url.  
```
Cloning into 'repo_name'...
error: unable to read askpass response from 'rpostback-askpass'
fatal: could not read Username for 'https://github.com': Device not configured
```


***

## RMarkdown Time!
    
    
### What is R Markdown?   
* RMarkdown is a variant of Markdown that has embedded R code chunks to be used with `knitr` to make it easy to create reproducible web-based reports.  
      + **Markdown:**  A system for writing simple, readable text that is easily converted to html. 
          + Allows you to write using an easy-to-read, easy-to-write plain text format.              
*  Rmd -> md -> html (docx, pdf)  
*  Can include both text and code to execute  
      
    
### Why R Markdown?
A convenient tool for reproducible and dynamic reports with R!       

- Execute code with `knitr`.   
- Easy to learn syntax.  
- Include LaTeX equations.  
- Don't need to worry about page breaks or figure placement.  
- Consolidate your code and write up into a single file:  
    + Slideshows, pdfs, html documents, word files  
- It's **so easy** to use with version control with Git!   

### Simple Workflow

![](Figures/rmd_workflow.jpeg)


#### How to Open an Rmd File
![](Figures/new.jpeg)  

![](Figures/newRMD.jpeg)



#### Choose Output
**YAML Header:**  A set of key value pairs at the start of your file.  Begin and end the header with a line of three dashes (---)

***R Studio template writes the YAML header for you***  

output: html_document  
output: pdf_document  
output: word_document  
output: beamer_presentation (beamer slideshow - pdf)  
output: ioslides_presentation (ioslides presentation - html)  

**For example:**  Here's the YAML header for this webpage with a table of contents.
```
---
title: "Reproducible Research Using RMarkdown and Git through RStudio"
subtitle: "Tutorial for NGS Workshop Week 3"
author: "Marian L. Schmidt, @micro_marian, marschmi@umich.edu"
date: "August 25, 2015"
output:
  html_document:
    theme: united
    toc: yes
---
```

#### Markdown basics 
Markdown is a simple formatting language that is easy to use

- Create lists with `*` or `+` sign   
      + like this
      + and this
- Use one or two asterisk marks to provide emphasis such as `*`*italics*`*` and `**`**bold**`**`.  Can even include tables:    

First Header  | Second Header
------------- | -------------
Content Cell  | Content Cell
Content Cell  | Content Cell


#### Markdown basics 
![](Figures/Rmd_QuickRef.jpeg)

## R Code Chunks 

```
Code blocks display with fixed-width font
```

```{r fig.width = 4, fig.height = 3, message = FALSE}
#quick summary
library(ggplot2)
min(diamonds$price)
mean(diamonds$price)
max(diamonds$price)
```


#### More R Code Chunks 
![](Figures/chunk_example.jpeg)


+ You can name the code chunk.  

+ echo = TRUE:  The code **will** be displayed.   

+ eval = TRUE: Yes, execute the code.

#### R Code Chunk Arguments
![](Figures/knitr_arguments.jpeg)

#### R Code Chunks:  Displaying Plots 
![](Figures/figure.jpeg)

```{r figure, echo = FALSE, fig.width = 5, fig.height = 4, fig.align='center'}
#quick plot of ggplot2's diamonds dataset
qplot(carat, price, data = diamonds, color = cut)
```


### Global Chunk Options

You may want to use the same set of chunk options throughout a document and you don't want to retype those options in every chunk.  

**Global chunk options are for you!**

```{r global_options, include = FALSE}
knitr::opts_chunk$set(fig.width = 12, fig.height = 8, 
                      echo = FALSE, warning = FALSE, message = FALSE)
```

![](Figures/global.jpeg)



### Inline R Code  

You can evaluate expressions inline by enclosing the expression within a single back-tick qualified with `r`.  

Inline code is underappreciated!  

Last night, I saw `r 5 + 2` shooting stars!  


![](Figures/shootingstars.jpeg)



### Rendering document
1.  Run `rmarkdown::render("<filepath>")`  
2.  Click the very cute **knit HTML** button at the top of the RStudio scripts pane  

When you render, R will:  

- Execute each embedded code chunk and insert the results into your report.  

- Build a new version of your report in the output file type.  

- Open a preview of the output file in the viewer pane.  

-  Save the output file in your working directory.  



## A Git Workflow within RStudio

1. Make some changes to your document that you would like to save a *copy* of.  
2. **Git add** by checking the box under "staged" in the git screen.  
    + *Hint: The git screen is in the same pane as the RStudio Environment and history.*  
3. Draft your commit message.  
    + It should be a **meaningful message!**  
    + Think of you in 6 months looking for changes you had made to your document.  Don't you want to be your own friend?  
    + Do not allow your commit messages to get <a href="https://xkcd.com/1296/" target="_blank">less informative</a> as your project continues  
4. Click **"Commit."**  
5. Perform **Git Push** by clicking the bright green arrow, which is the "git Push" button.  
6. Make sure everything is pushed to the remote repository without any errors.  

Click on the clock button to view your git history.  Here, you can also view the difference between documents.  


## Challenge  
#### Create an html output file with the analysis from the lesson on <a href="https://github.com/ngs-docs/angus/blob/2015/week3/visualizations/multivariate-tests/tests.md" target="_blank">Multivariate statistics By Ryan Williams</a> using data from <a href="http://germslab.org/" target="_blank">Adina Howe's Lab</a>. 



## Resources  
### Resources for Reproducible Research  
- Vince Buffalo's <a href="http://www.amazon.com/Bioinformatics-Data-Skills-Reproducible-Research/dp/1449367372" target="_blank">Bioinformatics Data Skills</a> book and it's helpful <a href="https://github.com/vsbuffalo/bds-files" target="_blank">github page.</a> 
    + Main source for this presentation.
- A course by Karl Broman at the University of Wisconsin-Madison on <a href="http://kbroman.org/Tools4RR/" target="_blank">Reproducible Research.</a>  
- Reproducible-Science-Curriculum <a href="https://github.com/Reproducible-Science-Curriculum/rr-init" target="_blank">Github repo for Reproducible Research Project Initialization</a>  
- <a href="http://ropensci.github.io/reproducibility-guide/" target="_blank">ROpenSci Reproducibility Research</a> guidelines  
- Publications:  
    + <a href="http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001745" target="_blank">Best Practices for Scientific Computing</a> by Wilson et al., 2014.  
    + <a href="http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1000424" target="_blank">A Quick Guide to Organizing Computational Biology Projects</a> by Noble, 2009.  
    + <a href="http://www.sciencemag.org/content/334/6060/1226.abstract" target="_blank">Reproducible Research in Computational Science</a> by Peng, 2011.  
- <a href="https://www.stat.wisc.edu/reproducible" target="_blank">Statistics Department Resources from the University of Wisconsin</a>  
- "Baby steps for the open-curious" from <a href="https://practicaldatamanagement.wordpress.com/2014/10/23/baby-steps-for-the-open-curious/" target="_blank">Christie Bahlai</a>  


### Resources for Rmarkdown, RStudio and R  
- Yihui Xie's <a href="http://www.amazon.com/Dynamic-Documents-knitr-Chapman-Series/dp/1482203537" target="_blank">Dynamic Documents with R and Knitr</a> and it's <a href="https://github.com/yihui/knitr-book/" target="_blank">github page.</a>  
- Resources from Jennifer Bryan's  <a href="http://stat545-ubc.github.io/git00_index.html" target="_blank">Stats 545</a>.  
- <a href="https://www.rstudio.com/wp-content/uploads/2015/03/rmarkdown-reference.pdf" target="_blank">RMarkdown Quick Reference Guide</a>  
- Christopher Gandrud's <a href="http://www.amazon.com/Reproducible-Research-Studio-Chapman-Series/dp/1466572841" target="_blank">Reproducible Research with R and RStudio</a> and it's <a href="https://github.com/christophergandrud/Rep-Res-Book" target="_blank">github page</a>  
- <a href="http://rmarkdown.rstudio.com/" target="_blank">RStudio RMarkdown Documentation</a>  
- <a href="http://www.rstudio.com/wp-content/uploads/2015/02/rmarkdown-cheatsheet.pdf" target="_blank">Rmd Cheatsheet</a>
- <a href="http://cran.r-project.org/web/packages/knitr/vignettes/knitr-refcard.pdf" target="_blank">Knitr Reference Card</a>
- <a href="http://www.cookbook-r.com/Graphs/" target="_blank">R Cookbook for ggplot</a>
  
  
### Other Resources  
- <a href="https://software-carpentry.org/" target="_blank">Software Carpentry Foundation</a>  
- <a href="http://datacarpentry.github.io/" target="_blank">Data Carpentry</a>  
- <a href="http://www.ted.com/talks/amy_cuddy_your_body_language_shapes_who_you_are?language=en" target="_blank">Power-Posing</a>



***

## About Me
- **Twitter:** @micro_marian  
- **e-mail:** marschmi@umich.edu
- I'm a native Michigander who likes:  
    + To code in R  
    + Sharing Code  
    + Reproducible science  
- PhD Candidate in the Department of Ecology and Evolutionary Biology at the University of Michigan in <a href="http://www-personal.umich.edu/~vdenef/" target="_blank">Vincent Denef's Lab</a>.  
- I recently got involved in the Software Carpentry foundation.  
- I'm a part of the U of M <a href="http://femmes.studentorgs.umich.edu/" target="_blank">**FEMMES**</a> (**F**emales **E**xcelling **M**ore in **Math** **E**nginnering and **S**cience) chapter.  FEMMES is a group that empowers 4-6th grade girls in STEM fields by holding after school science activities and a once-a-semester capstone "Day in the Lab" at U of M.  
- I went to **Hampshire College** for undergrad and wrote my thesis on the <a href="http://www.ncbi.nlm.nih.gov/pubmed/22865076" target="_blank">evolution of magnetotaxis in magnetotactic bacteria</a>.  
- Next, I worked as a lab tech for a year studying the intracacies of sulfate-reducing bacteria.  
- My dissertation is focused on the spatial and temporal distributions and activities of freshwater bacteria.  
    + Specifically, I work on the bacterial community composition of particle-associated (3-20 um) and free-living (0.22-3 um) bacteria within small inland lakes and in Lake Michigan.  
    + So far I have been using 16S rRNA amplicon data at the DNA-level. In the future, I hope to look at functional differences between the particle-associated and free-living bacterial fractions by using metatranscriptomics.
- I get to work in this beautiful place...  
![](Figures/LakeMichigan.jpeg)  
- When I'm not doing science-y things, you can find me here:  
![](Figures/climbing2.jpg) 

***

#### Session Info  
<a href="http://kbroman.org/knitr_knutshell/pages/reproducible.html" target="_blank">Karl Broman</a> recommends using the `session_info()` from the devtools package.
```{r Presentation session_info, include=TRUE, echo=TRUE, results='markup'}
devtools::session_info()
```