{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Data Provenance\n", "\n", "## [Provenance](https://www.merriam-webster.com/dictionary/provenance):\n", "1. origin, source\n", "2. the history of ownership of a valued object or work of art or literature\n", "\n", "## [Data Lineage/Data Provenance](https://en.wikipedia.org/wiki/Data_lineage):\n", "Data lineage includes the data's origins, what happens to it and where it moves over time. Data lineage gives visibility while greatly simplifying the ability to trace errors back to the root cause in a data analytics process.\n", "\n", "It also enables replaying specific portions or inputs of the data flow for step-wise debugging or regenerating lost output. Database systems use such information, called data provenance, to address similar validation and debugging challenges. Data provenance refers to records of the inputs, entities, systems, and processes that influence data of interest, providing a historical record of the data and its origins. The generated evidence supports forensic activities such as data-dependency analysis, error/compromise detection and recovery, auditing, and compliance analysis. \n", "\n", "## Shell Variables\n", "Assign the variables in this notebook." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "source bioinf_intro_config.sh" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Checksums\n", "### MD5SUM\n", "\n", "\n", "Get example code from TRICEM talk\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Running MD5SUM" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ls $RAW_FASTQS" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "head $RAW_FASTQS/Granek_4837_180427A5.checksum" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cd $DATA_BASE\n", "md5sum -c $RAW_FASTQS/Granek_4837_180427A5.checksum" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "md5sum --help" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data Protection\n", "### chmod\n", "Follow md5sum example code from TRICEM talk\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "DEMO=$CUROUT/demo_chmod\n", "mkdir -p $DEMO\n", "cd $DEMO" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pwd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# TriCEM Talk\n", "\n", "---\n", "title: \"Reproducible Analysis\"\n", "author: \"Josh Granek\"\n", "date: \"June 8, 2018\"\n", "output:\n", " md_document:\n", " variant: markdown_github\n", " beamer_presentation: default\n", " ioslides_presentation: default\n", "---\n", "\n", "```{r setup, include=FALSE}\n", "knitr::opts_chunk$set(echo = FALSE)\n", "```\n", "\n", "\n", "# Overview\n", "## Slides online\n", "A \"webpage\" version of these slides is available at https://bit.ly/2JD1lvm\n", "\n", "## What is Reproducible Analysis? {.build}\n", " \"Reproducible Analysis is an important part of reproducible research. Reproducible analysis requires that all components of the analysis be archived so that _anyone_ can independently repeat the analysis and arrive at exactly the same results\" - Josh Granek\n", "\n", "### Who is \"anyone\"\n", "> - Labmates\n", "> - Collaborators\n", "> - Competitors\n", "> - Everyone else interested in your work\n", "> - Someone who want's to apply your analysis to their own data\n", "> - You, 6 months from now\n", "\n", "## Not Covered Here\n", " - Reproducibility in the Wet Lab\n", " - Lots of Other Stuff\n", " - Important Details\n", "\n", "## Soapbox\n", " 1. Setup\n", " 2. Stand on\n", "\n", "## Reproducible Analysis\n", " . . . is like eating your vegetables\n", "\n", "## Three Pillars of Reproducible Analysis {.build}\n", " - Raw Data\n", " - Analysis\n", " - Compute Environment\n", " \n", "### Quick Start Guide\n", " - Archive It!\n", "\n", "# Raw Data\n", "\n", "## Raw Data\n", " - READ-ONLY\n", " - Provenance\n", " - Archive\n", "\n", "## READ-ONLY\n", "```{bash eval=FALSE, echo=TRUE}\n", "chmod -R a-w my_raw_data_directory\n", "```\n", "\n", "## Provenance: Checksums\n", "```{bash echo=TRUE, include=TRUE}\n", "mkdir -p /tmp/mydata\n", "echo \"1,2,3,4\" > /tmp/mydata/data1.csv\n", "echo \"5,6,7,8\" > /tmp/mydata/data2.csv\n", "\n", "md5sum /tmp/mydata/*.csv\n", "```\n", " \n", "## Provenance: Checksums (continued)\n", "\n", "```{bash echo=TRUE, include=TRUE}\n", "md5sum /tmp/mydata/*.csv > /tmp/mydata/mydata_md5.txt\n", "```\n", " \n", "```{bash echo=TRUE, include=TRUE}\n", "md5sum -c /tmp/mydata/mydata_md5.txt\n", "```\n", "\n", "```{bash echo=TRUE, include=TRUE, error=TRUE}\n", "echo \"5,6,7,8,9\" > /tmp/mydata/data2.csv\n", "md5sum -c /tmp/mydata/mydata_md5.txt\n", "```\n", "\n", "## Archiving Raw Sequence Data @ NCBI\n", " - [GEO: Gene \"Expression\" Data](https://www.ncbi.nlm.nih.gov/geo/)\n", " - GEO: Gene Expression Omnibus\n", " - RNA-Seq\n", " - ChIP-Seq\n", " - [SRA: Everything else](https://www.ncbi.nlm.nih.gov/sra)\n", " - SRA: Sequence Read Archive\n", "\n", "## Alternatives to NCBI\n", " - [International Nucleotide Sequence Database Collaboration](http://www.insdc.org)\n", " - [European Nucleotide Archive (ENA)](https://www.ebi.ac.uk/ena)\n", " - [DNA Database of Japan](https://www.ddbj.nig.ac.jp/index-e.html)\n", "\n", "## Other Data Types\n", " ???????????????\n", "\n", "# Analysis Methods\n", "## Analysis Methods\n", " - Script Everything\n", " - Use Version Control\n", " \n", "## Script Everything\n", " - [No Excel](https://www.bloomberg.com/news/articles/2013-04-18/faq-reinhart-rogoff-and-the-excel-error-that-changed-history)\n", "\n", "## Publish Full Analysis Pipeline\n", " - Scripts (e.g. R, Python, Matlab, etc)\n", " - run parameters are embedded\n", " - Metadata\n", " - Documentation\n", " - Manuscript (optional)\n", " \n", "## What is Version Control?\n", "```{r, out.width = \"400px\"}\n", "knitr::include_graphics(\"http://www.phdcomics.com/comics/archive/phd101212s.gif\")\n", "```\n", "\n", "## What is Version Control?\n", " - Track\n", " - Backup\n", " - Rewind\n", " - Branch\n", " - Collaborate\n", " - Publish\n", " \n", "## What is Version Control?\n", "```{r, out.height = \"400px\"}\n", "knitr::include_graphics(\"./git_diff.png\")\n", "```\n", "\n", "\n", " \n", "\n", "\n", " \n", "## Version Control Software\n", " - git\n", " - mercurial\n", " - etc\n", "\n", "## Git-repository Hosts\n", " - Github ([Education Discount](https://help.github.com/categories/teaching-and-learning-with-github-education/))\n", " - Bitbucket\n", " - Gitlab\n", " - Duke Gitlab\n", "\n", "## Reproducible Computing Environment\n", "### Containerization for Reproducible Research\n", "\n", " - *Versioning*: Lock down the specific computing environment used for an analysis\n", " - Portability: Runs on Linux, Mac, and Windows\n", " - Sharebility: [Docker Hub](https://hub.docker.com)/[Singularity Hub](https://singularity-hub.org)\n", " - Scalability: Runs on a laptop, massive server, and everything in between\n", " \n", "### Container Platforms\n", " - [Docker](https://docs.docker.com/get-started/#docker-concepts)\n", " - [Singularity](http://singularity.lbl.gov/)\n", "\n", "# Organization\n", "## My strategy\n", "1. Raw Data directory: must be *read-only*\n", "2. Output directory: everything generated by a script\n", "3. Git repository: Code and metadata \n", "\n", "## Alternatives\n", "1. [ProjectTemplate](https://swcarpentry.github.io/r-novice-gapminder/02-project-intro/#tip-projecttemplate---a-possible-solution)\n", "2. [Resources on Project Directory Organization](https://discuss.ropensci.org/t/resources-on-project-directory-organization/340)\n", "3. [A Quick Guide to Organizing Computational Biology Projects](https://doi.org/10.1371/journal.pcbi.1000424)\n", "4. [Designing projects](https://nicercode.github.io/blog/2013-04-05-projects/)\n", "\n", "# Resources\n", "## Resources\n", "### Scientific Computing Advice\n", " - [Good enough practices in scientific computing](https://doi.org/10.1371/journal.pcbi.1005510)\n", " - [Best Practices for Scientific Computing](https://doi.org/10.1371/journal.pbio.1001745)\n", "\n", "### Git for Version Control\n", " - [Introduction to Version Control with Git](https://swcarpentry.github.io/git-novice/)\n", " - [Installing Git](https://swcarpentry.github.io/workshop-template/#git)\n", " - [Sourcetree: a free GUI for git](https://www.sourcetreeapp.com)\n", " - [Git in RStudio](https://gitlab.oit.duke.edu/IBIEM/IBIEM_2017_2018/blob/master/git_material/git_overview.md)\n", "\n", "\n" ] } ], "metadata": { "kernelspec": { "display_name": "Bash", "language": "bash", "name": "bash" }, "language_info": { "codemirror_mode": "shell", "file_extension": ".sh", "mimetype": "text/x-sh", "name": "bash" } }, "nbformat": 4, "nbformat_minor": 1 }