{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Trimming and Filtering a FASTQ\n", "\n", "\n", "## Shell Variables\n", "Assign the variables in this notebook." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# We used these in the last notebook\n", "CUROUT=$HOME/work/scratch/2015_output\n", "DEMUX=$CUROUT/demux_fastqs\n", "\n", "# The following are new for this notebook\n", "TRIMMED=$CUROUT/trimmed_fastqs\n", "INFO=$HOME/work/myinfo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Making New Directories\n", "Make the directories that are new in this notebook" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "mkdir -p $TRIMMED\n", "mkdir -p $INFO" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's check to be sure that worked. We will run `ls` and check that these directories now exist in the `$CUROUT` directory." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "ls $CUROUT" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Trimming and Filtering\n", "Now we get into some actual preprocessing. We will use `fastq-mcf` to trim adapter from our reads and do some quality filtering. We need to trim adapter, because if a fragment is short enough, we will sequence all the way through the fragment and into the adapter. Obviously the adapter sequence in not found in the genome, and can keep the read from aligning properly. To do the trimming, we need to generate an adapter file.\n", "\n", "## Making an adapter file\n", "The first step is to get the adapter sequence. For our test file, the index is 16 (barcode CCGTCC), this is from NEBNext Multiplex Oligos for Illumina (Index Primers Set 2). So let's look in the manual: \n", "\n", "https://www.neb.com/~/media/Catalog/All-Products/6B6FC6C03B274E7FA0FDBF13015AB194/Datacards%20or%20Manuals/manualE7500.pdf\n", "\n", "We will get the sequence for the universal primer and the index primer. Now we need to make the adapter file; this needs to be in FASTA format.\n", "\n", "1. click on the jupyter \"File\" menu, and select \"Open\". \n", "2. When the the new browser window/tab opens, click on the \"Files\" tab if it is not already active.\n", "3. Click on the \"home\" symbol to go to the top level directory, then click on \"myinfo\"\n", "4. In the \"New\" menu select \"Text File\".\n", "5. In this text file, type \">adapter_3prime\", this is the name for the adapter sequence, the \">\" is part of the FASTA format indicating that it is the sequence name.\n", "6. Next copy the part of the 3' common part of the adapter, the part that is to the right of the barcode sequence, and paste it on the line after the name.\n", "7. We also want to include the reverse complement of the adapter, in case that is what will actually be sequenced. The easiest way to do that is to use https://www.bioinformatics.org/sms/rev_comp.html to generate the reverse complement, then name it something like \"adapter_3prime_RC\"\n", "8. Now clean up by making sure that . . .\n", " 1. The sequence is on its own line\n", " 2. The sequence has a name on the line before it\n", " 3. The name is preceded by a \">\"\n", " 4. All spaces and non-sequence characters have been removed\n", "9. We need to repeat steps 5-8 for the **universal** adapter sequence that is in the same PDF.\n", "10. Click on \"untitled.txt\" to change the file name to \"neb_adapters.fasta\"\n", "11. Save the file.\n", "\n", "\n", "## fastq-mcf\n", "You can run `fastq-mcf -h` to get details about running fastq-mcf. We will adjust run parameters, because some of the defaults set a low bar (even the author acknowleges this)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# the \"| cat\" is a hack that prevents problems with jupyter\n", "fastq-mcf -h | cat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Running fastq-mcf\n", "1. neb_adapters.fasta : the adapter file\n", "2. r1.8A_pilot.fq.gz : the FASTQ with the data (fastq-mcf, like most NGS analysis software, detects gzipped files and automatically decompresses on the fly)\n", "3. -q 20 : if a read has any bases with quality score lower than this, trim them and anything 3' of that base\n", "4. -x 0.5 : if this percentage (or higher) of the reads have an \"N\" in a given position, trim all reads to that position\n", "5. -o r1.test.trim.fastq.gz : output file (the .gz ending tells fastq-mcf to compress the output file)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "fastq-mcf $INFO/neb_adapters.fasta \\\n", " $DEMUX/r1.8A_pilot.fq.gz \\\n", " -q 20 -x 0.5 \\\n", " -o $TRIMMED/r1.8A_pilot.trim.fastq.gz" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that it was the reverse complement of the adapter that was actually found, so its a good thing that we actually included it.\n", "\n", "at this point we could run fastqc on the output of fastq-mcf to see if statistics have improved, but we will skip that for now." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "ls $TRIMMED" ] } ], "metadata": { "kernelspec": { "display_name": "Bash", "language": "bash", "name": "bash" }, "language_info": { "codemirror_mode": "shell", "file_extension": ".sh", "mimetype": "text/x-sh", "name": "bash" } }, "nbformat": 4, "nbformat_minor": 0 }