Cleaning illumina reads - LSCC Lab Meeting - Fri 23 Nov 2012

Cleaning Illumina reads
Torsten Seemann
VLSCI :: LSCC - Lab Meeting - Fri 23 Nov 2012

Illumina reads
● Usually 100 or 150 bp
○ 250bp rolling out now, 400bp next year
● Indel errors rare
● Homopolymer errors rare
● Substitution errors < 1%
○ Error rate higher at 3' end
● Adaptor issues
○ rare in HiSeq (TruSeq prep)
○ more common in MiSeq (Nextera prep)
● Very high quality overall

Illumina libraries
●"single end"
○just a shotgun read sequenced from one end
●"paired end"
○~500bp fragments sequenced at both ends
○very reliable
●"mate pair"
○circularized 2-10 kbp fragments sequencing
○then "paired end" protocol
○reliability varies

● garbage reads
○ instrument weirdness
● duplicate reads
○ low complexity library, PCR duplicate
● adaptor read-through
○ fragment too short
● indel errors
○ skipping bases, inserting extra bases
● uncalled base
○ couldn't reliably estimate, replace with "N"
● substitution errors
○ reading wrong base
Sequences have errors
More common
Less common

Why clean reads?
● Erroneous data may cause software to:
○ run more slowly
○ use more RAM
○ produce poor / biased / incorrect results
● Cleaning can:
○ improve overall average quality of the reads
■ hopefully giving a reliable result
○ reduce the volume of reads
■ some algorithms are O(N.logN) or O(N2
)
■ enable processing when otherwise couldn't
● (some software does handle them appropriately)

DNA sequence quality
● DNA sequences often have a
quality value associated with each
nucleotide
● A measure of reliability for each base
○ as it is derived from physical process
■ chromatogram (Sanger sequencing)
■ pH reading (Ion Torrent sequencing)
● Formalised by the Phred software for the
Human Genome Project

Phred qualities
● Q = -10 log10
P <=> P = 10- Q / 10
○ Q = Phred quality score
○ P = probability of base call being incorrect
Quality value Chance it is wrong Accuracy
10 1 in 10 90%
20 1 in 100 99%
30 1 in 1000 99.9%
40 1 in 10,000 99.99%
50 1 in 100,000 99.999%

Illumina quality plot
Y-axis is "Phred" quality values (higher is better)

Anatomy of a FASTQ entry
@read00179
AGTCTGATATGCTGTACCTATTATAATTCTAGGCGCTCAT
GCCCGCGGATATCGTAGCTATATGCTTCA
+
8;ACCCD?DD???@B9<9<CAC@=AAAA8A;B<A@882,+
495;;3990,02..-&-&-*,,,,(0**#
Start symbol
Sequence ID
Sequence
Separator line
Encoded quality values,
one symbol per nucleotide

FASTQ quality encoding
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJ
| | | | |
Q0 Q10 Q20 Q30 Q40
bad maybe ok good excellent
Uses letters/symbols to represent numbers:
Mnemonic:
"swear" words!
Mnemonic:
Q20 = '2' and '0'
Mnemonic:
'HI' = high Q

Spoiled for choice
Solexa
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefgh
..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
| | | | |
33 59 64 73 104
Illumina 1.3
...............................IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
| | | | |
33 59 64 73 104
Illumina 1.5
.................................JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
| | | | |
33 59 64 73 104
Illumina 1.8 (Sanger)
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS...............................
| | | | |
33 59 64 73 104

Ambiguous bases
● If there is ambiguity in the base call, an "N" is used
@ILLUMINA:6:1:964:115#GATCAG/1
GGACCTGAGAGTGTGCATGAAGAGGGCAGCGCGCACNGCATCA
+
HHHGFGEEECDEBA@BBBA<=<;:98743720&,+**_%$#"!
● Possible software responses:
○ Crash!
○ Ignore it
○ Silently convert to fixed or random base (Velvet > 'A')
○ Handle it appropriately (Bowtie2)
● Small proportion overall, safer to discard

Homopolymers
● A read consisting of all the same base
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+
● Often occur from clusters at edge of flowcell lane
● Early Illumina software called '?' as 'A' rather than N
● Unlikely to be present in real DNA
● Best to discard
● Less common with newer Illumina OLB software

Quality trimming
●Remove low quality sequence
○Q=13 corresponds to 5% error (p=0.05)
○Q=0..13 encoded by: !"#@%&'()*+,-/
GGACCTGAGAGTGTGCATGAAGAGGGCAGCCCCGCACTGCATG
+
●Can trim per
○each base
○window moving average eg. 3 base mean
○minimum % good per window eg. need 4 of 5

Illumina Adaptors
● Used in the sequencing chemistry
● Can appear at ends of read sequences
● Worse for mate-pair than for paired-end reads
● PCR Primer
CAAGCAGAAGACGGCATACGAGCTCTTCCGATCT
● Genomic DNA Sequencing Primer
CACTCTTTCCCTACACGACGCTCTTCCGATCT
● All other TruSeq & Nextera Adaptors
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN

Adaptor clipping
● Method
○ Align 3' and 5' read end against all adaptor sequences
○ If there is an anchored "match", trim the read
● Minimum length of match?
○ want to remove adaptor, but not real sequence [10 bp]
● Allow substitutions in match?
○ as reads have errors, need some tolerance [1 sub]
● Allow gaps/indels in match?
○ indels are unlikely in Illumina reads [no]
● Slow to perform compared to other preprocessing steps

Decloning
●Illumina "mate pair" sequencing
○Requires a lot of starting DNA
○Challenging protocol to implement reliably
○Not enough final DNA leads to PCR duplicates
○Coverage is highly non-uniform and sporadic
○Causes bias in analyses
●Decloning
○Replace clones with a single representative
○Choose representative with highest quality
○Helps salvage usable information content

Read length
●Enforce a minimum read length L
●k-mer based tools
○break reads into k-mers, so L < k is pointless
○Velvet, Trinity, Gossamer, ...
●Uniqueness
○desire reasonable uniqueness of sequence,
otherwise multiple mapping will take forever!
○L=24 is bare minimum (I reckon)
○BWA, Bowtie, Shrimp, ...

Other strategies
●Digital normalisation
○remove low frequency k-mers
○remove reads w/ too many low freq k-mers
●Error correction
○replace low frequency k-mers with their
"closest" high-frequency k-mer
○other methods I don't understand yet
●Just use local alignment to "solve" the problem

Walk-through
1. Original read + quality =43bp
GTTAGCGCGCTGACCATGATTCAAGGAACTGGCCCCATTNATA
hhhhghfeefaa^a^[[[^X[[XX^^^^`SSTQPZZBBBBBBB
2. Homopolymer? No
3. Ambiguious N bases? Yes, 1
4. Quality < 20 ? Yes, at 3' end
5. Adaptor sequences > 8bp ? Yes, 9 bp at 5'
end
6. Combine all masks Logical intersection
7. Extract longest sub-sequence =19bp
TGACCATGATTCAAGGAAC

The coral genome
● Raw data (A.millepora Illumina)
○ 9 libraries - 3 x PE, 6 x MP - 200bp to 10kbp
○ 92.0 Gbp, 943M reads, average length 98bp
● Method
○ Decloned all MP libs, disallow Ns, reject homopolymers,
trim Q < 20 + clip adaptors, minimum length 55bp
● Cleaned data
○ 42.5 Gbp, 478M reads, average length 88bp
● Effect
○ Good - de novo Velvet assembly improved overall
■ validated by RNA-Seq, ESTs
○ Bad - lower coverage
■ coral is very heterozygous > 10%

Per library yields (Gbp)
●Library Raw Cleaned %Kept
●pe_193 9.55 6.71 70
●pe_463 19.19 13.89 72
●pe_580 4.87 3.18 65
●mp_2200 18.48 8.48 46
●mp_2820 13.54 1.54 11 *
●mp_4628 12.95 0.85 6 *
●mp_5000 6.92 2.98 43
●mp_8000 4.33 1.64 38
●mp_10000 2.15 0.25 11 *
●single n/a 3.00 n/a
●TOTAL 92.00 42.50

The GKP FFPE BWA PE mystery
●FFPE sample
○formalin-fixed, paraffin embedded
○long term tissue archival storage method
●Sequencing
○HiSeq-2000
○22 million x 100bp paired-end reads
○quality looks good ... but not mapping!

FastQC is giving us some hints...

Nesoni
●Implemented primarily by Paul Harrison @ VBC
○swiss army knife
○snp calling, phylogenetics, DGE, ....
○extendible pipeline system (Python)
●We used the "nesoni clip:" module
○adaptor clipping = on
○min quality = Q10
○allow Ns = no
○min length = 2

Nesoni command line
nesoni clip:
--quality 10
--length 24
--out-separate yes
SM_AdMP1_ID_07B26948_L004_CLIPPED
pairs:
SM_AdMP1_ID_07B26948_L004_R1.fastq.gz
SM_AdMP1_ID_07B26948_L004_R2.fastq.gz
SM_AdMP1_ID_07B26948_L004_CLIPPED_R1.fq.gz
SM_AdMP1_ID_07B26948_L004_CLIPPED_R2.fq.gz
SM_AdMP1_ID_07B26948_L004_CLIPPED_single.fq.gz
Orphans

Nesoni clip: R1 @ 5' end
(> SM_AdMP1_ID_07B26948_L004_CLIPPED 161,553 read-1 adaptors clipped at start
10 bases: 11757 avg errors: 0.95 3948xPE_Read_2_Sequencing_Primer 1260xMultiplexing_Adapters_1 ...
11 bases: 3212 avg errors: 0.74 758xMultiplexing_PCR_Primer_2.0 269xPE_Read_2_Sequencing_Primer ...
12 bases: 1739 avg errors: 0.62 545xTruSeq_Adapter_Index_1 453xMultiplexing_PCR_Primer_2.0 ...
13 bases: 3054 avg errors: 0.38 2377xTruSeq_Universal_Adapter 429xMultiplexing_PCR_Primer_2.0 ...
14 bases: 760 avg errors: 0.34 357xMultiplexing_PCR_Primer_2.0 163xMultiplexing_Adapters_1 ...
20 bases: 2791 avg errors: 0.07 1481xMultiplexing_Adapters_1 1286xMultiplexing_PCR_Primer_2.0 ...
21 bases: 3946 avg errors: 0.05 3814xMultiplexing_PCR_Primer_2.0 123x3p_RNA_Adapter ...
23 bases: 4207 avg errors: 0.02 4162xMultiplexing_PCR_Primer_2.0 43xTruSeq_Adapter_Index_1 ...
25 bases: 1630 avg errors: 0.04 1612xMultiplexing_PCR_Primer_2.0 18xTruSeq_Adapter_Index_4
<snip>

(> SM_AdMP1_ID_07B26948_L004_CLIPPED 9,934,544 read-1 adaptors clipped at end
10 bases: 243222 avg errors: 0.04 224121xTruSeq_Universal_Adapter 11677xPCR_Primer_Index_1 ...
14 bases: 305000 avg errors: 0.02 295013xMultiplexing_PCR_Primer_2.0 7554xPCR_Primer_Index_3 ...
23 bases: 194228 avg errors: 0.02 183170xMultiplexing_PCR_Primer_2.0 5698xv1.5_Small_RNA_3p_Adapter ...
34 bases: 379388 avg errors: 0.02 209558xTruSeq_Adapter_Index_3 165637xMultiplexing_PCR_Primer_2.0 ...
35 bases: 217332 avg errors: 0.02 212927xTruSeq_Adapter_Index_4 3930xPCR_Primer_Index_4 ...
<snip>

(> SM_AdMP1_ID_07B26948_L004_CLIPPED 142,354 read-2 adaptors clipped at start
10 bases: 12898 avg errors: 0.97 3080xPE_Read_2_Sequencing_Primer 1837x3p_RNA_Adapter ...
11 bases: 4329 avg errors: 0.94 1123xTruSeq_Adapter_Index_1 940xTruSeq_Universal_Adapter ...
12 bases: 1813 avg errors: 0.42 1045xOligonucleotide_sequences_for_Genomic_DNA_Adapters_2 158xTruSeq_Universal_Adapter
13 bases: 3153 avg errors: 0.11 2431xOligonucleotide_sequences_for_Genomic_DNA_Adapters_2 569xTruSeq_Universal_Adapte
14 bases: 739 avg errors: 0.25 403xOligonucleotide_sequences_for_Genomic_DNA_Adapters_2 213xMultiplexing_PCR_Primer
16 bases: 465 avg errors: 0.20 295xOligonucleotide_sequences_for_Genomic_DNA_Adapters_2 82xMultiplexing_PCR_Primer_
18 bases: 2689 avg errors: 0.15 2280xOligonucleotide_sequences_for_Genomic_DNA_Adapters_2 237xMultiplexing_PCR_
21 bases: 1033 avg errors: 0.06 633xOligonucleotide_sequences_for_Genomic_DNA_Adapters_2 375xTruSeq_Universal_Adapter.
22 bases: 1530 avg errors: 0.04 1215xOligonucleotide_sequences_for_Genomic_DNA_Adapters_2 312xTruSeq_Universal_Adaptr
25 bases: 3437 avg errors: 0.02 2140xTruSeq_Universal_Adapter 1297xOligonucleotide_sequences_for_Genomic_DNA_Adapte
27 bases: 12247 avg errors: 0.01 11399xTruSeq_Universal_Adapter 48xOligonucleotide_sequences_for_Genomic_DNA_Adapters_2
29 bases: 3129 avg errors: 0.03 1833xOligonucleotide_sequences_for_Genomic_DNA_Adapters_21296xTruSeq_Universal_Adap
31 bases: 1533 avg errors: 0.04 1082xOligonucleotide_sequences_for_Genomic_DNA_Adapters_2 448xTruSeq_Universal_Adapt
32 bases: 2822 avg errors: 0.04 2398xOligonucleotide_sequences_for_Genomic_DNA_Adapters_2 419xTruSeq_Universal_Adapt
33 bases: 12425 avg errors: 0.03 10171xOligonucleotide_sequences_for_Genomic_DNA_Adapters_2254xTruSeq_Universal_Adapter
34 bases: 277 avg errors: 0.03 277xTruSeq_Universal_Adapter

(> SM_AdMP1_ID_07B26948_L004_CLIPPED 9,331,287 read-2 adaptors clipped at end
10 bases: 204601 avg errors: 0.03 200835xTruSeq_Universal_Adapter 556xOligonucleotide_sequences_for_G
11 bases: 200731 avg errors: 0.02 199519xTruSeq_Universal_Adapter 414xOligonucleotide_sequences_for_G
12 bases: 212465 avg errors: 0.01 212195xTruSeq_Universal_Adapter 45xTruSeq_Adapter_Index_1 ...
13 bases: 227861 avg errors: 0.01 227757xTruSeq_Universal_Adapter 31xPE_Adapters_1 ...
14 bases: 313533 avg errors: 0.01 312402xTruSeq_Universal_Adapter 830xPCR_Primers_2 ...
<snip>

Nesoni clip: summary
nesoni 0.92
FASTQ offset seems to be 33
(> SM_AdMP1_ID_07B26948_L004_CLIPPED 22,611,994 read-pairs
(> SM_AdMP1_ID_07B26948_L004_CLIPPED 121,236 read-1 too short after quality clip
(> SM_AdMP1_ID_07B26948_L004_CLIPPED 483,729 read-1 too short after adaptor clip
(> SM_AdMP1_ID_07B26948_L004_CLIPPED 22,007,029 read-1 kept
(> SM_AdMP1_ID_07B26948_L004_CLIPPED 100.000 read-1 average input length
(> SM_AdMP1_ID_07B26948_L004_CLIPPED 75.426 read-1 average output length
(> SM_AdMP1_ID_07B26948_L004_CLIPPED 673,175 read-2 too short after quality clip
(> SM_AdMP1_ID_07B26948_L004_CLIPPED 419,343 read-2 too short after adaptor clip
(> SM_AdMP1_ID_07B26948_L004_CLIPPED 21,519,476 read-2 kept
(> SM_AdMP1_ID_07B26948_L004_CLIPPED 100.000 read-2 average input length
(> SM_AdMP1_ID_07B26948_L004_CLIPPED 77.201 read-2 average output length
(> SM_AdMP1_ID_07B26948_L004_CLIPPED 21,147,301 pairs kept after clipping
(> SM_AdMP1_ID_07B26948_L004_CLIPPED 1,231,903 reads kept after clipping
started 21 November 2012 11:02 AM
finished 21 November 2012 11:43 AM
run time 0:40:06

Cleaning illumina reads - LSCC Lab Meeting - Fri 23 Nov 2012

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à Cleaning illumina reads - LSCC Lab Meeting - Fri 23 Nov 2012

Similaire à Cleaning illumina reads - LSCC Lab Meeting - Fri 23 Nov 2012 (20)

Plus de Torsten Seemann

Plus de Torsten Seemann (6)

Dernier

Dernier (20)

Cleaning illumina reads - LSCC Lab Meeting - Fri 23 Nov 2012