diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/AssignSomaticStatus.java b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/AssignSomaticStatus.java new file mode 100644 index 000000000..d0e318d8a --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/AssignSomaticStatus.java @@ -0,0 +1,283 @@ +/* +* By downloading the PROGRAM you agree to the following terms of use: +* +* BROAD INSTITUTE +* SOFTWARE LICENSE AGREEMENT +* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY +* +* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”). +* +* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and +* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions. +* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows: +* +* 1. DEFINITIONS +* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE. +* +* 2. LICENSE +* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute. LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation. +* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement. +* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement. +* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM. +* +* 3. PHONE-HOME FEATURE +* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM. Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time. Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation. +* +* 4. OWNERSHIP OF INTELLECTUAL PROPERTY +* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication. +* Copyright 2012-2015 Broad Institute, Inc. +* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc. +* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes. +* +* 5. INDEMNIFICATION +* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement. +* +* 6. NO REPRESENTATIONS OR WARRANTIES +* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME. +* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING. +* +* 7. ASSIGNMENT +* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void. +* +* 8. MISCELLANEOUS +* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries. +* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes. +* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4. +* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt. +* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter. +* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement. +* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles. +*/ + +package org.broadinstitute.gatk.tools.walkers.cancer; + +import org.broadinstitute.gatk.utils.commandline.Argument; +import org.broadinstitute.gatk.utils.commandline.ArgumentCollection; +import org.broadinstitute.gatk.utils.commandline.Output; +import org.broadinstitute.gatk.engine.arguments.StandardVariantContextInputArgumentCollection; +import org.broadinstitute.gatk.utils.contexts.AlignmentContext; +import org.broadinstitute.gatk.utils.contexts.ReferenceContext; +import org.broadinstitute.gatk.utils.refdata.RefMetaDataTracker; +import org.broadinstitute.gatk.engine.walkers.RodWalker; +import org.broadinstitute.gatk.engine.walkers.TreeReducible; +import org.broadinstitute.gatk.utils.MathUtils; +import org.broadinstitute.gatk.utils.QualityUtils; +import org.broadinstitute.gatk.engine.SampleUtils; +import org.broadinstitute.gatk.utils.exceptions.UserException; +import org.broadinstitute.gatk.engine.GATKVCFUtils; +import org.broadinstitute.gatk.utils.variant.GATKVariantContextUtils; +import htsjdk.variant.variantcontext.Genotype; +import htsjdk.variant.variantcontext.VariantContext; +import htsjdk.variant.variantcontext.VariantContextBuilder; +import htsjdk.variant.variantcontext.VariantContextUtils; +import htsjdk.variant.variantcontext.writer.VariantContextWriter; +import htsjdk.variant.vcf.*; + +import java.util.*; + +/** + * Assigns somatic status to a set of calls + */ +public class AssignSomaticStatus extends RodWalker implements TreeReducible { + @ArgumentCollection + protected StandardVariantContextInputArgumentCollection variantCollection = new StandardVariantContextInputArgumentCollection(); + + @Argument(shortName="n", fullName="normalSample", required=true, doc="The normal sample") + public String normalSample; + + @Argument(shortName="t", fullName="tumorSample", required=true, doc="The tumor sample") + public String tumorSample; + + @Argument(shortName="somaticPriorQ", fullName="somaticPriorQ", required=false, doc="Phred-scaled probability that a site is a somatic mutation") + public byte somaticPriorQ = 60; + + @Argument(shortName="somaticMinLOD", fullName="somaticMinLOD", required=false, doc="Phred-scaled min probability that a site should be called somatic mutation") + public byte somaticMinLOD = 1; + + @Argument(shortName="minimalVCF", fullName="minimalVCF", required=false, doc="If provided, the attributes of the output VCF will only contain the somatic status fields") + public boolean minimalVCF = false; + + @Output + protected VariantContextWriter vcfWriter = null; + + private final String SOMATIC_LOD_TAG_NAME = "SOMATIC_LOD"; + private final String SOMATIC_AC_TAG_NAME = "SOMATIC_AC"; + private final String SOMATIC_NONREF_TAG_NAME = "SOMATIC_NNR"; + + private final Set samples = new HashSet(2); + + /** + * Parse the familial relationship specification, and initialize VCF writer + */ + public void initialize() { + List rodNames = new ArrayList(); + rodNames.add(variantCollection.variants.getName()); + + Map vcfRods = GATKVCFUtils.getVCFHeadersFromRods(getToolkit(), rodNames); + Set vcfSamples = SampleUtils.getSampleList(vcfRods, GATKVariantContextUtils.GenotypeMergeType.REQUIRE_UNIQUE); + + // set up tumor and normal samples + if ( !vcfSamples.contains(normalSample) ) + throw new UserException.BadArgumentValue("--normalSample", "the normal sample " + normalSample + " doesn't match any sample from the input VCF"); + if ( !vcfSamples.contains(tumorSample) ) + throw new UserException.BadArgumentValue("--tumorSample", "the tumor sample " + tumorSample + " doesn't match any sample from the input VCF"); + + logger.info("Normal sample: " + normalSample); + logger.info("Tumor sample: " + tumorSample); + + Set headerLines = new HashSet(); + headerLines.addAll(GATKVCFUtils.getHeaderFields(this.getToolkit())); + headerLines.add(new VCFInfoHeaderLine(VCFConstants.SOMATIC_KEY, 0, VCFHeaderLineType.Flag, "Is this a confidently called somatic mutation")); + headerLines.add(new VCFInfoHeaderLine(SOMATIC_LOD_TAG_NAME, 1, VCFHeaderLineType.Float, "log10 probability that the site is a somatic mutation")); + headerLines.add(new VCFInfoHeaderLine(SOMATIC_AC_TAG_NAME, 1, VCFHeaderLineType.Integer, "Allele count of samples with somatic event")); + headerLines.add(new VCFInfoHeaderLine(SOMATIC_NONREF_TAG_NAME, 1, VCFHeaderLineType.Integer, "Number of samples with somatic event")); + + samples.add(normalSample); + samples.add(tumorSample); + vcfWriter.writeHeader(new VCFHeader(headerLines, samples)); + } + + private double log10pNonRefInSamples(final VariantContext vc, final String sample) { + return log10PLFromSamples(vc, sample, false); + } + + private double log10pRefInSamples(final VariantContext vc, final String sample) { + return log10PLFromSamples(vc, sample, true); + } + + private double log10PLFromSamples(final VariantContext vc, final String sample, boolean calcRefP) { + + Genotype g = vc.getGenotype(sample); + double log10pSample = -1000; + if ( ! g.isNoCall() ) { + final double[] gLikelihoods = MathUtils.normalizeFromLog10(g.getLikelihoods().getAsVector()); + log10pSample = Math.log10(calcRefP ? gLikelihoods[0] : 1 - gLikelihoods[0]); + log10pSample = Double.isInfinite(log10pSample) ? -10000 : log10pSample; + } + return log10pSample; + } + + private int calculateTumorAC(final VariantContext vc) { + int ac = 0; + switch ( vc.getGenotype(tumorSample).getType() ) { + case HET: ac += 1; break; + case HOM_VAR: ac += 2; break; + case NO_CALL: case UNAVAILABLE: case HOM_REF: break; + } + return ac; + } + + private int calculateTumorNNR(final VariantContext vc) { + int nnr = 0; + switch ( vc.getGenotype(tumorSample).getType() ) { + case HET: case HOM_VAR: nnr += 1; break; + case NO_CALL: case UNAVAILABLE: case HOM_REF: break; + } + return nnr; + } + + /** + * P(somatic | D) + * = P(somatic) * P(D | somatic) + * = P(somatic) * P(D | normals are ref) * P(D | tumors are non-ref) + * + * P(! somatic | D) + * = P(! somatic) * P(D | ! somatic) + * = P(! somatic) * + * * ( P(D | normals are non-ref) * P(D | tumors are non-ref) [germline] + * + P(D | normals are ref) * P(D | tumors are ref)) [no-variant at all] + * + * @param vc + * @return + */ + private double calcLog10pSomatic(final VariantContext vc) { + // walk over tumors + double log10pNonRefInTumors = log10pNonRefInSamples(vc, tumorSample); + double log10pRefInTumors = log10pRefInSamples(vc, tumorSample); + + // walk over normals + double log10pNonRefInNormals = log10pNonRefInSamples(vc, normalSample); + double log10pRefInNormals = log10pRefInSamples(vc, normalSample); + + // priors + double log10pSomaticPrior = QualityUtils.qualToErrorProbLog10(somaticPriorQ); + double log10pNotSomaticPrior = Math.log10(1 - QualityUtils.qualToErrorProb(somaticPriorQ)); + + double log10pNotSomaticGermline = log10pNonRefInNormals + log10pNonRefInTumors; + double log10pNotSomaticNoVariant = log10pRefInNormals + log10pRefInTumors; + + double log10pNotSomatic = log10pNotSomaticPrior + MathUtils.log10sumLog10(new double[]{log10pNotSomaticGermline, log10pNotSomaticNoVariant}); + double log10pSomatic = log10pSomaticPrior + log10pNonRefInTumors + log10pRefInNormals; + double lod = log10pSomatic - log10pNotSomatic; + + return Double.isInfinite(lod) ? -10000 : lod; + } + + /** + * For each variant in the file, determine the phasing for the child and replace the child's genotype with the trio's genotype + * + * @param tracker the reference meta-data tracker + * @param ref the reference context + * @param context the alignment context + * @return null + */ + @Override + public Integer map(RefMetaDataTracker tracker, ReferenceContext ref, AlignmentContext context) { + if (tracker != null) { + for ( VariantContext vc : tracker.getValues(variantCollection.variants, context.getLocation()) ) { + vc = vc.subContextFromSamples(samples); + if ( !vc.isPolymorphicInSamples() ) + continue; + + double log10pSomatic = calcLog10pSomatic(vc); + + // write in the somatic status probability + Map attrs = new HashMap(); // vc.getAttributes()); + if ( ! minimalVCF ) attrs.putAll(vc.getAttributes()); + attrs.put(SOMATIC_LOD_TAG_NAME, log10pSomatic); + if ( log10pSomatic > somaticMinLOD ) { + attrs.put(VCFConstants.SOMATIC_KEY, true); + attrs.put(SOMATIC_NONREF_TAG_NAME, calculateTumorNNR(vc)); + attrs.put(SOMATIC_AC_TAG_NAME, calculateTumorAC(vc)); + + } + final VariantContextBuilder builder = new VariantContextBuilder(vc).attributes(attrs); + VariantContextUtils.calculateChromosomeCounts(builder, false); + VariantContext newvc = builder.make(); + + vcfWriter.add(newvc); + } + + return null; + } + + return null; + } + + /** + * Provide an initial value for reduce computations. + * + * @return Initial value of reduce. + */ + @Override + public Integer reduceInit() { + return null; + } + + /** + * Reduces a single map with the accumulator provided as the ReduceType. + * + * @param value result of the map. + * @param sum accumulator for the reduce. + * @return accumulator with result of the map taken into account. + */ + @Override + public Integer reduce(Integer value, Integer sum) { + return null; + } + + @Override + public Integer treeReduce(Integer sum1, Integer sum2) { + return reduce(sum1, sum2); + } +} diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/BaseQualitySumPerAlleleBySample.java b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/BaseQualitySumPerAlleleBySample.java new file mode 100644 index 000000000..6e2ce6bc0 --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/BaseQualitySumPerAlleleBySample.java @@ -0,0 +1,190 @@ +/* +* By downloading the PROGRAM you agree to the following terms of use: +* +* BROAD INSTITUTE +* SOFTWARE LICENSE AGREEMENT +* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY +* +* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”). +* +* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and +* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions. +* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows: +* +* 1. DEFINITIONS +* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE. +* +* 2. LICENSE +* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute. LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation. +* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement. +* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement. +* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM. +* +* 3. PHONE-HOME FEATURE +* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM. Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time. Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation. +* +* 4. OWNERSHIP OF INTELLECTUAL PROPERTY +* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication. +* Copyright 2012-2015 Broad Institute, Inc. +* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc. +* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes. +* +* 5. INDEMNIFICATION +* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement. +* +* 6. NO REPRESENTATIONS OR WARRANTIES +* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME. +* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING. +* +* 7. ASSIGNMENT +* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void. +* +* 8. MISCELLANEOUS +* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries. +* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes. +* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4. +* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt. +* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter. +* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement. +* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles. +*/ + +package org.broadinstitute.gatk.tools.walkers.cancer; + +import htsjdk.variant.variantcontext.Allele; +import htsjdk.variant.variantcontext.Genotype; +import htsjdk.variant.variantcontext.GenotypeBuilder; +import htsjdk.variant.variantcontext.VariantContext; +import htsjdk.variant.vcf.VCFFormatHeaderLine; +import htsjdk.variant.vcf.VCFHeaderLineCount; +import htsjdk.variant.vcf.VCFHeaderLineType; +import org.apache.log4j.Logger; +import org.broadinstitute.gatk.tools.walkers.annotator.interfaces.AnnotatorCompatible; +import org.broadinstitute.gatk.tools.walkers.annotator.interfaces.GenotypeAnnotation; +import org.broadinstitute.gatk.tools.walkers.annotator.interfaces.StandardSomaticAnnotation; +import org.broadinstitute.gatk.tools.walkers.cancer.m2.MuTect2; +import org.broadinstitute.gatk.utils.QualityUtils; +import org.broadinstitute.gatk.utils.contexts.AlignmentContext; +import org.broadinstitute.gatk.utils.contexts.ReferenceContext; +import org.broadinstitute.gatk.utils.exceptions.GATKException; +import org.broadinstitute.gatk.utils.genotyper.MostLikelyAllele; +import org.broadinstitute.gatk.utils.genotyper.PerReadAlleleLikelihoodMap; +import org.broadinstitute.gatk.utils.refdata.RefMetaDataTracker; +import org.broadinstitute.gatk.utils.sam.GATKSAMRecord; +import org.broadinstitute.gatk.utils.sam.ReadUtils; +import org.broadinstitute.gatk.utils.variant.GATKVCFConstants; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Map; + + +/** + * Sum of evidence in reads supporting each allele for each sample + * + *

In the domain of somatic variants, a variant call can be supported by a few high quality reads. The + * BaseQualitySumPerAlleleBySample annotation aims to give the user an estimate of the quality of the evidence supporting + * a variant.

+ * + *

Notes

+ * BaseQualitySumPerAlleleBySample is called and used by MuTect2 for variant filtering. This annotation is applied to SNPs + * and INDELs. Qualities are not literal base qualities, but instead are derived from the per-allele likelihoods derived + * from the assembly engine. + * + *

Caveats

+ *
    + *
  • At this time, BaseQualitySumPerAlleleBySample can only be called from MuTect2
    • + *
+ */ +public class BaseQualitySumPerAlleleBySample extends GenotypeAnnotation implements StandardSomaticAnnotation { + private final static Logger logger = Logger.getLogger(BaseQualitySumPerAlleleBySample.class); + private boolean walkerIdentityCheckWarningLogged = false; + + public List getKeyNames() { return Arrays.asList(GATKVCFConstants.QUALITY_SCORE_SUM_KEY); } + + + public void annotate(final RefMetaDataTracker tracker, + final AnnotatorCompatible walker, + final ReferenceContext ref, + final AlignmentContext stratifiedContext, + final VariantContext vc, + final Genotype g, + final GenotypeBuilder gb, + final PerReadAlleleLikelihoodMap alleleLikelihoodMap) { + + // Can only call from MuTect2 + if ( !(walker instanceof MuTect2) ) { + if ( !walkerIdentityCheckWarningLogged ) { + if ( walker != null ) + logger.warn("Annotation will not be calculated, can only be called from MuTect2, not " + walker.getClass().getName()); + else + logger.warn("Annotation will not be calculated, can only be called from MuTect2"); + walkerIdentityCheckWarningLogged = true; + } + return; + } + + if ( g == null || !g.isCalled() || ( stratifiedContext == null && alleleLikelihoodMap == null) ) + return; + + if (alleleLikelihoodMap != null) { + annotateWithLikelihoods(alleleLikelihoodMap, vc, gb); + } + } + + protected void annotateWithLikelihoods(final PerReadAlleleLikelihoodMap perReadAlleleLikelihoodMap, final VariantContext vc, final GenotypeBuilder gb) { + final ArrayList refQuals = new ArrayList<>(); + final ArrayList altQuals = new ArrayList<>(); + + // clean up + fillQualsFromLikelihoodMap(vc.getAlleles(), vc.getStart(), perReadAlleleLikelihoodMap, refQuals, altQuals); + double refQualSum = 0; + for(Double d : refQuals) { refQualSum += d; } + + double altQualSum = 0; + for(Double d : altQuals) { altQualSum += d; } + + gb.attribute(GATKVCFConstants.QUALITY_SCORE_SUM_KEY, new Integer[]{ (int) refQualSum, (int) altQualSum}); + } + + public List getDescriptions() { + return Arrays.asList(new VCFFormatHeaderLine(getKeyNames().get(0), VCFHeaderLineCount.A, VCFHeaderLineType.Integer, "Sum of base quality scores for each allele")); + } + + // from rank sum test */ + protected void fillQualsFromLikelihoodMap(final List alleles, + final int refLoc, + final PerReadAlleleLikelihoodMap likelihoodMap, + final List refQuals, + final List altQuals) { + for ( final Map.Entry> el : likelihoodMap.getLikelihoodReadMap().entrySet() ) { + final MostLikelyAllele a = PerReadAlleleLikelihoodMap.getMostLikelyAllele(el.getValue()); + if ( ! a.isInformative() ) + continue; // read is non-informative + + final GATKSAMRecord read = el.getKey(); + if ( isUsableRead(read) ) { + final Double value = getBaseQualityForRead(read, refLoc); + if ( value == null ) + continue; + + if ( a.getMostLikelyAllele().isReference() ) + refQuals.add(value); + else if ( alleles.contains(a.getMostLikelyAllele()) ) + altQuals.add(value); + } + } + } + + protected boolean isUsableRead(final GATKSAMRecord read) { + return !( read.getMappingQuality() == 0 || + read.getMappingQuality() == QualityUtils.MAPPING_QUALITY_UNAVAILABLE ); + } + + + protected Double getBaseQualityForRead(final GATKSAMRecord read, final int refLoc) { + return (double)read.getBaseQualities()[ReadUtils.getReadCoordinateForReferenceCoordinateUpToEndOfRead(read, refLoc, ReadUtils.ClippingTail.RIGHT_TAIL)]; + } + +} \ No newline at end of file diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/OxoGReadCounts.java b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/OxoGReadCounts.java new file mode 100644 index 000000000..fabfdb167 --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/OxoGReadCounts.java @@ -0,0 +1,197 @@ +/* +* By downloading the PROGRAM you agree to the following terms of use: +* +* BROAD INSTITUTE +* SOFTWARE LICENSE AGREEMENT +* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY +* +* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”). +* +* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and +* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions. +* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows: +* +* 1. DEFINITIONS +* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE. +* +* 2. LICENSE +* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute. LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation. +* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement. +* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement. +* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM. +* +* 3. PHONE-HOME FEATURE +* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM. Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time. Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation. +* +* 4. OWNERSHIP OF INTELLECTUAL PROPERTY +* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication. +* Copyright 2012-2015 Broad Institute, Inc. +* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc. +* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes. +* +* 5. INDEMNIFICATION +* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement. +* +* 6. NO REPRESENTATIONS OR WARRANTIES +* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME. +* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING. +* +* 7. ASSIGNMENT +* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void. +* +* 8. MISCELLANEOUS +* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries. +* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes. +* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4. +* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt. +* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter. +* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement. +* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles. +*/ + +package org.broadinstitute.gatk.tools.walkers.cancer; + +import htsjdk.variant.variantcontext.Allele; +import htsjdk.variant.variantcontext.Genotype; +import htsjdk.variant.variantcontext.GenotypeBuilder; +import htsjdk.variant.variantcontext.VariantContext; +import htsjdk.variant.vcf.VCFFormatHeaderLine; +import htsjdk.variant.vcf.VCFHeaderLineCount; +import htsjdk.variant.vcf.VCFHeaderLineType; +import org.apache.log4j.Logger; +import org.broadinstitute.gatk.tools.walkers.annotator.interfaces.AnnotatorCompatible; +import org.broadinstitute.gatk.tools.walkers.annotator.interfaces.GenotypeAnnotation; +import org.broadinstitute.gatk.tools.walkers.annotator.interfaces.StandardAnnotation; +import org.broadinstitute.gatk.tools.walkers.cancer.m2.MuTect2; +import org.broadinstitute.gatk.utils.QualityUtils; +import org.broadinstitute.gatk.utils.contexts.AlignmentContext; +import org.broadinstitute.gatk.utils.contexts.ReferenceContext; +import org.broadinstitute.gatk.utils.exceptions.GATKException; +import org.broadinstitute.gatk.utils.genotyper.MostLikelyAllele; +import org.broadinstitute.gatk.utils.genotyper.PerReadAlleleLikelihoodMap; +import org.broadinstitute.gatk.utils.refdata.RefMetaDataTracker; +import org.broadinstitute.gatk.utils.sam.GATKSAMRecord; +import org.broadinstitute.gatk.utils.sam.ReadUtils; +import org.broadinstitute.gatk.utils.variant.GATKVCFConstants; +import org.broadinstitute.gatk.utils.variant.GATKVCFHeaderLines; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Map; + + +/** + * Count of read pairs in the F1R2 and F2R1 configurations supporting the reference and alternate alleles + * + *

This is an annotation that gathers information about the read pair configuration for the reads supporting each + * allele. It can be used along with downstream filtering steps to identify and filter out erroneous variants that occur + * with higher frequency in one read pair orientation.

+ * + *

References

+ *

For more details about the mechanism of oxoG artifact generation, see + * "Discovery and characterization of artefactual mutations in deep coverage targeted capture sequencing data due to oxidative DNA damage during sample preparation." + * by Costello et al.

+ * + *

Caveats

+ *
    + *
  • At present, this annotation can only be called from MuTect2
    • + *
  • The FOXOG annotation is only calculated for SNPs
    • + *
+ */ +public class OxoGReadCounts extends GenotypeAnnotation { + private final static Logger logger = Logger.getLogger(OxoGReadCounts.class); + private boolean walkerIdentityCheckWarningLogged = false; + Allele refAllele; + Allele altAllele; + + public List getKeyNames() { + return Arrays.asList(GATKVCFConstants.OXOG_ALT_F1R2_KEY, GATKVCFConstants.OXOG_ALT_F2R1_KEY, GATKVCFConstants.OXOG_REF_F1R2_KEY, GATKVCFConstants.OXOG_REF_F2R1_KEY, GATKVCFConstants.OXOG_FRACTION_KEY); + } + + + public void annotate(final RefMetaDataTracker tracker, + final AnnotatorCompatible walker, + final ReferenceContext ref, + final AlignmentContext stratifiedContext, + final VariantContext vc, + final Genotype g, + final GenotypeBuilder gb, + final PerReadAlleleLikelihoodMap alleleLikelihoodMap) { + + // Can only call from MuTect2 + if ( !(walker instanceof MuTect2) ) { + if ( !walkerIdentityCheckWarningLogged ) { + if ( walker != null ) + logger.warn("Annotation will not be calculated, can only be called from MuTect2, not " + walker.getClass().getName()); + else + logger.warn("Annotation will not be calculated, can only be called from MuTect2"); + walkerIdentityCheckWarningLogged = true; + } + return; + } + + if (g == null || !g.isCalled() || (stratifiedContext == null && alleleLikelihoodMap == null)) + return; + + refAllele = vc.getReference(); + altAllele = vc.getAlternateAllele(0); + + if (alleleLikelihoodMap != null) { + annotateWithLikelihoods(alleleLikelihoodMap, vc, gb); + } + } + + protected void annotateWithLikelihoods(final PerReadAlleleLikelihoodMap perReadAlleleLikelihoodMap, final VariantContext vc, final GenotypeBuilder gb) { + int ALT_F1R2, ALT_F2R1, REF_F1R2, REF_F2R1; + ALT_F1R2 = ALT_F2R1 = REF_F1R2 = REF_F2R1 = 0; + double numerator, denominator; + + for ( final Map.Entry> el : perReadAlleleLikelihoodMap.getLikelihoodReadMap().entrySet() ) { + final MostLikelyAllele a = PerReadAlleleLikelihoodMap.getMostLikelyAllele(el.getValue()); + if ( ! a.isInformative() || ! isUsableRead(el.getKey())) + continue; // read is non-informative or MQ0 + if (a.getAlleleIfInformative().equals(refAllele, true) && el.getKey().getReadPairedFlag()) { + if (el.getKey().getReadNegativeStrandFlag() == el.getKey().getFirstOfPairFlag()) + REF_F2R1++; + else + REF_F1R2++; + } + else if (a.getAlleleIfInformative().equals(altAllele,true) && el.getKey().getReadPairedFlag()){ + if (el.getKey().getReadNegativeStrandFlag() == el.getKey().getFirstOfPairFlag()) + ALT_F2R1++; + else + ALT_F1R2++; + } + } + + denominator = ALT_F1R2 + ALT_F2R1; + Double fOxoG = null; + if (vc.isSNP() && denominator > 0) { + if (refAllele.equals(Allele.create((byte) 'C', true)) || refAllele.equals(Allele.create((byte) 'A', true))) + numerator = ALT_F2R1; + else + numerator = ALT_F1R2; + fOxoG = (float) numerator / denominator; + } + + gb.attribute(GATKVCFConstants.OXOG_ALT_F1R2_KEY, new Integer(ALT_F1R2)); + gb.attribute(GATKVCFConstants.OXOG_ALT_F2R1_KEY, new Integer(ALT_F2R1)); + gb.attribute(GATKVCFConstants.OXOG_REF_F1R2_KEY, new Integer(REF_F1R2)); + gb.attribute(GATKVCFConstants.OXOG_REF_F2R1_KEY, new Integer(REF_F2R1)); + gb.attribute(GATKVCFConstants.OXOG_FRACTION_KEY, fOxoG); + } + + public List getDescriptions() { + return Arrays.asList(GATKVCFHeaderLines.getFormatLine(GATKVCFConstants.OXOG_ALT_F1R2_KEY), + GATKVCFHeaderLines.getFormatLine(GATKVCFConstants.OXOG_ALT_F2R1_KEY), + GATKVCFHeaderLines.getFormatLine(GATKVCFConstants.OXOG_REF_F1R2_KEY), + GATKVCFHeaderLines.getFormatLine(GATKVCFConstants.OXOG_REF_F2R1_KEY), + GATKVCFHeaderLines.getFormatLine(GATKVCFConstants.OXOG_FRACTION_KEY)); + } + + protected boolean isUsableRead(final GATKSAMRecord read) { + return !( read.getMappingQuality() == 0 || + read.getMappingQuality() == QualityUtils.MAPPING_QUALITY_UNAVAILABLE ); + } +} \ No newline at end of file diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/AnnotatePopulationAFWalker.java b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/AnnotatePopulationAFWalker.java new file mode 100755 index 000000000..23811a463 --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/AnnotatePopulationAFWalker.java @@ -0,0 +1,186 @@ +/* +* By downloading the PROGRAM you agree to the following terms of use: +* +* BROAD INSTITUTE +* SOFTWARE LICENSE AGREEMENT +* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY +* +* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”). +* +* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and +* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions. +* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows: +* +* 1. DEFINITIONS +* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE. +* +* 2. LICENSE +* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute. LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation. +* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement. +* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement. +* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM. +* +* 3. PHONE-HOME FEATURE +* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM. Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time. Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation. +* +* 4. OWNERSHIP OF INTELLECTUAL PROPERTY +* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication. +* Copyright 2012-2015 Broad Institute, Inc. +* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc. +* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes. +* +* 5. INDEMNIFICATION +* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement. +* +* 6. NO REPRESENTATIONS OR WARRANTIES +* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME. +* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING. +* +* 7. ASSIGNMENT +* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void. +* +* 8. MISCELLANEOUS +* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries. +* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes. +* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4. +* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt. +* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter. +* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement. +* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles. +*/ + +package org.broadinstitute.gatk.tools.walkers.cancer.contamination; + + +import org.broadinstitute.gatk.utils.commandline.Argument; +import org.broadinstitute.gatk.utils.commandline.Input; +import org.broadinstitute.gatk.utils.commandline.Output; +import org.broadinstitute.gatk.utils.commandline.RodBinding; +import org.broadinstitute.gatk.utils.contexts.AlignmentContext; +import org.broadinstitute.gatk.utils.contexts.ReferenceContext; +import org.broadinstitute.gatk.utils.refdata.RefMetaDataTracker; +import org.broadinstitute.gatk.engine.samples.Sample; +import org.broadinstitute.gatk.engine.walkers.DataSource; +import org.broadinstitute.gatk.engine.walkers.Requires; +import org.broadinstitute.gatk.engine.walkers.RodWalker; +import org.broadinstitute.gatk.engine.walkers.TreeReducible; +import htsjdk.variant.vcf.VCFHeader; +import htsjdk.variant.vcf.VCFHeaderLine; +import org.broadinstitute.gatk.utils.exceptions.UserException; +import htsjdk.variant.variantcontext.Allele; +import htsjdk.variant.variantcontext.Genotype; +import htsjdk.variant.variantcontext.VariantContext; +import htsjdk.variant.variantcontext.VariantContextBuilder; +import htsjdk.variant.variantcontext.writer.VariantContextWriter; + +import java.util.*; + +/** + * Given a input VCF representing a collection of populations, split the input into each population, and annotate each record with population allele frequencies + */ +// @Requires(DataSource.SAMPLE) <- require the sample data when this works +public class AnnotatePopulationAFWalker extends RodWalker implements TreeReducible { + // control the output + @Output(doc="File to which variants should be written",required=true) + protected VariantContextWriter writer = null; + + // our mapping of population to sample list + private final Map> popMapping = new LinkedHashMap>(); + + @Input(fullName="population", shortName = "pop", doc="the VCF containing large populations of samples", required=true) + public RodBinding pop; + + // either load the lanes into our name list, or the samples, depending on the command line parameters + public void initialize() { + // get the sample information + for (Sample sp: getToolkit().getSampleDB().getSamples()) + if (sp.getOtherPhenotype() != null) { + if (!popMapping.containsKey(sp.getOtherPhenotype())) + popMapping.put(sp.getOtherPhenotype(),new ArrayList()); + popMapping.get(sp.getOtherPhenotype()).add(sp); + } + + // this is a stop-gap until the @Requires tag is working with sample information + if (popMapping.size() == 0) + throw new UserException.BadInput("we require a sample file that contains population information. Please see the wiki about how to supply one"); + + // setup our VCF + // TODO: add code to get the samples from the input VCF, if they set 'preserveGenotypes' above + Set hInfo = new HashSet(); + + VCFHeader vcfHeader = new VCFHeader(hInfo); + writer.writeHeader(vcfHeader); + } + + + // boilerplate code - the standard reduce function for integers + @Override public Integer reduceInit() { return 0; } + @Override public Integer reduce(Integer value, Integer sum) { return(value + sum); } + public Integer treeReduce(Integer lhs, Integer rhs) { return lhs + rhs; } + + @Override + public Integer map(RefMetaDataTracker tracker, ReferenceContext ref, AlignmentContext context) { + if (tracker == null) return 0; + + // get the variant contexts, and return if we have anything other than one record at this site + Collection vcs = tracker.getValues(pop); + if (vcs.size() != 1) return 0; + VariantContext originalVC = vcs.iterator().next(); + + if (!originalVC.isSNP()) return 0; + VariantContext vc = originalVC; + + // get the list of alleles + List vcAlleles = vc.getAlleles(); + // setup the mapping of allele to population map + Map> popToAlleleFreq = new LinkedHashMap>(); + + // initialize all pops + Map allPopAC = new LinkedHashMap(); + int allPopTotal = 0; + for (Allele a : vcAlleles) allPopAC.put(a,0); + + // find the sub-population allele frequencies, and annotate them + for (Map.Entry> pop : popMapping.entrySet()) { + Map thisPopAC = new LinkedHashMap(); + int total = 0; + for (Allele a : vcAlleles) thisPopAC.put(a,0); + for (Sample s : pop.getValue()) { + Genotype g = vc.getGenotype(s.getID()); + if (g == null) continue; + for (Allele a : vcAlleles) { + int count = a.length(); + + total += count; + thisPopAC.put(a,thisPopAC.get(a) + count); + + allPopTotal += count; + allPopAC.put(a, allPopAC.get(a) + count); + } + } + Map thisPopAF = new LinkedHashMap(); + for (Map.Entry entry : thisPopAC.entrySet()) + thisPopAF.put(entry.getKey(),String.format("%1.5f", (total == 0) ? 0 : (double)entry.getValue()/(double)total)); + popToAlleleFreq.put(pop.getKey(),thisPopAF); + } + + // add the all pops value as well + Map allPopAF = new LinkedHashMap(); + for (Map.Entry entry : allPopAC.entrySet()) + allPopAF.put(entry.getKey(), String.format("%1.5f", (allPopTotal == 0) ? 0 : (double)entry.getValue()/(double)allPopTotal)); + + popToAlleleFreq.put("ALL", allPopAF); + + // add the population af annotations + VariantContextBuilder vcb = new VariantContextBuilder(vc); + Map popToAlleleFreqAsObject = new LinkedHashMap(); + for (Map.Entry> mp : popToAlleleFreq.entrySet()) { + popToAlleleFreqAsObject.put(mp.getKey(),(Object)mp.getValue()); + } + vcb.attributes(popToAlleleFreqAsObject); + writer.add(vc); + return 1; + } + + +} diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContEst.java b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContEst.java new file mode 100755 index 000000000..75e7c9fde --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContEst.java @@ -0,0 +1,729 @@ +/* +* By downloading the PROGRAM you agree to the following terms of use: +* +* BROAD INSTITUTE +* SOFTWARE LICENSE AGREEMENT +* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY +* +* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”). +* +* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and +* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions. +* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows: +* +* 1. DEFINITIONS +* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE. +* +* 2. LICENSE +* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute. LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation. +* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement. +* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement. +* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM. +* +* 3. PHONE-HOME FEATURE +* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM. Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time. Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation. +* +* 4. OWNERSHIP OF INTELLECTUAL PROPERTY +* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication. +* Copyright 2012-2015 Broad Institute, Inc. +* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc. +* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes. +* +* 5. INDEMNIFICATION +* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement. +* +* 6. NO REPRESENTATIONS OR WARRANTIES +* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME. +* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING. +* +* 7. ASSIGNMENT +* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void. +* +* 8. MISCELLANEOUS +* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries. +* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes. +* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4. +* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt. +* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter. +* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement. +* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles. +*/ + +package org.broadinstitute.gatk.tools.walkers.cancer.contamination; + +import htsjdk.samtools.SAMReadGroupRecord; +import htsjdk.samtools.util.StringUtil; +import org.broadinstitute.gatk.engine.CommandLineGATK; +import org.broadinstitute.gatk.engine.walkers.*; +import org.broadinstitute.gatk.tools.walkers.genotyper.afcalc.AFCalculatorProvider; +import org.broadinstitute.gatk.tools.walkers.genotyper.afcalc.FixedAFCalculatorProvider; +import org.broadinstitute.gatk.utils.commandline.*; +import org.broadinstitute.gatk.engine.GenomeAnalysisEngine; +import org.broadinstitute.gatk.utils.contexts.AlignmentContext; +import org.broadinstitute.gatk.utils.contexts.ReferenceContext; +import org.broadinstitute.gatk.utils.help.DocumentedGATKFeature; +import org.broadinstitute.gatk.utils.help.HelpConstants; +import org.broadinstitute.gatk.utils.sam.SAMReaderID; +import org.broadinstitute.gatk.utils.refdata.RefMetaDataTracker; +import org.broadinstitute.gatk.tools.walkers.genotyper.UnifiedArgumentCollection; +import org.broadinstitute.gatk.tools.walkers.genotyper.UnifiedGenotypingEngine; +import org.broadinstitute.gatk.tools.walkers.genotyper.VariantCallContext; +import org.broadinstitute.gatk.utils.GenomeLoc; +import org.broadinstitute.gatk.utils.exceptions.GATKException; +import org.broadinstitute.gatk.utils.exceptions.UserException; +import org.broadinstitute.gatk.utils.pileup.ReadBackedPileup; +import htsjdk.variant.variantcontext.*; + +import java.io.*; +import java.util.*; + +/** + * Estimate cross-sample contamination + * + * This tool determine the percent contamination of an input bam by sample, by lane, or in aggregate across all the input reads. + * + *

Usage examples

+ *

These are example commands that show how to run ContEst for typical use cases. Square brackets ("[ ]") + * indicate optional arguments. Note that parameter values and/or resources shown here may not be the latest recommended; see the Best Practices documentation for detailed recommendations.

+ * + *
+ *

Contamination estimation using a VCF containing the normal sample's genotypes (as might be derived from a genotyping array)

+ * 
+ *   java
+ *     -jar GenomeAnalysisTK.jar \
+ *     -T ContEst \
+ *     -R reference.fasta \
+ *     -I tumor.bam \
+ *     --genotypes normalGenotypes.vcf \
+ *     --popFile populationAlleleFrequencies.vcf \
+ *     -L populationSites.interval_list
+ *     [-L targets.interval_list] \
+ *     -isr INTERSECTION \
+ *     -o output.txt
+ *
+ * + *
+ *

Contamination estimation using the normal BAM for genotyping on-the-fly

+ * 
+ *   java
+ *     -jar GenomeAnalysisTK.jar \
+ *     -T ContEst \
+ *     -R reference.fasta \
+ *     -I:eval tumor.bam \
+ *     -I:genotype normal.bam \
+ *     --popFile populationAlleleFrequencies.vcf \
+ *     -L populationSites.interval_list
+ *     [-L targets.interval_list] \
+ *     -isr INTERSECTION \
+ *     -o output.txt
+ *
+ * + *

Output

+ * A text file containing estimated percent contamination, as well as error bars on this estimate. + * + *

Notes

+ * Multiple modes are supported simultaneously, e.g. contamination by sample and readgroup can be computed in the same run. + */ +@DocumentedGATKFeature( groupName = HelpConstants.DOCS_CAT_QC, extraDocs = {CommandLineGATK.class} ) +@Allows(value = {DataSource.READS, DataSource.REFERENCE}) +@Requires(value = {DataSource.READS, DataSource.REFERENCE}, referenceMetaData = @RMD(name = "genotypes", type = VariantContext.class)) +@By(DataSource.READS) +public class ContEst extends RodWalker>, ContaminationResults> { + + // ------------------------------------------------------------------------------------------------------------------------------------------------------ + // Some constants we use + // ------------------------------------------------------------------------------------------------------------------------------------------------------ + /** what type of run stats would we like: */ + public enum ContaminationRunType { + SAMPLE, // calculate contamination for each sample + READGROUP, // for each read group + META // for all inputs as a single source + } + // ------------------------------------------------------------------------------------------------------------------------------------------------------ + // inputs + // ------------------------------------------------------------------------------------------------------------------------------------------------------ + + // the genotypes ROD; this contains information about the genotypes from our sample + @Input(fullName="genotypes", shortName = "genotypes", doc="the genotype information for our sample", required=false) + public RodBinding genotypes; + + // the population information; the allele frequencies for each position in known populations + @Input(fullName="popfile", shortName = "pf", doc="the variant file containing information about the population allele frequencies", required=true) + public RodBinding pop; + + // ------------------------------------------------------------------------------------------------------------------------------------------------------ + // outputs and args + // ------------------------------------------------------------------------------------------------------------------------------------------------------ + @Output + PrintStream out; // the general output of the tool + + @Argument(fullName = "min_qscore", required = false, doc = "threshold for minimum base quality score") + public int MIN_QSCORE = 20; + + @Argument(fullName = "min_mapq", required = false, doc = "threshold for minimum mapping quality score") + public int MIN_MAPQ = 20; + + @Argument(fullName = "trim_fraction", doc = "at most, what fraction of sites should be trimmed based on BETA_THRESHOLD", required = false) + public double TRIM_FRACTION = 0.01; + + @Argument(fullName = "beta_threshold", doc = "threshold for p(f>=0.5) to trim", required = false) + public double BETA_THRESHOLD = 0.95; + + @Argument(shortName = "llc", fullName = "lane_level_contamination", doc = "set to META (default), SAMPLE or READGROUP to produce per-bam, per-sample or per-lane estimates", required = false) + private Set laneStats = null; + + @Argument(shortName = "sn", fullName = "sample_name", doc = "The sample name; used to extract the correct genotypes from mutli-sample truth vcfs", required = false) + private String sampleName = "unknown"; + + @Argument(shortName = "pc", fullName = "precision", doc = "the degree of precision to which the contamination tool should estimate (e.g. the bin size)", required = false) + private double precision = 0.1; + + @Argument(shortName = "br", fullName = "base_report", doc = "Where to write a full report about the loci we processed", required = false) + public PrintStream baseReport = null; + + @Argument(shortName = "lf", fullName = "likelihood_file", doc = "write the likelihood values to the specified location", required = false) + public PrintStream likelihoodFile = null; + + @Argument(shortName = "vs", fullName = "verify_sample", doc = "should we verify that the sample name is in the genotypes file?", required = false) + public boolean verifySample = false; + + @Argument(shortName = "mbc", fullName = "minimum_base_count", doc = "what minimum number of bases do we need to see to call contamination in a lane / sample?", required = false) + public Integer minBaseCount = 500; + + @Argument(shortName = "population", fullName = "population", doc = "evaluate contamination for just a single contamination population", required = false) + public String population = "CEU"; + + @Argument(shortName = "gm", fullName = "genotype_mode", doc = "which approach should we take to getting the genotypes (only in array-free mode)", required = false) + public SeqGenotypeMode genotypeMode = SeqGenotypeMode.HARD_THRESHOLD; + + // ------------------------------------------------------------------------------------------------------------------------------------------------------ + // hidden arguments + // ------------------------------------------------------------------------------------------------------------------------------------------------------ + @Hidden + @Argument(fullName = "trim_interval", doc = "progressively trim from 0 to TRIM_FRACTION by this interval", required = false) + public double TRIM_INTERVAL = 0; + + @Hidden + @Argument(fullName = "min_site_depth", required = false, doc = "minimum depth at a site to consider in calculation") + public int MIN_SITE_DEPTH = 0; + + @Hidden + @Argument(fullName = "fixed_epsilon_qscore", required = false, doc = "use a constant epsilon (phred scale) for calculation") + public Byte FIXED_EPSILON = null; + + @Hidden + @Argument(fullName = "min_genotype_depth", required = false, doc = "what minimum depth is required to call a site in seq genotype mode") + public int MIN_GENOTYPE_DEPTH_FOR_SEQ = 50; + + @Hidden + @Argument(fullName = "min_genotype_ratio", required = false, doc = "the ratio of alt to other bases to call a site a hom non-ref variant") + public double MIN_GENOTYPE_RATIO = 0.80; + + @Hidden + @Argument(fullName = "min_genotype_llh", required = false, doc = "the min log likelihood for UG to call a genotype") + public double MIN_UG_LOG_LIKELIHOOD = 5; + // ------------------------------------------------------------------------------------------------------------------------------------------------------ + // global variables to the walker + // ------------------------------------------------------------------------------------------------------------------------------------------------------ + private static final Map alleles = new HashMap(); // the set of alleles we work with + private boolean verifiedSampleName = false; // have we yet verified the sample name? + private final Map contaminationNames = new LinkedHashMap(); // a list, containing the contamination names, be it read groups or bam file names + private static String[] ALL_POPULATIONS = new String[]{"ALL", "CHD", "LWK", "CHB", "CEU", "MXL", "GIH", "MKK", "TSI", "CLM", "GBR", "ASW", "YRI", "IBS", "FIN", "PUR", "JPT", "CHS"}; + private String[] populationsToEvaluate; + + // variables involved in the array-free mode + private boolean useSequencingGenotypes = false; // if false we're using the sequencing geneotypes; otherwise we require array genotypes + public static final String EVAL_BAM_TAG = "eval"; + public static final String GENOTYPE_BAM_TAG = "genotype"; + String evalSample = null; + String genotypeSample = null; + + + // counts for each of the possible combinations + int totalSites = 0; + int countPopulationSites = 0; + int countGenotypeNonHomVar = 0; + int countGenotypeHomVar = 0; + int countPassCoverage = 0; + int countResults = 0; + + public enum SeqGenotypeMode { HARD_THRESHOLD, UNIFIED_GENOTYPER } + // create our list of allele characters for conversion + static { + alleles.put(0,Allele.create((byte) 'A')); + alleles.put(1,Allele.create((byte) 'C')); + alleles.put(2,Allele.create((byte) 'G')); + alleles.put(3,Allele.create((byte) 'T')); + } + + // a bunch of setup to initialize the walker + public void initialize() { + // set the genotypes source - figure out what to do if we're not using arrays + if (genotypes == null || !genotypes.isBound()) { + logger.info("Running in sequencing mode"); + useSequencingGenotypes = true; + // if were not using arrays, we need to figure out what samples are what + for(SAMReaderID id : getToolkit().getReadsDataSource().getReaderIDs()) { + if (id.getTags().getPositionalTags().size() == 0) + throw new UserException.BadInput("BAMs must be tagged with " + GENOTYPE_BAM_TAG + " and " + EVAL_BAM_TAG + " when running in array-free mode. Please see the ContEst documentation for more details"); + + // now sort out what tags go with what bam + for (String tag : id.getTags().getPositionalTags()) { + if (GENOTYPE_BAM_TAG.equalsIgnoreCase(tag)) { + try { + if (getToolkit().getReadsDataSource().getHeader(id).getReadGroups().size() == 0) + throw new RuntimeException("No Read Groups found for Genotyping BAM -- Read Groups are Required in sequencing genotype mode!"); + genotypeSample = getToolkit().getReadsDataSource().getHeader(id).getReadGroups().get(0).getSample(); + } catch (NullPointerException npe) { + throw new UserException.BadInput("Unable to fetch read group from the bam files tagged with " + GENOTYPE_BAM_TAG); + } + } else if (EVAL_BAM_TAG.equalsIgnoreCase(tag)) { + try { + if (getToolkit().getReadsDataSource().getHeader(id).getReadGroups().size() == 0) + throw new RuntimeException("No Read Groups found for Genotyping BAM -- Read Groups are Required in sequencing genotype mode!"); + evalSample = getToolkit().getReadsDataSource().getHeader(id).getReadGroups().get(0).getSample(); + } catch (NullPointerException npe) { + throw new UserException.BadInput("Unable to fetch read group from the bam files tagged with " + EVAL_BAM_TAG); + } + } else { + throw new UserException.BadInput("Unable to process " + tag + " tag, it's not either of the two accepted values: " + GENOTYPE_BAM_TAG + " or " + EVAL_BAM_TAG); + } + } + } + if (evalSample == null || genotypeSample == null) + throw new UserException.BadInput("You must provide both a " + GENOTYPE_BAM_TAG + " tagged bam and a " + EVAL_BAM_TAG + " tagged bam file. Please see the ContEst documentation"); + + } else { + logger.info("Running in array mode"); + } + if (laneStats == null) { + laneStats = new HashSet(); + laneStats.add(ContaminationRunType.META); + } + + for (ContaminationRunType type : laneStats) { + if (type == ContaminationRunType.READGROUP) { + for (SAMReadGroupRecord name : getToolkit().getSAMFileHeader().getReadGroups()) + this.contaminationNames.put(name.getId(),ContaminationRunType.READGROUP); + } else if (type == ContaminationRunType.SAMPLE) { + for (SAMReadGroupRecord name : getToolkit().getSAMFileHeader().getReadGroups()) + this.contaminationNames.put(name.getSample(),ContaminationRunType.SAMPLE); + } else if (type == ContaminationRunType.META) + this.contaminationNames.put("META",ContaminationRunType.META); + else + throw new IllegalArgumentException("Unknown type name " + laneStats); + } + if (baseReport != null) + baseReport.println("lane\tchrom\tposition\trs_id\tref\tfreq_major_allele\tfreq_minor_allele\tgeli_gt\tmaf\tmajor_allele_counts\tminor_allele_counts\ta_counts\tc_counts\tg_counts\tt_counts"); + + this.populationsToEvaluate = (population == null || "EVERY".equals(population)) ? ALL_POPULATIONS : new String[]{population}; + + } + /** + * our map function, which emits a contamination stats for each of the subgroups (lanes, samples, etc) that we encounter + * + * @param tracker the reference meta data tracker, from which we get the array truth data + * @param ref the reference information at this position + * @param context the read context, where we get the alignment data + * @return a mapping of our subgroup name to contamination estimate + */ + @Override + public Map> map(RefMetaDataTracker tracker, ReferenceContext ref, AlignmentContext context) { + totalSites++; + if (tracker == null) return null; + if (context == null) return null; + + VariantContext popVC = tracker.getFirstValue(pop); + byte referenceBase = ref.getBase(); + if (popVC == null) return null; + countPopulationSites++; + Genotype genotype = getGenotype(tracker,context,ref,useSequencingGenotypes); + + // only use homozygous sites + if (genotype == null || !genotype.isHomVar()) { + countGenotypeNonHomVar++; + return null; + } else { + countGenotypeHomVar++; + } + + + // only use non-reference sites + byte myBase = genotype.getAllele(0).getBases()[0]; + + String rsNumber = ""; + + // our map of contamination results + Map> stats = new HashMap>(); + + // get the base pileup. This is only really required when we have both a genotyping and EVAL_BAM_TAG tagged bams + // becuase we only want contamination estimates drawn from the eval tagged bam + ReadBackedPileup defaultPile; + if (this.useSequencingGenotypes) + defaultPile = context.getBasePileup().getPileupForSample(evalSample); + else + defaultPile = context.getBasePileup(); + + // if we're by-lane, get those stats + for (Map.Entry namePair : contaminationNames.entrySet()) { + ReadBackedPileup pile; + if (namePair.getValue() == ContaminationRunType.READGROUP) + pile = defaultPile.getPileupForReadGroup(namePair.getKey()); + else if (namePair.getValue() == ContaminationRunType.META) + pile = defaultPile; + else if (namePair.getValue() == ContaminationRunType.SAMPLE) + pile = defaultPile.getPileupForSample(namePair.getKey()); + else + throw new IllegalStateException("Unknown state, contamination type = " + laneStats + " is unsupported"); + if (pile != null) { + + ReadBackedPileup filteredPile = + pile.getBaseAndMappingFilteredPileup(MIN_QSCORE, MIN_MAPQ); + + byte[] bases = filteredPile.getBases(); + + // restrict to sites that have greater than our required total depth + if (bases.length < MIN_SITE_DEPTH) { + continue; + } else { + countPassCoverage++; + } + + byte[] quals; + if (FIXED_EPSILON == null) { + quals = filteredPile.getQuals(); + } else { + quals = new byte[bases.length]; + Arrays.fill(quals, FIXED_EPSILON); + } + + Map results = + calcStats(referenceBase, + bases, + quals, + myBase, + rsNumber, + popVC, + baseReport, + context.getLocation(), + precision, + namePair.getKey(), + populationsToEvaluate); + + if (results.size() > 0) { + countResults++; + stats.put(namePair.getKey(), results); + } + } + } + // return our collected stats + return stats; + } + + /** + * get the genotype for the sample at the current position + * @param tracker the reference meta data (RODs) + * @param context the reads + * @param referenceContext the reference information + * @param useSeq are we using sequencing to get our genotypes + * @return a genotype call, which could be null + */ + private Genotype getGenotype(RefMetaDataTracker tracker, AlignmentContext context, ReferenceContext referenceContext, boolean useSeq) { + if (!useSeq) { + Genotype g = getGenotypeFromArray(tracker, this.genotypes,this.verifiedSampleName,this.verifySample,this.sampleName); + if (g != null) this.verifiedSampleName = true; + return g; + } else { + return getGenotypeFromSeq( + context, + referenceContext, + this.alleles, + this.genotypeMode, + this.MIN_GENOTYPE_RATIO, + this.MIN_GENOTYPE_DEPTH_FOR_SEQ, + this.MIN_UG_LOG_LIKELIHOOD, + this.genotypeSample, + this.sampleName, + this.getToolkit()); + } + } + + static Genotype getGenotypeFromSeq(AlignmentContext context, + ReferenceContext referenceContext, + Map alleles, + SeqGenotypeMode genotypeMode, + double minGenotypeRatio, + int minGenotypingDepth, + double minGenotypingLOD, + String genotypingSample, + String sampleName, + GenomeAnalysisEngine toolKit) { + ReadBackedPileup pileup = context.getBasePileup().getPileupForSample(genotypingSample); + if (pileup == null || pileup.isEmpty()) return null; + + // which genotyping mode are we using + if (genotypeMode == SeqGenotypeMode.HARD_THRESHOLD) { + if (sum(pileup.getBaseCounts()) < minGenotypingDepth) return null; + int[] bases = pileup.getBaseCounts(); + int mx = maxPos(bases); + int allGenotypes = sum(bases); + String refBase = String.valueOf((char)referenceContext.getBase()); + if (bases[mx] / (float)allGenotypes >= minGenotypeRatio && !refBase.equals(alleles.get(mx).getBaseString())) { + List al = new ArrayList(); + al.add(alleles.get(mx)); + GenotypeBuilder builder = new GenotypeBuilder(sampleName, al); + return builder.make(); + } + } else if (genotypeMode == SeqGenotypeMode.UNIFIED_GENOTYPER) { + UnifiedArgumentCollection basicUAC = new UnifiedArgumentCollection(); + UnifiedGenotypingEngine engine = new UnifiedGenotypingEngine(basicUAC, FixedAFCalculatorProvider.createThreadSafeProvider(toolKit, basicUAC, logger),toolKit); + AlignmentContext contextSubset = new AlignmentContext(context.getLocation(),pileup,0,true); + List callContexts = engine.calculateLikelihoodsAndGenotypes(null, referenceContext, contextSubset); + if (callContexts != null && callContexts.size() == 1) + for (Genotype g : callContexts.get(0).getGenotypes()){ + if (g.isCalled() && g.isHomVar() && g.getLog10PError() > minGenotypingLOD) + return g; + } + } + else { + throw new GATKException("Unknown genotyping mode, being an enum this really shouldn't be seen ever."); + } + return null; + } + + // utils + private static int sum(int[] a) {int sm = 0; for (int i : a) {sm = sm + i;} return sm;} + private static int maxPos(int[] a) {int mx = 0; for (int i = 0;i < a.length; i++) {if (a[i] > a[mx]) mx = i;} return mx;} + + private static Genotype getGenotypeFromArray(RefMetaDataTracker tracker, RodBinding genotypes, boolean verifiedSampleName, boolean verifySample, String sampleName) { + // get the truthForSample and the hapmap information for this site; if either are null we can't move forward + Collection truths = tracker.getValues(genotypes); + if (truths == null || truths.size() == 0) return null; + + VariantContext truthForSample = truths.iterator().next(); + + // verify that the sample name exists in the input genotype file + if (!verifiedSampleName && verifySample) { + if (!truthForSample.getSampleNames().contains(sampleName)) + throw new UserException.BadInput("The sample name was set to " + sampleName + " but this sample isn't in your genotypes file. Please Verify your sample name"); + verifiedSampleName = true; + } + + GenotypesContext gt = truthForSample.getGenotypes(); + + // if we are supposed to verify the sample name, AND the sample doesn't exist in the genotypes -- skip this site + if (verifySample && !gt.containsSample(sampleName)) return null; + + // if the sample doesn't exist in genotypes AND there is more than one sample in the genotypes file -- skip this site + if (!gt.containsSample(sampleName) && gt.size() != 1) return null; + + // if there is more than one sample in the genotypes file, get it by name. Otherwise just get the sole sample genotype + return gt.size() != 1 ? gt.get(sampleName) : gt.get(0); + } + + + private static class PopulationFrequencyInfo { + private byte majorAllele; + private byte minorAllele; + private double minorAlleleFrequency; + + private PopulationFrequencyInfo(byte majorAllele, byte minorAllele, double minorAlleleFrequency) { + this.majorAllele = majorAllele; + this.minorAllele = minorAllele; + this.minorAlleleFrequency = minorAlleleFrequency; + } + + public byte getMajorAllele() { + return majorAllele; + } + + public byte getMinorAllele() { + return minorAllele; + } + + public double getMinorAlleleFrequency() { + return minorAlleleFrequency; + } + } + + private static PopulationFrequencyInfo parsePopulationFrequencyInfo(VariantContext variantContext, String population) { + PopulationFrequencyInfo info = null; + + List values = (List) variantContext.getAttribute(population); + + if (values != null) { + byte majorAllele = 0; + byte minorAllele = 0; + double maf = -1; + + for (String str : values) { + // strip off the curly braces and trim whitespace + if (str.startsWith("{")) str = str.substring(1, str.length()); + if (str.contains("}")) str = str.substring(0, str.indexOf("}")); + str = str.trim(); + String spl[] = str.split("="); + + byte allele = (byte) spl[0].trim().charAt(0); + double af = Double.valueOf(spl[1].trim()); + + if (af <= 0.5 && minorAllele == 0) { + minorAllele = allele; + maf = af; + } else { + majorAllele = allele; + } + + } + + info = new PopulationFrequencyInfo(majorAllele, minorAllele, maf); + } + return info; + } + + + /** + * Calculate the contamination values per division, be it lane, meta, sample, etc + * @param referenceBase the reference base + * @param bases the bases seen + * @param quals and the bases qual values + * @param myAllele the allele we have (our hom var genotype allele) + * @param rsNumber the dbsnp number if available + * @param popVC the population variant context from hapmap + * @param baseReport if we're writing a base report, write it here + * @param loc our location + * @param precision the percision we're aiming for + * @param lane the lane name information + * @param pops our pops to run over + * @return a mapping of each target population to their estimated contamination + */ + private static Map calcStats(byte referenceBase, + byte[] bases, + byte[] quals, + byte myAllele, + String rsNumber, + VariantContext popVC, + PrintStream baseReport, + GenomeLoc loc, + Double precision, + String lane, + String[] pops) { + int[] alts = new int[4]; + int total = 0; + // get the depth ratio we are aiming for + for (byte base : bases) { + if (base == 'A' || base == 'a') alts[0]++; + if (base == 'C' || base == 'c') alts[1]++; + if (base == 'G' || base == 'g') alts[2]++; + if (base == 'T' || base == 't') alts[3]++; + total++; + } + + Map ret = new HashMap(); + + for (String pop : pops) { + PopulationFrequencyInfo info = parsePopulationFrequencyInfo(popVC, pop); + double alleleFreq = info.getMinorAlleleFrequency(); + if (alleleFreq > 0.5) { + throw new RuntimeException("Minor allele frequency is greater than 0.5, this is an error; we saw AF of " + alleleFreq); + } + + int majorCounts = alts[getBaseIndex(info.getMajorAllele())]; + int minorCounts = alts[getBaseIndex(info.getMinorAllele())]; + int otherCounts = total - majorCounts - minorCounts; + + + // only use sites where this is the minor allele + if (myAllele == info.minorAllele) { + + if (pops.length == 1) { + if (baseReport != null) { + baseReport.print( + StringUtil.join("\t", + lane, + loc.getContig(), + "" + loc.getStart(), + rsNumber, + "" + (char) referenceBase, + "" + (char) info.getMajorAllele(), + "" + (char) info.getMinorAllele(), + "" + (char) info.getMinorAllele() + "" + (char) info.getMinorAllele(), + String.format("%1.4f", alleleFreq), + "" + majorCounts, + "" + minorCounts)); + + for (long cnt : alts) + baseReport.print("\t" + cnt); + baseReport.println(); + } + } + + ContaminationEstimate est = new ContaminationEstimate(precision, alleleFreq, bases, quals, info.getMinorAllele(), info.getMajorAllele(), pop, loc); + ret.put(pop, new ContaminationStats(loc, 1, alleleFreq, minorCounts, majorCounts, otherCounts, alts, est)); + + } + + } + return ret; + } + + private static int getBaseIndex(byte base) { + if (base == 'A' || base == 'a') return 0; + if (base == 'C' || base == 'c') return 1; + if (base == 'G' || base == 'g') return 2; + if (base == 'T' || base == 't') return 3; + return -1; + } + + // create a ContaminationResults to store the run information + @Override + public ContaminationResults reduceInit() { + return new ContaminationResults(precision); + } + + + @Override + public ContaminationResults reduce(Map> value, ContaminationResults sum) { + if (value != null) + sum.add(value); + return sum; + } + + /** + * on traversal done, output all the stats to the appropriate files + * + * @param result the results of our contamination estimate + */ + public void onTraversalDone(ContaminationResults result) { + + // filter out lanes / samples that don't have the minBaseCount + Map> cleanedMap = new HashMap>(); + for (Map.Entry> entry : result.getStats().entrySet()) { + + Map newMap = new HashMap(); + + Map statMap = entry.getValue(); + for (String popKey : statMap.keySet()) { + ContaminationStats stat = statMap.get(popKey); + if (stat.getBasesMatching() + stat.getBasesMismatching() >= minBaseCount) newMap.put(popKey, stat); + } + + + if (newMap.size() > 0) + cleanedMap.put(entry.getKey(), newMap); + else + out.println("Warning: We're throwing out lane " + entry.getKey() + " since it has fewer than " + minBaseCount + + " read bases at genotyped positions"); + } + + // output results at the end, based on the input parameters + result.setStats(cleanedMap); + result.outputReport(precision, out, TRIM_FRACTION, TRIM_INTERVAL, BETA_THRESHOLD); + if (likelihoodFile != null) result.writeCurves(likelihoodFile); + logger.info("Total sites: " + totalSites); + logger.info("Population informed sites: " + countPopulationSites); + logger.info("Non homozygous variant sites: " + countGenotypeNonHomVar); + logger.info("Homozygous variant sites: " + countGenotypeHomVar); + logger.info("Passed coverage: " + countPassCoverage); + logger.info("Results: " + countResults); + } +} diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContaminationEstimate.java b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContaminationEstimate.java new file mode 100755 index 000000000..40b70f619 --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContaminationEstimate.java @@ -0,0 +1,234 @@ +/* +* By downloading the PROGRAM you agree to the following terms of use: +* +* BROAD INSTITUTE +* SOFTWARE LICENSE AGREEMENT +* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY +* +* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”). +* +* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and +* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions. +* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows: +* +* 1. DEFINITIONS +* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE. +* +* 2. LICENSE +* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute. LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation. +* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement. +* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement. +* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM. +* +* 3. PHONE-HOME FEATURE +* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM. Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time. Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation. +* +* 4. OWNERSHIP OF INTELLECTUAL PROPERTY +* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication. +* Copyright 2012-2015 Broad Institute, Inc. +* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc. +* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes. +* +* 5. INDEMNIFICATION +* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement. +* +* 6. NO REPRESENTATIONS OR WARRANTIES +* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME. +* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING. +* +* 7. ASSIGNMENT +* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void. +* +* 8. MISCELLANEOUS +* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries. +* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes. +* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4. +* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt. +* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter. +* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement. +* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles. +*/ + +package org.broadinstitute.gatk.tools.walkers.cancer.contamination; + + +import org.broadinstitute.gatk.utils.GenomeLoc; +import org.broadinstitute.gatk.utils.collections.Pair; + +import java.util.Arrays; + +/** + * a class that estimates and stores the contamination values for a site. + */ +class ContaminationEstimate { + private final double precision; // to what precision do we want to run; e.g. if set to 1, we run using 1% increments + private final double[] bins; // the bins representing the discrete contamination levels we're evaluating + private double populationFit = 0.0; + private String popultationName = ""; + + private static double[] precalculatedEpsilon; + + private int arrayAlleleObservations = 0; + private int alternateAlleleObservations = 0; + + // precalculate the 128 values of epsilon that are possible + static { + precalculatedEpsilon = new double[Byte.MAX_VALUE+1]; + + for(int i=0; i <= (int)Byte.MAX_VALUE; i++) { + precalculatedEpsilon[i] = Math.pow(10.0,-1.0*(((double)i)/10.0)); + } + } + + /** + * create the contamination estimate, given: + * @param precision the precision value, to what level are we calculating the contamination + */ + public ContaminationEstimate(double precision, + double maf, + byte[] bases, + byte[] quals, + byte arrayAllele, + byte hapmapAlt, + String popName, + GenomeLoc locus + ) { + // setup the bins to the correct precision + this.precision = precision; + bins = new double[(int)Math.ceil(100/precision)+1]; + if (maf == 0) maf = 0.00001; + + popultationName = popName; + + Arrays.fill(bins,0.0); // just to make sure we don't have any residual values + + // convert the quals + double[] realQuals = new double[quals.length]; + int qIndex = 0; + for (byte qual : quals) {realQuals[qIndex++] = Math.pow(10.0,-1.0*(qual/10.0));} + + // check our inputs + if (maf > 1.0 || maf < 0.0) throw new IllegalArgumentException("Invalid allele Freq: must be between 0 and 1 (inclusive), maf was " + maf + " for population " + popName); + + // calculate the contamination for each bin + int qualOffset = 0; + for (byte base : bases) { + + if (base == arrayAllele) { arrayAlleleObservations++; } + if (base == hapmapAlt) { alternateAlleleObservations++; } + double epsilon = precalculatedEpsilon[quals[qualOffset++]]; + + for (int index = 0; index < bins.length; index++) { + + + double contaminationRate = (1.0 - (double) index / (double) bins.length); + + if (base == arrayAllele) { + bins[index] += Math.log((1.0 - contaminationRate) * (1.0 - epsilon) + + contaminationRate * ((maf) * (1.0 - epsilon) + (1.0 - maf) * (epsilon/3.0))); + populationFit += Math.log(epsilon); + + } else if(hapmapAlt == base) { + bins[index] += Math.log((1.0 - contaminationRate) * (epsilon / 3.0) + + contaminationRate * ((maf) * (epsilon/3.0) + (1.0 - maf) * (1.0 - epsilon))); + + populationFit += Math.log(maf + epsilon); + } + } + } + } + + public double[] getBins() { + return bins; + } + + public void setPopulationFit(double populationFit) { + this.populationFit = populationFit; + } + + public double getPopulationFit() { + return populationFit; + } + + public String getPopultationName() { + return popultationName; + } + + public static class ConfidenceInterval { + + private double start; + private double stop; + private double contamination; + private double maxLikelihood; + double[] newBins; + + public ConfidenceInterval(double bins[], double intervalArea) { + // make a copy of the bins in non-log space + int maxIndex = 0; + for (int x = 0; x < bins.length; x++) if (bins[x] > bins[maxIndex]) maxIndex = x; + newBins = new double[bins.length]; + maxLikelihood = bins[maxIndex]; + + int index = 0; + double total = 0.0; + for (double d : bins) { + newBins[index] = Math.pow(10,(bins[index] - bins[maxIndex])); + total += newBins[index]; + index++; + } + + for (int x = 0; x < newBins.length; x++) { + newBins[x] = newBins[x] / total; + } + double areaUnderCurve = 0; + int leftIndex = maxIndex; + int rightIndex = maxIndex; + while (areaUnderCurve < 0.95) { + + // if the "left" bin is bigger, and can be moved, move it + if (newBins[leftIndex] >= newBins[rightIndex] && leftIndex > 0) { + leftIndex--; + } else { + // otherwise move the right bin if possible + if (rightIndex < bins.length - 1) { + rightIndex++; + } else { + // and if not move the left bin, or die + if (leftIndex > 0) { + leftIndex--; + } else { + throw new RuntimeException("Error trying to compute confidence interval"); + } + } + } + + areaUnderCurve = 0.0; + for (int x = leftIndex; x <= rightIndex; x++) + areaUnderCurve += newBins[x]; + } + start = (bins.length - rightIndex) * (100.0/bins.length); + stop = (bins.length - leftIndex) * (100.0/bins.length); + contamination = (bins.length - maxIndex) * (100.0/bins.length); + } + + public double getStart() { + return start; + } + + public double getStop() { + return stop; + } + + public double getContamination() { + return contamination; + } + + public double getMaxLikelihood() { + return maxLikelihood; + } + + public String toString() { + return contamination + "[" + start + " - " + stop + "] log likelihood = " + maxLikelihood; + } + } +} diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContaminationResults.java b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContaminationResults.java new file mode 100755 index 000000000..fa11db5fc --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContaminationResults.java @@ -0,0 +1,304 @@ +/* +* By downloading the PROGRAM you agree to the following terms of use: +* +* BROAD INSTITUTE +* SOFTWARE LICENSE AGREEMENT +* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY +* +* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”). +* +* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and +* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions. +* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows: +* +* 1. DEFINITIONS +* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE. +* +* 2. LICENSE +* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute. LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation. +* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement. +* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement. +* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM. +* +* 3. PHONE-HOME FEATURE +* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM. Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time. Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation. +* +* 4. OWNERSHIP OF INTELLECTUAL PROPERTY +* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication. +* Copyright 2012-2015 Broad Institute, Inc. +* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc. +* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes. +* +* 5. INDEMNIFICATION +* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement. +* +* 6. NO REPRESENTATIONS OR WARRANTIES +* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME. +* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING. +* +* 7. ASSIGNMENT +* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void. +* +* 8. MISCELLANEOUS +* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries. +* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes. +* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4. +* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt. +* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter. +* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement. +* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles. +*/ + +package org.broadinstitute.gatk.tools.walkers.cancer.contamination; + + +import org.apache.commons.math.MathException; +import org.apache.commons.math.distribution.BetaDistribution; +import org.apache.commons.math.distribution.BetaDistributionImpl; +import org.broadinstitute.gatk.utils.GenomeLoc; +import org.broadinstitute.gatk.utils.Utils; + +import java.io.*; +import java.util.*; + +/** + * our contamination results object; this object aggregates the results of the contamination run over lanes, samples, + * or whatever other divisor we've used on the read data + */ +public class ContaminationResults { + + public static class ContaminationData implements Comparable { + private GenomeLoc site; + private long basesMatching = 0l; + private long basesMismatching = 0l; + private double mismatchFraction = -1d; + private double[] bins; + private double p; + + public long getBasesMatching() { + return basesMatching; + } + + public long getBasesMismatching() { + return basesMismatching; + } + + public double getMismatchFraction() { + return mismatchFraction; + } + + public double[] getBins() { + return bins; + } + + public double getP() { + return p; + } + + public ContaminationData(GenomeLoc site, long basesMatching, long basesMismatching, double[] bins) { + this.site = site; + this.basesMatching = basesMatching; + this.basesMismatching = basesMismatching; + this.bins = bins; + long totalBases = this.basesMatching + this.basesMismatching; + if (totalBases != 0) { + this.mismatchFraction = (double)this.basesMismatching / (double) totalBases; + } + + int a = (int) this.getBasesMismatching() + 1; + int b = (int) this.getBasesMatching() + 1; + BetaDistribution dist = new BetaDistributionImpl(a,b); + try { + this.p = 1.0d - dist.cumulativeProbability(0.5d); + } catch (MathException me) { + throw new RuntimeException("Error! - " + me.toString()); + } + + } + + public int compareTo(ContaminationData other) { + return -Double.compare(this.getP(), other.getP()); + } + + @Override + public String toString() { + return "ContaminationData{" + + "site=" + site + + ", basesMatching=" + basesMatching + + ", basesMismatching=" + basesMismatching + + ", mismatchFraction=" + mismatchFraction + + '}'; + } + } + + + // what precision are we using in our calculations + private final double precision; + + // a map of our contamination targets and their stats + // key: aggregation entity ("META", sample name, or lane name) + // value: ContaminationStats (whcih + private Map> stats = new HashMap>(); + + public ContaminationResults(double precision) { + this.precision = precision; + } + + + Map>> storedData = new HashMap>>(); + + /** + * add to the stats + * + * @param newAggregationStats a mapping of the stat name to their statistics collected + */ + public void add(Map> newAggregationStats) { + + // for each aggregation level + for (String aggregationKey : newAggregationStats.keySet()) { + Map populationContaminationStats = newAggregationStats.get(aggregationKey); + + + // a new way of doing this... store all the data until the end... + if (!storedData.containsKey(aggregationKey)) { storedData.put(aggregationKey, new HashMap>()); } + for (String pop : populationContaminationStats.keySet()) { + ContaminationStats newStats = populationContaminationStats.get(pop); + + // if it exists... just merge it + if (!storedData.get(aggregationKey).containsKey(pop)) { + storedData.get(aggregationKey).put(pop, new ArrayList()); + } + + double[] newData = new double[newStats.getContamination().getBins().length]; + System.arraycopy(newStats.getContamination().getBins(),0,newData,0,newStats.getContamination().getBins().length); + storedData.get(aggregationKey).get(pop).add(new ContaminationData(newStats.getSite(), newStats.getBasesMatching(), newStats.getBasesMismatching(), newData)); + } + + + + // merge the sets + if (stats.containsKey(aggregationKey)) { + + // and for each population + for (String pop : populationContaminationStats.keySet()) { + ContaminationStats newStats = populationContaminationStats.get(pop); + + // if it exists... just merge it + if (stats.get(aggregationKey).containsKey(pop)) { + stats.get(aggregationKey).get(pop).add(newStats); + } else { + stats.get(aggregationKey).put(pop, newStats); + } + } + } else { + stats.put(aggregationKey, populationContaminationStats); + } + } + } + + /** + * output the contamination data, and return the contamination data + * @param out the output source + * @return the contamination value + */ + public void outputReport(double precision, PrintStream out, double fractionToTrim, double trimInterval, double betaThreshold) { + out.println("name\tpopulation\tpopulation_fit\tcontamination\tconfidence_interval_95_width\tconfidence_interval_95_low\tconfidence_interval_95_high\tsites"); + + for (Map.Entry> entry : stats.entrySet()) { + for (ContaminationStats stats : entry.getValue().values()) { + String aggregationLevel = entry.getKey(); + String population = stats.getContamination().getPopultationName(); + + List newStats = storedData.get(aggregationLevel).get(population); + String pm = "%3." + Math.round(Math.log10(1/precision)) +"f"; + + int bins = newStats.iterator().next().getBins().length; + int maxTrim = (int) Math.floor((double)(newStats.size()) * fractionToTrim); + + // sort the collection + Collections.sort(newStats); + + List data = new ArrayList(newStats); + + // trim sites with > 95% p of being > 0.5 f (based on beta distribution) + int trimmed = 0; + for(Iterator i = data.iterator(); trimmed < maxTrim && i.hasNext();) { + ContaminationData x = i.next(); + if (x.getP() >= betaThreshold) { + System.out.println("Trimming " + x.toString() + " with p(f>=0.5) >= " + betaThreshold + " with a value of " + x.getP()); + i.remove(); + trimmed++; + } + } + + double[][] matrix = new double[bins][data.size()]; + + for (int i = 0; i> entry : stats.entrySet()) { + for (ContaminationStats stats : entry.getValue().values()) { + if (!outputBins) { + String[] bins = new String[stats.getContamination().getBins().length]; + for (int index = 0; index < stats.getContamination().getBins().length; index++) + bins[index] = String.valueOf(100.0 * (1 - (double) index / stats.getContamination().getBins().length)); + outputBins = true; + out.print("name,pop,"); + out.println(Utils.join(",",bins)); + } + String[] bins = new String[stats.getContamination().getBins().length]; + int index = 0; + for (double value : stats.getContamination().getBins()) + bins[index++] = String.valueOf(value); + out.print(entry.getKey()+",\""+stats.getContamination().getPopultationName()+"\","); + out.println(Utils.join(",", bins)); + } + } + } + + public Map> getStats() { + return Collections.unmodifiableMap(stats); + } + + public void setStats(Map> stats) { + this.stats = stats; + } +} \ No newline at end of file diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContaminationStats.java b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContaminationStats.java new file mode 100755 index 000000000..7d60d3ca0 --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContaminationStats.java @@ -0,0 +1,125 @@ +/* +* By downloading the PROGRAM you agree to the following terms of use: +* +* BROAD INSTITUTE +* SOFTWARE LICENSE AGREEMENT +* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY +* +* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”). +* +* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and +* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions. +* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows: +* +* 1. DEFINITIONS +* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE. +* +* 2. LICENSE +* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute. LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation. +* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement. +* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement. +* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM. +* +* 3. PHONE-HOME FEATURE +* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM. Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time. Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation. +* +* 4. OWNERSHIP OF INTELLECTUAL PROPERTY +* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication. +* Copyright 2012-2015 Broad Institute, Inc. +* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc. +* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes. +* +* 5. INDEMNIFICATION +* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement. +* +* 6. NO REPRESENTATIONS OR WARRANTIES +* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME. +* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING. +* +* 7. ASSIGNMENT +* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void. +* +* 8. MISCELLANEOUS +* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries. +* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes. +* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4. +* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt. +* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter. +* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement. +* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles. +*/ + +package org.broadinstitute.gatk.tools.walkers.cancer.contamination; + + +import org.broadinstitute.gatk.utils.GenomeLoc; +import org.omg.PortableInterceptor.SYSTEM_EXCEPTION; + +/** + * a class that tracks our contamination stats; both the estimate of contamination, as well as the number of sites and other + * run-specific data + */ +public class ContaminationStats { + final static int ALLELE_COUNT = 4; + private GenomeLoc site; + private int numberOfSites = 0; + private double sumOfAlleleFrequency = 0.0; + private long basesFor = 0l; + private long basesAgainst = 0l; + private long basesOther = 0l; + private ContaminationEstimate contaminationEstimate; + private final int[] alleleBreakdown; + + public ContaminationStats(GenomeLoc site, int numberOfSites, double sumOfAlleleFrequency, long basesFor, long basesAgainst, long basesOther, int alleleBreakdown[], ContaminationEstimate estimate) { + this.site = site; + this.numberOfSites = numberOfSites; + this.sumOfAlleleFrequency = sumOfAlleleFrequency; + this.basesFor = basesFor; + this.basesAgainst = basesAgainst; + this.contaminationEstimate = estimate; + if (alleleBreakdown.length != ALLELE_COUNT) throw new IllegalArgumentException("Allele breakdown should have length " + ALLELE_COUNT); + this.alleleBreakdown = alleleBreakdown; + } + + public int getNumberOfSites() { + return numberOfSites; + } + + public double getMinorAlleleFrequency() { + return sumOfAlleleFrequency /(double)numberOfSites; + } + + public long getBasesMatching() { + return basesFor; + } + + public long getBasesOther() { + return basesOther; + } + + public long getBasesMismatching() { + return basesAgainst; + } + + public ContaminationEstimate getContamination() { + return this.contaminationEstimate; + } + + public GenomeLoc getSite() { + return site; + } + + public void add(ContaminationStats other) { + if (other == null) return; + this.numberOfSites += other.numberOfSites; + this.sumOfAlleleFrequency += other.sumOfAlleleFrequency; + this.basesOther += other.basesOther; + this.basesFor += other.basesFor; + this.basesAgainst += other.basesAgainst; + for (int x = 0; x < ALLELE_COUNT; x++) this.alleleBreakdown[x] += other.alleleBreakdown[x]; + for (int i = 0; i < this.contaminationEstimate.getBins().length; i++) { + this.contaminationEstimate.getBins()[i] += other.contaminationEstimate.getBins()[i]; + } + this.contaminationEstimate.setPopulationFit(this.contaminationEstimate.getPopulationFit() +other.contaminationEstimate.getPopulationFit()); + } +} \ No newline at end of file diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/Dream_Evaluations.md b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/Dream_Evaluations.md new file mode 100644 index 000000000..dd86aed24 --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/Dream_Evaluations.md @@ -0,0 +1,77 @@ +# Dream Challenge Evaluation + +In order to evaluate the performance of M2, we use two sets of data from the SMC DREAM Challenge. Specifically challenges #3 and #4. + +All scripts referenced here are relative to the current working directory of ``` +/dsde/working/mutect/dream_smc``` + +### Current Performance (Unmasked) +From the output of the evaluation method + +(gsa-unstable 7/13/15, commit:9e93a70) + +|set | subset | type | sensitivity | specificity | accuracy | +|----|--------|------|-------------|-------------|----------| +|SMC 3|chr21|SNP|0.935897435897|0.935897435897|0.935897435897| +|SMC 3|chr21|INDEL|0.904255319149|0.977011494253|0.940633406701| +|SMC 3|wgs|SNP|0.930532709098|0.955188985583|0.94286084734| +|SMC 3|wgs|INDEL|0.902139907396|0.970516962843|0.93632843512| +|SMC 4|chr21|SNP|0.769607843137|0.969135802469|0.869371822803| +|SMC 4|chr21|INDEL|0.771241830065|0.991596638655|0.88141923436| +|SMC 4|wgs|SNP|0.764507007622|0.975374480433|0.869940744028| +|SMC 4|wgs|INDEL|0.768634634353|0.989389679877|0.879012157115| + + + +### How To Run +The SCALA script for running M2 can be found in the gsa-unstable repository under ```private/gatk-tools-private/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2``` + +First, chose the appropriate settings (runnable as environment variables here) +``` +QUEUE_JAR= +OUT_VCF= +GSA_UNSTABLE_HOME= + +# for Dream 3 +NORMAL_BAM=/dsde/working/mutect/dream_smc/bams/synthetic.challenge.set3.normal.bam +TUMOR_BAM=/dsde/working/mutect/dream_smc/bams/synthetic.challenge.set3.tumor.bam + +# for Dream 4 +NORMAL_BAM=/dsde/working/mutect/dream_smc/bams/synthetic.challenge.set4.normal.bam +TUMOR_BAM=/dsde/working/mutect/dream_smc/bams/synthetic.challenge.set4.tumor.bam + +# for WGS +INTERVALS=/dsde/working/mutect/dream_smc/bams/wgs_calling_regions.v1.interval_list + +# for chromosome 21 only +INTERVALS=/dsde/working/mutect/ts/c21_wgs_calling_regions.v1.interval_list + +TEMPDIR=/broad/hptmp/kcibul/mutect +``` + +and then run the following Queue command +``` +java \ + -Djava.io.tmpdir=$TEMPDIR \ + -jar $QUEUE_JAR \ + -S $GSA_UNSTABLE_HOME/private/gatk-tools-private/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/run_M2_dream.scala \ + --job_queue gsa -qsub -jobResReq virtual_free=5G -startFromScratch \ + -sc 200 \ + -normal $NORMAL_BAM \ + -tumor $TUMOR_BAM \ + -L $INTERVALS \ + -o $OUT_VCF \ + -run +``` + +### How To Evaluate + +Run the following +``` +/dsde/working/mutect/dream_smc/dream_eval.pl [3|4] [wgs|21] [SNV|INDEL] input.vcf +``` +where + - [3|4] the dream challenge round + - [wgs|21] evaluate the whole genome, or just a subset (chromosome 21) + - [SNV|INDEL] evaulate SNV (SNPs) or INDELS + diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/M2ArgumentCollection.java b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/M2ArgumentCollection.java new file mode 100644 index 000000000..97ffcf3cf --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/M2ArgumentCollection.java @@ -0,0 +1,117 @@ +/* +* By downloading the PROGRAM you agree to the following terms of use: +* +* BROAD INSTITUTE +* SOFTWARE LICENSE AGREEMENT +* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY +* +* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”). +* +* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and +* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions. +* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows: +* +* 1. DEFINITIONS +* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE. +* +* 2. LICENSE +* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute. LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation. +* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement. +* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement. +* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM. +* +* 3. PHONE-HOME FEATURE +* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM. Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time. Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation. +* +* 4. OWNERSHIP OF INTELLECTUAL PROPERTY +* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication. +* Copyright 2012-2015 Broad Institute, Inc. +* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc. +* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes. +* +* 5. INDEMNIFICATION +* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement. +* +* 6. NO REPRESENTATIONS OR WARRANTIES +* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME. +* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING. +* +* 7. ASSIGNMENT +* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void. +* +* 8. MISCELLANEOUS +* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries. +* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes. +* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4. +* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt. +* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter. +* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement. +* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles. +*/ + +package org.broadinstitute.gatk.tools.walkers.cancer.m2; + +import org.broadinstitute.gatk.tools.walkers.haplotypecaller.AssemblyBasedCallerArgumentCollection; +import org.broadinstitute.gatk.utils.commandline.Advanced; +import org.broadinstitute.gatk.utils.commandline.Argument; + +public class M2ArgumentCollection extends AssemblyBasedCallerArgumentCollection { + @Advanced + @Argument(fullName="m2debug", shortName="m2debug", doc="Print out very verbose M2 debug information", required = false) + public boolean M2_DEBUG = false; + + /** + * Artifact detection mode is used to prepare a panel of normals. This maintains the specified tumor LOD threshold, + * but disables the remaining pragmatic filters. See M2 usage examples for more information. + */ + @Advanced + @Argument(fullName = "artifact_detection_mode", required = false, doc="Enable artifact detection for creating panels of normals") + public boolean ARTIFACT_DETECTION_MODE = false; + + /** + * This is the tumor LOD threshold to output the variant in the VCF, although it may be filtered + */ + @Argument(fullName = "initial_tumor_lod", required = false, doc = "Initial LOD threshold for calling tumor variant") + public double INITIAL_TUMOR_LOD_THRESHOLD = 4.0; + + + @Argument(fullName = "initial_normal_lod", required = false, doc = "Initial LOD threshold for calling normal variant") + public double INITIAL_NORMAL_LOD_THRESHOLD = 0.5; + + /** + * Only variants with tumor LODs exceeding this thresholds can pass filtration + */ + @Argument(fullName = "tumor_lod", required = false, doc = "LOD threshold for calling tumor variant") + public double TUMOR_LOD_THRESHOLD = 6.3; + + /** + * This is a measure of the minimum evidence to show that a variant observed in the tumor is not also present in its normal + */ + @Argument(fullName = "normal_lod", required = false, doc = "LOD threshold for calling normal non-germline") + public double NORMAL_LOD_THRESHOLD = 2.2; + + /** + * The LOD threshold for the normal is typically made more strict if the variant has been seen in dbSNP (i.e. another + * normal sample). We thus require MORE evidence that a variant is NOT seen in this tumor's normal if it has been observed as a germline variant before. + */ + @Argument(fullName = "dbsnp_normal_lod", required = false, doc = "LOD threshold for calling normal non-variant at dbsnp sites") + public double NORMAL_DBSNP_LOD_THRESHOLD = 5.5; + + /** + * This argument is used for the M2 internal "alt_allele_in_normal" filter + **/ + @Argument(fullName = "max_alt_alleles_in_normal_count", required = false, doc="Threshold for maximum alternate allele counts in normal") + public int MAX_ALT_ALLELES_IN_NORMAL_COUNT = 2; + + /** + * This argument is used for the M2 internal "alt_allele_in_normal" filter + */ + @Argument(fullName = "max_alt_alleles_in_normal_qscore_sum", required = false, doc="Threshold for maximum alternate allele quality score sum in normal") + public int MAX_ALT_ALLELES_IN_NORMAL_QSCORE_SUM = 20; + + /** + * This argument is used for the M2 internal "alt_allele_in_normal" filter + */ + @Argument(fullName = "max_alt_allele_in_normal_fraction", required = false, doc="Threshold for maximum alternate allele fraction in normal") + public double MAX_ALT_ALLELE_IN_NORMAL_FRACTION = 0.03; +} diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/M2_Contamination_Evaluation.md b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/M2_Contamination_Evaluation.md new file mode 100644 index 000000000..19324dae6 --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/M2_Contamination_Evaluation.md @@ -0,0 +1,157 @@ +# M2 Contamination Correction Evaluation + +In order to evaluate the efficacy of the contamination correction in M2 (adapted from HaplotypeCaller), we created synthetic data consisting of the four intra-run CRSP NA12878 replicates, each contaminated with 1-5% of the HCC-1143 normal. + +### Creating Artificial Contamination Data +``` QUEUE_JAR= + GSA_UNSTABLE_HOME= + BASELINE_BAM=/crsp/picard_aggregation/000007820918/SM-612V3/current/SM-612V3.bam + CONTAMINANT_BAM=/seq/tier3b/picard_aggregation/C970/HCC1143_BL/v1/HCC1143_BL.bam + ``` + +``` +java -jar $QUEUE_JAR \ +-S $GSA_UNSTABLE_HOME/private/gatk-queue-extensions-internal/src/main/qscripts/org/broadinstitute/gatk/queue/qscripts/dev/CreateSyntheticContaminationScript.scala \ +-b2 CONTAMINANT_BAM -b1 BASELINE_BAM \ +-f 0.01 -f 0.02 -f 0.03 -f 0.04 -f 0.05 +``` + +Repeat for the other three NA12878 replicates. + + BASELINE_BAM=/crsp/picard_aggregation/000007820818/SM-612V4/current/SM-612V4.bam + + BASELINE_BAM=/crsp/picard_aggregation/000007820718/SM-612V5/current/SM-612V5.bam + + BASELINE_BAM=/crsp/picard_aggregation/000007820618/SM-612V6/current/SM-612V6.bam + +Use ContEst to get the contamination estimate in the data to be passed into M2. (Note that for these data, the ContEst estimate is on the order of 1% higher than the value used to generate the contaminated data.) + + TEMPDIR=/broad/hptmp/$USER + BAM1=HCC1143_BL.small.0.04.contaminated.with.SM-612V3.small.0.96.bam + BAM2=/crsp/picard_aggregation/000007820818/SM-612V4/current/SM-612V4.bam + OUT_TXT=ContEst_0.04HCC1143inNA12878.txt + + java -Djava.io.tmpdir=$TEMPDIR \ + -Xmx512m -jar /xchip/tcga/gdac_prod/applications/process_mgmt/firehose_task_registry/cga/ContaminationAnalysis/broadinstitute.org/cancer.genome.analysis/00262/107//Queue-1.4-437-g6b8a9e1-svn-35362.jar \ + -S /xchip/tcga/gdac_prod/applications/process_mgmt/firehose_task_registry/cga/ContaminationAnalysis/broadinstitute.org/cancer.genome.analysis/00262/107//ContaminationPipeline.scala \ + -reference /seq/references/Homo_sapiens_assembly19/v1/Homo_sapiens_assembly19.fasta -interval /xchip/cga/reference/hg19/whole_exome_agilent_1.1_refseq_plus_3_boosters_plus_10bp_padding_minus_mito.Homo_sapiens_assembly19.targets.interval_list \ + -out $OUT_TXT \ + -bam $BAM1 -nbam $BAM2 \ + -array /xchip/gcc_data/results2/production/TEENY/BySample/hg19.vcf.txt/TEENY_p_TCGA_302_303_N_GenomeWideSNP_6_F04_1344608.hg19.vcf.txt.store/verstore.00000.TEENY_p_TCGA_302_303_N_GenomeWideSNP_6_F04_1344608.hg19.vcf.txt \ + -pop /xchip/cga/reference/hg19/hg19_population_stratified_af_hapmap_3.3.fixed.vcf -faf true -run -array_interval /xchip/cga/reference/hg19/SNP6.hg19.interval_list + + BAM1=HCC1143_BL.small.0.03.contaminated.with.SM-612V3.small.0.97.bam + BAM2=/crsp/picard_aggregation/000007820818/SM-612V4/current/SM-612V4.bam + OUT_TXT=ContEst_0.03HCC1143inNA12878.txt + + BAM1=HCC1143_BL.small.0.02.contaminated.with.SM-612V3.small.0.98.bam + BAM2=/crsp/picard_aggregation/000007820818/SM-612V4/current/SM-612V4.bam + OUT_TXT=ContEst_0.02HCC1143inNA12878.txt + + BAM1=HCC1143_BL.small.0.01.contaminated.with.SM-612V3.small.0.99.bam + BAM2=/crsp/picard_aggregation/000007820818/SM-612V4/current/SM-612V4.bam + OUT_TXT=ContEst_0.01HCC1143inNA12878.txt + + And so on for the other replicates. + + ContEst estimates for the four replicates at the five contamination levels are as follows: + |Sample|Input Contamination Level|ContEst Estimate| + |------|-------------------------|----------------| + |SM-612V3|0.01|0.016| + |SM-612V3|0.02|0.030| + |SM-612V3|0.03|0.042| + |SM-612V3|0.04|0.055| + |SM-612V3|0.05|0.067| + |SM-612V4|0.01|0.015| + |SM-612V4|0.02|0.028| + |SM-612V4|0.03|0.039| + |SM-612V4|0.04|0.051| + |SM-612V4|0.05|0.063| + |SM-612V5|0.01|0.016| + |SM-612V5|0.02|0.030| + |SM-612V5|0.03|0.042| + |SM-612V5|0.04|0.054| + |SM-612V5|0.05|0.066| + |SM-612V6|0.01|0.015| + |SM-612V6|0.02|0.027| + |SM-612V6|0.03|0.040| + |SM-612V6|0.04|0.051| + |SM-612V6|0.05|0.062| + +###Prepare the inputs for the normal-normal calling script +Create a list of all contamination levels for each replicate + + ls -1 HCC*contam*V3*.bam > HCC1143withNA12878_3.bams.list + ls -1 HCC*contam*V4*.bam > HCC1143withNA12878_4.bams.list + ls -1 HCC*contam*V5*.bam > HCC1143withNA12878_5.bams.list + ls -1 HCC*contam*V6*.bam > HCC1143withNA12878_6.bams.list + +Create a list of the other, uncontaminated normals to call against + + ls -1 /humgen/gsa-hpprojects/NA12878Collection/bams/crsp_ice_validation/SM-612V[^37D].bam > /dsde/working/mutect/laura/contamination/NA12878_not3.list + ls -1 /humgen/gsa-hpprojects/NA12878Collection/bams/crsp_ice_validation/SM-612V[^47D].bam > /dsde/working/mutect/laura/contamination/NA12878_not4.list + ls -1 /humgen/gsa-hpprojects/NA12878Collection/bams/crsp_ice_validation/SM-612V[^57D].bam > /dsde/working/mutect/laura/contamination/NA12878_not5.list + ls -1 /humgen/gsa-hpprojects/NA12878Collection/bams/crsp_ice_validation/SM-612V[^67D].bam > /dsde/working/mutect/laura/contamination/NA12878_not6.list + +###Run the Caller +Run M2 on contaminated bams versus other all other replicates. Run one loop for each contaminated replicate, passing in contamination estimates as given above + + java -jar $QUEUE_JAR -S /dsde/working/mutect/laura/contamination/Qscript_M2_normalNormalLoop.scala -normal /dsde/working/mutect/laura/contamination/NA12878_not3.list -tumor /dsde/working/mutect/laura/contamination/HCC1143withNA12878_3.bams.list -o M2_NA12878run3_ -f 0.016 -f 0.03 -f 0.042 -f 0.055 -f 0.067 + + java -jar $QUEUE_JAR -S /dsde/working/mutect/laura/contamination/Qscript_M2_normalNormalLoop.scala -normal /dsde/working/mutect/laura/contamination/NA12878_not4.list -tumor /dsde/working/mutect/laura/contamination/HCC1143withNA12878_4.bams.list -o M2_NA12878run4_ -f 0.015 -f 0.028 -f 0.039 -f 0.051 -f 0.063 + + java -jar $QUEUE_JAR -S /dsde/working/mutect/laura/contamination/Qscript_M2_normalNormalLoop.scala -normal /dsde/working/mutect/laura/contamination/NA12878_not5.list -tumor /dsde/working/mutect/laura/contamination/HCC1143withNA12878_5.bams.list -o M2_NA12878run5_ -f 0.016 -f 0.030 -f 0.042 -f 0.054 -f 0.066 + + java -jar $QUEUE_JAR -S /dsde/working/mutect/laura/contamination/Qscript_M2_normalNormalLoop.scala -normal /dsde/working/mutect/laura/contamination/NA12878_not6.list -tumor /dsde/working/mutect/laura/contamination/HCC1143withNA12878_6.bams.list -o M2_NA12878run6_ -f 0.015 -f 0.027 -f 0.040 -f 0.051 -f 0.062 + +###Count the False Positives +Pull out passing SNPs not in PON for each contamination level: + + for vcf in M2_NA12878run[0-9]_HCC1143_BL.small.0.01.contaminated.with.SM-612V*.bam.vcf + do + bedtools intersect -a $vcf -b ICE.corrected.bed | grep PASS | awk '{ if ( length($4) + length($5) == 2) print $0 }' | wc -l + done + + for vcf in M2_NA12878run[0-9]_HCC1143_BL.small.0.02.contaminated.with.SM-612V*.bam.vcf + do + bedtools intersect -a $vcf -b ICE.corrected.bed | grep PASS | awk '{ if ( length($4) + length($5) == 2) print $0 }' | wc -l + done + + for vcf in M2_NA12878run[0-9]_HCC1143_BL.small.0.03.contaminated.with.SM-612V*.bam.vcf + do + bedtools intersect -a $vcf -b ICE.corrected.bed | grep PASS | awk '{ if ( length($4) + length($5) == 2) print $0 }' | wc -l + done + + for vcf in M2_NA12878run[0-9]_HCC1143_BL.small.0.04.contaminated.with.SM-612V*.bam.vcf + do + bedtools intersect -a $vcf -b ICE.corrected.bed | grep PASS | awk '{ if ( length($4) + length($5) == 2) print $0 }' | wc -l + done + + for vcf in M2_NA12878run[0-9]_HCC1143_BL.small.0.05.contaminated.with.SM-612V*.bam.vcf + do + bedtools intersect -a $vcf -b ICE.corrected.bed | grep PASS | awk '{ if ( length($4) + length($5) == 2) print $0 }' | wc -l + done + +(I pasted the results from the terminal into Excel because it's just so easy.) + +###Comparison Without Downsampling +To run normal-normal contaminated calling without downsampling, the above /dsde/working/mutect/laura/contamination/Qscript_M2_normalNormalLoop.scala commands can be used, passing in -f 0 for each contamination level instead, e.g.: + + java -jar $QUEUE_JAR -S /dsde/working/mutect/laura/contamination/Qscript_M2_normalNormalLoop.scala -normal /dsde/working/mutect/laura/contamination/NA12878_not3.list -tumor /dsde/working/mutect/laura/contamination/HCC1143withNA12878_3.bams.list -o M2_NA12878run3_noContam_ -f 0.0 -f 0.0 -f 0.0 -f 0.0 -f 0.0 + +###Comparison to M1 +To run normal-normal contaminated calling using M1, run the above Queue commands using a Queue jar containing MuTect and passing in /dsde/working/mutect/laura/contamination/Qscript_M1_normalNormalLoop.scala instead of Qscript_M2_normalNormalLoop.scala, e.g.: + + java -jar $QUEUE_JAR_WITH_M1 -S /dsde/working/mutect/laura/contamination/Qscript_M1_normalNormalLoop.scala -normal /dsde/working/mutect/laura/contamination/NA12878_not3.list -tumor /dsde/working/mutect/laura/contamination/HCC1143withNA12878_3.bams.list -o M1_NA12878run3_ -f 0.016 -f 0.03 -f 0.042 -f 0.055 -f 0.067 + +(The MuTect-containing Queue jar can be built from the gsa-unstable branch ldg_MuTect1.) + +###Latest Results +|M2 SNPs no correction|M2 SNPs with correction|M1 SNPs no correction|M1 SNPs with correction|M2 INDELs no correction|M2 INDELs with correction| +|---------------------|-----------------------|---------------------|-----------------------|-----------------------|-------------------------| +|0%|93|93|181|181|25|25| +|1%|938|258|854|317|68|30| +|2%|2550|464|1941|385|92|21| +|3%|4171|596|3061|515|134|18| +|4%|5513|707|4002|589|162|21| +|5%|6475|794|4854|624|188|29| \ No newline at end of file diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/M2_Panel_Of_Normals_Creation.md b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/M2_Panel_Of_Normals_Creation.md new file mode 100644 index 000000000..72e475b4a --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/M2_Panel_Of_Normals_Creation.md @@ -0,0 +1,35 @@ +# M2 Panel of Normals + +In order to reduce false positives, we use a panel of "normal" (ie non-cancer) samples to filter out both germline events as well as systematic noise. The form of the panel is a VCF file, which is produced via a Queue scripts + +You must supply: +- the reference (defaults to hg19) +- the intervals to evaluate +- the list of BAM files + +### How To Run +The Queue script for producing the PON can be found in the gsa-unstable repository under ```private/gatk-tools-private/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2``` + +First, chose the appropriate settings (runnable as environment variables here) +``` +QUEUE_JAR= +GSA_UNSTABLE_HOME= +TEMPDIR=/broad/hptmp/$USER +``` + +and then run the following Queue command +``` +java \ + -Djava.io.tmpdir=$TEMPDIR \ + -jar $QUEUE_JAR \ + -S $GSA_UNSTABLE_HOME/private/gatk-tools-private/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/run_M2_ICE_NN.scala \ + --job_queue gsa -qsub -jobResReq virtual_free=5G -startFromScratch \ + -sc 50 \ + --allbams \ + --intervals \ + --outputprefix \ + --start_from_scratch --keep_intermediate_outputs \ + -run +``` + +This will produce many VCFs (1 per sample), plus \.genotypes.vcf and \.vcf which are the panel of normals VCF both with and without sample-genotype information. Typically the latter is the one used as input to M2, although either will work. \ No newline at end of file diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/MuTect2.java b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/MuTect2.java new file mode 100644 index 000000000..ec0038bab --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/MuTect2.java @@ -0,0 +1,1291 @@ +/* +* By downloading the PROGRAM you agree to the following terms of use: +* +* BROAD INSTITUTE +* SOFTWARE LICENSE AGREEMENT +* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY +* +* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”). +* +* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and +* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions. +* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows: +* +* 1. DEFINITIONS +* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE. +* +* 2. LICENSE +* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute. LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation. +* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement. +* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement. +* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM. +* +* 3. PHONE-HOME FEATURE +* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM. Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time. Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation. +* +* 4. OWNERSHIP OF INTELLECTUAL PROPERTY +* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication. +* Copyright 2012-2015 Broad Institute, Inc. +* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc. +* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes. +* +* 5. INDEMNIFICATION +* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement. +* +* 6. NO REPRESENTATIONS OR WARRANTIES +* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME. +* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING. +* +* 7. ASSIGNMENT +* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void. +* +* 8. MISCELLANEOUS +* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries. +* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes. +* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4. +* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt. +* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter. +* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement. +* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles. +*/ + +package org.broadinstitute.gatk.tools.walkers.cancer.m2; + +import htsjdk.samtools.SAMFileWriter; +import htsjdk.variant.variantcontext.*; +import htsjdk.variant.variantcontext.writer.VariantContextWriter; +import htsjdk.variant.vcf.*; +import org.broadinstitute.gatk.engine.CommandLineGATK; +import org.broadinstitute.gatk.engine.GenomeAnalysisEngine; +import org.broadinstitute.gatk.engine.arguments.DbsnpArgumentCollection; +import org.broadinstitute.gatk.engine.filters.BadMateFilter; +import org.broadinstitute.gatk.engine.io.DirectOutputTracker; +import org.broadinstitute.gatk.engine.io.stubs.SAMFileWriterStub; +import org.broadinstitute.gatk.engine.iterators.ReadTransformer; +import org.broadinstitute.gatk.engine.walkers.*; +import org.broadinstitute.gatk.tools.walkers.annotator.VariantAnnotatorEngine; +import org.broadinstitute.gatk.tools.walkers.annotator.interfaces.AnnotatorCompatible; +import org.broadinstitute.gatk.tools.walkers.genotyper.GenotypingOutputMode; +import org.broadinstitute.gatk.tools.walkers.genotyper.afcalc.FixedAFCalculatorProvider; +import org.broadinstitute.gatk.tools.walkers.haplotypecaller.*; +import org.broadinstitute.gatk.tools.walkers.haplotypecaller.readthreading.ReadThreadingAssembler; +import org.broadinstitute.gatk.utils.GenomeLoc; +import org.broadinstitute.gatk.utils.GenomeLocParser; +import org.broadinstitute.gatk.utils.QualityUtils; +import org.broadinstitute.gatk.utils.activeregion.ActiveRegion; +import org.broadinstitute.gatk.utils.activeregion.ActiveRegionReadState; +import org.broadinstitute.gatk.utils.activeregion.ActivityProfileState; +import org.broadinstitute.gatk.utils.clipping.ReadClipper; +import org.broadinstitute.gatk.utils.commandline.*; +import org.broadinstitute.gatk.utils.contexts.AlignmentContext; +import org.broadinstitute.gatk.utils.contexts.AlignmentContextUtils; +import org.broadinstitute.gatk.utils.contexts.ReferenceContext; +import org.broadinstitute.gatk.utils.downsampling.AlleleBiasedDownsamplingUtils; +import org.broadinstitute.gatk.utils.downsampling.DownsamplingUtils; +import org.broadinstitute.gatk.utils.exceptions.UserException; +import org.broadinstitute.gatk.utils.fasta.CachingIndexedFastaSequenceFile; +import org.broadinstitute.gatk.utils.fragments.FragmentCollection; +import org.broadinstitute.gatk.utils.fragments.FragmentUtils; +import org.broadinstitute.gatk.utils.genotyper.*; +import org.broadinstitute.gatk.utils.haplotype.Haplotype; +import org.broadinstitute.gatk.utils.haplotypeBAMWriter.HaplotypeBAMWriter; +import org.broadinstitute.gatk.utils.help.DocumentedGATKFeature; +import org.broadinstitute.gatk.utils.help.HelpConstants; +import org.broadinstitute.gatk.utils.pairhmm.PairHMM; +import org.broadinstitute.gatk.utils.pileup.PileupElement; +import org.broadinstitute.gatk.utils.pileup.ReadBackedPileup; +import org.broadinstitute.gatk.utils.refdata.RefMetaDataTracker; +import org.broadinstitute.gatk.utils.sam.*; +import org.broadinstitute.gatk.utils.variant.GATKVCFConstants; +import org.broadinstitute.gatk.utils.variant.GATKVCFHeaderLines; + +import java.io.FileNotFoundException; +import java.util.*; + +import static java.lang.Math.pow; + +/** + * Call somatic SNPs and indels via local re-assembly of haplotypes + * + *

MuTect2 is a somatic SNP and indel caller that combines the DREAM challenge-winning somatic genotyping engine of the original MuTect (Cibulskis et al., 2013) with the assembly-based machinery of HaplotypeCaller.

+ * + *

The basic operation of MuTect2 proceeds similarly to that of the HaplotypeCaller

+ * + *

Differences from HaplotypeCaller

+ *

While the HaplotypeCaller relies on a ploidy assumption (diploid by default) to inform its genotype likelihood and + * variant quality calculations, MuTect2 allows for a varying allelic fraction for each variant, as is often seen in tumors with purity less + * than 100%, multiple subclones, and/or copy number variation (either local or aneuploidy). MuTect2 also differs from the HaplotypeCaller in that it does apply some hard filters + * to variants before producing output.

+ * + *

Usage examples

+ *

These are example commands that show how to run MuTect2 for typical use cases. Square brackets ("[ ]") + * indicate optional arguments. Note that parameter values shown here may not be the latest recommended; see the + * Best Practices documentation for detailed recommendations.

+ * + *
+ *

Tumor/Normal variant calling

+ * 
+ *   java
+ *     -jar GenomeAnalysisTK.jar \
+ *     -T MuTect2 \
+ *     -R reference.fasta \
+ *     -I:tumor tumor.bam \
+ *     -I:normal normal.bam \
+ *     [--dbsnp dbSNP.vcf] \
+ *     [--cosmic COSMIC.vcf] \
+ *     [-L targets.interval_list] \
+ *     -o output.vcf
+ *
+ * + *

Normal-only calling for panel of normals creation

+ * 
+ *   java
+ *     -jar GenomeAnalysisTK.jar
+ *     -T HaplotypeCaller
+ *     -R reference.fasta
+ *     -I:tumor normal1.bam \
+ *     [--dbsnp dbSNP.vcf] \
+ *     [--cosmic COSMIC.vcf] \
+ *     --artifact_detection_mode \
+ *     [-L targets.interval_list] \
+ *     -o output.normal1.vcf
+ *
+ *
+ * For full PON creation, call each of your normals separately in artifact detection mode. Then use CombineVariants to + * output only sites where a variant was seen in at least two samples: + * 
+ * java -jar GenomeAnalysisTK.jar
+ *     -T CombineVariants
+ *     -R reference.fasta
+ *     -V output.normal1.vcf -V output.normal2.vcf [-V output.normal2.vcf ...] \
+ *     -minN 2 \
+ *     --setKey "null" \
+ *     --filteredAreUncalled \
+ *     --filteredrecordsmergetype KEEP_IF_ANY_UNFILTERED \
+ *     [-L targets.interval_list] \
+ *     -o MuTect2_PON.vcf
+ *
+ * + *

Caveats

+ *
    + *
  • MuTect2 currently only supports the calling of a single tumor-normal pair at a time
    • + *
+ * + */ +@DocumentedGATKFeature( groupName = HelpConstants.DOCS_CAT_VARDISC, extraDocs = {CommandLineGATK.class} ) +@PartitionBy(PartitionType.LOCUS) +@BAQMode(ApplicationTime = ReadTransformer.ApplicationTime.FORBIDDEN) +@ActiveRegionTraversalParameters(extension=100, maxRegion=300) +public class MuTect2 extends ActiveRegionWalker, Integer> implements AnnotatorCompatible, NanoSchedulable { + public static final String BAM_TAG_TUMOR = "tumor"; + public static final String BAM_TAG_NORMAL = "normal"; + + protected Set tumorSAMReaderIDs = new HashSet<>(); + protected Set normalSAMReaderIDs = new HashSet<>(); + protected String tumorSampleName; + protected String normalSampleName; + + protected SampleList samplesList; + protected boolean printTCGAsampleHeader = false; + + // fasta reference reader to supplement the edges of the reference sequence + protected CachingIndexedFastaSequenceFile referenceReader; + + // the assembly engine + protected LocalAssemblyEngine assemblyEngine = null; + + // the likelihoods engine + protected ReadLikelihoodCalculationEngine likelihoodCalculationEngine = null; + + // the genotyping engine + protected HaplotypeCallerGenotypingEngine genotypingEngine = null; + + + private byte MIN_TAIL_QUALITY; + private double log10GlobalReadMismappingRate; + + + + @ArgumentCollection + protected M2ArgumentCollection MTAC = new M2ArgumentCollection(); + + @ArgumentCollection + protected ReadThreadingAssemblerArgumentCollection RTAC = new ReadThreadingAssemblerArgumentCollection(); + + @ArgumentCollection + protected LikelihoodEngineArgumentCollection LEAC = new LikelihoodEngineArgumentCollection(); + + + @Argument(fullName = "debug_read_name", required = false, doc="trace this read name through the calling process") + public String DEBUG_READ_NAME = null; + + @Hidden + @Advanced + @Argument(fullName = "MQ_filtering_level", shortName = "MQthreshold", required = false, doc="Set an alternate MQ threshold for debugging") + final public int MQthreshold = 20; + + + /***************************************/ + // Reference Metadata inputs + /***************************************/ + /** + * MuTect2 has the ability to use COSMIC data in conjunction with dbSNP to adjust the threshold for evidence of a variant + * in the normal. If a variant is present in dbSNP, but not in COSMIC, then more evidence is required from the normal + * sample to prove the variant is not present in germline. + */ + @Input(fullName="cosmic", shortName = "cosmic", doc="VCF file of COSMIC sites", required=false) + public List> cosmicRod = Collections.emptyList(); + + /** + * A panel of normals can be a useful (optional) input to help filter out commonly seen sequencing noise that may appear as low allele-fraction somatic variants. + */ + @Input(fullName="normal_panel", shortName = "PON", doc="VCF file of sites observed in normal", required=false) + public List> normalPanelRod = Collections.emptyList(); + + private HaplotypeBAMWriter haplotypeBAMWriter; + + @Override + public void initialize() { + super.initialize(); + + samplesList = new IndexedSampleList(new ArrayList<>(ReadUtils.getSAMFileSamples(getToolkit().getSAMFileHeader()))); + + // MUTECT: check that we have at least one tumor bam + for(SAMReaderID id : getToolkit().getReadsDataSource().getReaderIDs()) { + if (id.getTags().getPositionalTags().size() == 0) { + throw new RuntimeException("BAMs must be tagged as either 'tumor' or 'normal'"); + } + + // only supports single-sample BAMs (ie first read group is representative) + String bamSampleName = getToolkit().getReadsDataSource().getHeader(id).getReadGroups().get(0).getSample(); + + for(String tag : id.getTags().getPositionalTags()) { + if (BAM_TAG_TUMOR.equalsIgnoreCase(tag)) { + tumorSAMReaderIDs.add(id); + if (tumorSampleName == null) { + tumorSampleName = bamSampleName; + } else { + if (!tumorSampleName.equals(bamSampleName)) { + throw new UserException.BadInput("Found more than one tumor sample name in read data"); + } + } + } else if (BAM_TAG_NORMAL.equalsIgnoreCase(tag)) { + normalSAMReaderIDs.add(id); + if (normalSampleName == null) { + normalSampleName = bamSampleName; + } else { + if (!normalSampleName.equals(bamSampleName)) { + throw new UserException.BadInput("Found more than one normal sample name in read data"); + } + } + } else { + throw new RuntimeException("Unknown BAM tag '" + tag + "' must be either 'tumor' or 'normal'"); + } + } + } + + //If the samples specified are exactly one normal and one tumor, use the TCGA VCF sample header format + if (samplesList.sampleCount() == 2 && normalSampleName != null && tumorSampleName != null && ReadUtils.getSAMFileSamples(getToolkit().getSAMFileHeader()).size() == 2) + printTCGAsampleHeader = true; + + final VariantAnnotatorEngine annotationEngine = initializeVCFOutput(); + + try { + // fasta reference reader to supplement the edges of the reference sequence + referenceReader = new CachingIndexedFastaSequenceFile(getToolkit().getArguments().referenceFile); + } catch( FileNotFoundException e ) { + throw new UserException.CouldNotReadInputFile(getToolkit().getArguments().referenceFile, e); + } + + // create and setup the assembler + assemblyEngine = new ReadThreadingAssembler(RTAC.maxNumHaplotypesInPopulation, RTAC.kmerSizes, RTAC.dontIncreaseKmerSizesForCycles, RTAC.allowNonUniqueKmersInRef, RTAC.numPruningSamples); + + assemblyEngine.setErrorCorrectKmers(RTAC.errorCorrectKmers); + assemblyEngine.setPruneFactor(RTAC.MIN_PRUNE_FACTOR); + assemblyEngine.setDebug(MTAC.DEBUG); + assemblyEngine.setDebugGraphTransformations(RTAC.debugGraphTransformations); + assemblyEngine.setAllowCyclesInKmerGraphToGeneratePaths(RTAC.allowCyclesInKmerGraphToGeneratePaths); + assemblyEngine.setRecoverDanglingBranches(!RTAC.doNotRecoverDanglingBranches); + assemblyEngine.setMinBaseQualityToUseInAssembly(MIN_BASE_QUALTY_SCORE); + + MIN_TAIL_QUALITY = (byte)(MIN_BASE_QUALTY_SCORE - 1); + + if ( RTAC.graphWriter != null ) assemblyEngine.setGraphWriter(RTAC.graphWriter); + + // setup the likelihood calculation engine + if ( LEAC.phredScaledGlobalReadMismappingRate < 0 ) LEAC.phredScaledGlobalReadMismappingRate = -1; + + // configure the global mismapping rate + if ( LEAC.phredScaledGlobalReadMismappingRate < 0 ) { + log10GlobalReadMismappingRate = - Double.MAX_VALUE; + } else { + log10GlobalReadMismappingRate = QualityUtils.qualToErrorProbLog10(LEAC.phredScaledGlobalReadMismappingRate); + logger.info("Using global mismapping rate of " + LEAC.phredScaledGlobalReadMismappingRate + " => " + log10GlobalReadMismappingRate + " in log10 likelihood units"); + } + + //static member function - set number of threads + PairHMM.setNumberOfThreads(getToolkit().getTotalNumberOfThreads()); + // create our likelihood calculation engine + likelihoodCalculationEngine = createLikelihoodCalculationEngine(); + + final MergeVariantsAcrossHaplotypes variantMerger = new MergeVariantsAcrossHaplotypes(); + + final GenomeAnalysisEngine toolkit = getToolkit(); + final GenomeLocParser genomeLocParser = toolkit.getGenomeLocParser(); + + genotypingEngine = new SomaticGenotypingEngine( MTAC, samplesList, genomeLocParser, FixedAFCalculatorProvider.createThreadSafeProvider(getToolkit(), MTAC, logger), !doNotRunPhysicalPhasing, MTAC); + + genotypingEngine.setCrossHaplotypeEventMerger(variantMerger); + genotypingEngine.setAnnotationEngine(annotationEngine); + + + if ( MTAC.bamWriter != null ) { + // we currently do not support multi-threaded BAM writing, so exception out + if ( getToolkit().getTotalNumberOfThreads() > 1 ) + throw new UserException.BadArgumentValue("bamout", "Currently cannot emit a BAM file from the HaplotypeCaller in multi-threaded mode."); + haplotypeBAMWriter = HaplotypeBAMWriter.create(MTAC.bamWriterType, MTAC.bamWriter, getToolkit().getSAMFileHeader()); + } + + // why isn't this a constructor (instead of initialize)? Since the method is package-friendly + trimmer.initialize(getToolkit().getGenomeLocParser(), MTAC.DEBUG, + MTAC.genotypingOutputMode == GenotypingOutputMode.GENOTYPE_GIVEN_ALLELES, false); + + // KCIBUL: what's the right way to set this sensible default for somatic mutation calling from here? + trimmer.snpPadding = 50; + + samplesList = toolkit.getReadSampleList(); + Set sampleSet = SampleListUtils.asSet(samplesList); + + if( MTAC.CONTAMINATION_FRACTION_FILE != null ) + MTAC.setSampleContamination(AlleleBiasedDownsamplingUtils.loadContaminationFile(MTAC.CONTAMINATION_FRACTION_FILE, MTAC.CONTAMINATION_FRACTION, sampleSet, logger)); + + } + + private VariantAnnotatorEngine initializeVCFOutput() { + // initialize the output VCF header + final VariantAnnotatorEngine annotationEngine = new VariantAnnotatorEngine(Arrays.asList(annotationClassesToUse), annotationsToUse, annotationsToExclude, this, getToolkit()); + + Set headerInfo = new HashSet<>(); + + // all annotation fields from VariantAnnotatorEngine + headerInfo.addAll(annotationEngine.getVCFAnnotationDescriptions()); + + // all callers need to add these standard FORMAT field header lines + VCFStandardHeaderLines.addStandardFormatLines(headerInfo, true, + VCFConstants.GENOTYPE_KEY, + VCFConstants.GENOTYPE_ALLELE_DEPTHS, + VCFConstants.GENOTYPE_QUALITY_KEY, + VCFConstants.DEPTH_KEY, + VCFConstants.GENOTYPE_PL_KEY); + + headerInfo.addAll(getM2HeaderLines()); + headerInfo.addAll(getSampleHeaderLines()); + + List outputSampleNames = getOutputSampleNames(); + + vcfWriter.writeHeader(new VCFHeader(headerInfo, outputSampleNames)); + + return annotationEngine; + } + + private Set getM2HeaderLines(){ + Set headerInfo = new HashSet<>(); + headerInfo.add(GATKVCFHeaderLines.getInfoLine(GATKVCFConstants.NORMAL_LOD_KEY)); + headerInfo.add(GATKVCFHeaderLines.getInfoLine(GATKVCFConstants.TUMOR_LOD_KEY)); + headerInfo.add(GATKVCFHeaderLines.getInfoLine(GATKVCFConstants.PANEL_OF_NORMALS_COUNT_KEY)); + headerInfo.add(GATKVCFHeaderLines.getInfoLine(GATKVCFConstants.HAPLOTYPE_COUNT_KEY)); + headerInfo.add(GATKVCFHeaderLines.getInfoLine(GATKVCFConstants.EVENT_COUNT_IN_HAPLOTYPE_KEY)); + headerInfo.add(GATKVCFHeaderLines.getInfoLine(GATKVCFConstants.EVENT_DISTANCE_MIN_KEY)); + headerInfo.add(GATKVCFHeaderLines.getInfoLine(GATKVCFConstants.EVENT_DISTANCE_MAX_KEY)); + + headerInfo.add(GATKVCFHeaderLines.getFormatLine(GATKVCFConstants.ALLELE_FRACTION_KEY)); + + headerInfo.add(GATKVCFHeaderLines.getFilterLine(GATKVCFConstants.STR_CONTRACTION_FILTER_NAME)); + headerInfo.add(GATKVCFHeaderLines.getFilterLine(GATKVCFConstants.PON_FILTER_NAME)); + headerInfo.add(GATKVCFHeaderLines.getFilterLine(GATKVCFConstants.ALT_ALLELE_IN_NORMAL_FILTER_NAME)); + headerInfo.add(GATKVCFHeaderLines.getFilterLine(GATKVCFConstants.MULTI_EVENT_ALT_ALLELE_IN_NORMAL_FILTER_NAME)); + headerInfo.add(GATKVCFHeaderLines.getFilterLine(GATKVCFConstants.HOMOLOGOUS_MAPPING_EVENT_FILTER_NAME)); + headerInfo.add(GATKVCFHeaderLines.getFilterLine(GATKVCFConstants.CLUSTERED_EVENTS_FILTER_NAME)); + headerInfo.add(GATKVCFHeaderLines.getFilterLine(GATKVCFConstants.TUMOR_LOD_FILTER_NAME)); + headerInfo.add(GATKVCFHeaderLines.getFilterLine(GATKVCFConstants.GERMLINE_RISK_FILTER_NAME)); + headerInfo.add(GATKVCFHeaderLines.getFilterLine(GATKVCFConstants.TRIALLELIC_SITE_FILTER_NAME)); + + if ( ! doNotRunPhysicalPhasing ) { + headerInfo.add(GATKVCFHeaderLines.getFormatLine(GATKVCFConstants.HAPLOTYPE_CALLER_PHASING_ID_KEY)); + headerInfo.add(GATKVCFHeaderLines.getFormatLine(GATKVCFConstants.HAPLOTYPE_CALLER_PHASING_GT_KEY)); + } + return headerInfo; + } + + private Set getSampleHeaderLines(){ + Set sampleLines = new HashSet<>(); + if (printTCGAsampleHeader) { + //NOTE: This will only list the first bam file for each tumor/normal sample if there is more than one + Map normalSampleHeaderAttributes = new HashMap<>(); + normalSampleHeaderAttributes.put("ID", "NORMAL"); + normalSampleHeaderAttributes.put("SampleName", normalSampleName); + if (normalSAMReaderIDs.iterator().hasNext() && !getToolkit().getArguments().disableCommandLineInVCF) + normalSampleHeaderAttributes.put("File", normalSAMReaderIDs.iterator().next().getSamFilePath()); + VCFSimpleHeaderLine normalSampleHeader = new VCFSimpleHeaderLine("SAMPLE", normalSampleHeaderAttributes); + Map tumorSampleHeaderAttributes = new HashMap<>(); + tumorSampleHeaderAttributes.put("ID", "TUMOR"); + tumorSampleHeaderAttributes.put("SampleName", tumorSampleName); + if (tumorSAMReaderIDs.iterator().hasNext() && !getToolkit().getArguments().disableCommandLineInVCF) + tumorSampleHeaderAttributes.put("File", tumorSAMReaderIDs.iterator().next().getSamFilePath()); + VCFSimpleHeaderLine tumorSampleHeader = new VCFSimpleHeaderLine("SAMPLE", tumorSampleHeaderAttributes); + + sampleLines.add(normalSampleHeader); + sampleLines.add(tumorSampleHeader); + } + return sampleLines; + } + + private List getOutputSampleNames(){ + if (printTCGAsampleHeader) { + //Already checked for exactly 1 tumor and 1 normal in printTCGAsampleHeader assignment in initialize() + List sampleNamePlaceholders = new ArrayList<>(2); + sampleNamePlaceholders.add("TUMOR"); + sampleNamePlaceholders.add("NORMAL"); + return sampleNamePlaceholders; + } + else { + return SampleListUtils.asList(samplesList); + } + } + + @Override + public ActivityProfileState isActive(RefMetaDataTracker tracker, ReferenceContext ref, AlignmentContext context) { + if( context == null || context.getBasePileup().isEmpty() ) + // if we don't have any data, just abort early + return new ActivityProfileState(ref.getLocus(), 0.0); + + final Map splitContexts = AlignmentContextUtils.splitContextBySampleName(context); + AlignmentContext tumorContext = splitContexts.get(tumorSampleName); + AlignmentContext normalContext = splitContexts.get(normalSampleName); + + // if there are no tumor reads... there is no activity! + if (tumorContext == null) { + return new ActivityProfileState(ref.getLocus(), 0); + } + + // KCIBUL -- this method was inlined and modified from ReferenceConfidenceModel + ReadBackedPileup tumorPileup = tumorContext.getBasePileup().getMappingFilteredPileup(MQthreshold); + final double[] tumorGLs = calcGenotypeLikelihoodsOfRefVsAny(tumorPileup, ref.getBase(), MIN_BASE_QUALTY_SCORE); + final double tumorLod = tumorGLs[1] - tumorGLs[0]; + + // NOTE: do I want to convert to a probability (or just keep this as a LOD score) + + // also at this point, we should throw out noisy sites (hence the nonRefInNormalCheck) but this is non-optimal + double prob = 0; + if (tumorLod > MTAC.INITIAL_TUMOR_LOD_THRESHOLD) { + + // TODO: should we even do this performance optimization? + // in any case, we have to handle the case where there is no normal (and thus no normal context) which is + // different than having a normal but having no reads (where we should not enter the active region) + if (normalSampleName != null && normalContext != null) { + int nonRefInNormal = getCountOfNonRefEvents(normalContext.getBasePileup(), ref.getBase(), MIN_BASE_QUALTY_SCORE); + + final double[] normalGLs = calcGenotypeLikelihoodsOfRefVsAny(normalContext.getBasePileup(), ref.getBase(), MIN_BASE_QUALTY_SCORE, 0.5f); + final double normalLod = normalGLs[0] - normalGLs[1]; + + // TODO: parameterize these + if (normalLod > 1.0 && nonRefInNormal < 4) { + prob = 1; + logger.debug("At " + ref.getLocus().toString() + " tlod: " + tumorLod + " nlod: " + normalLod + " with normal non-ref of " + nonRefInNormal); + } + } else { + prob = 1; + logger.debug("At " + ref.getLocus().toString() + " tlod: " + tumorLod + " and no-normal calling"); + } + + } + + return new ActivityProfileState( ref.getLocus(), prob, ActivityProfileState.Type.NONE, null); + } + + private final static List NO_CALLS = Collections.emptyList(); + @Override + public List map( final ActiveRegion originalActiveRegion, final RefMetaDataTracker metaDataTracker ) { + if ( justDetermineActiveRegions ) + // we're benchmarking ART and/or the active region determination code in the HC, just leave without doing any work + return NO_CALLS; + + if( !originalActiveRegion.isActive() ) + // Not active so nothing to do! + return referenceModelForNoVariation(originalActiveRegion, true); + + // No reads here so nothing to do! + if( originalActiveRegion.size() == 0 ) { return referenceModelForNoVariation(originalActiveRegion, true); } + + logReadInfo(DEBUG_READ_NAME, originalActiveRegion.getReads(), "Present in original active region"); + + // create the assembly using just high quality reads (Q20 or higher). We want to use lower + // quality reads in the PairHMM (and especially in the normal) later, so we can't use a ReadFilter + ActiveRegion assemblyActiveRegion = new ActiveRegion(originalActiveRegion.getLocation(), originalActiveRegion.getSupportingStates(),originalActiveRegion.isActive(), getToolkit().getGenomeLocParser(), originalActiveRegion.getExtension()); + for (GATKSAMRecord rec : originalActiveRegion.getReads()) { + if (rec.getMappingQuality() >= MQthreshold ) { + assemblyActiveRegion.add(rec); + } + } + + logReadInfo(DEBUG_READ_NAME, assemblyActiveRegion.getReads(), "Present in assembly active region"); + + // run the local assembler, getting back a collection of information on how we should proceed + final List givenAlleles = new ArrayList<>(); + final AssemblyResultSet untrimmedAssemblyResult = assembleReads(assemblyActiveRegion, givenAlleles); + + + final TreeSet allVariationEvents = untrimmedAssemblyResult.getVariationEvents(); + // TODO - line bellow might be unecessary : it might be that assemblyResult will always have those alleles anyway + // TODO - so check and remove if that is the case: + allVariationEvents.addAll(givenAlleles); + + final ActiveRegionTrimmer.Result trimmingResult = trimmer.trim(originalActiveRegion,allVariationEvents); + + + // Stop the trimming madness!!! + if (!trimmingResult.isVariationPresent()) + return referenceModelForNoVariation(originalActiveRegion,false); + + logReadInfo(DEBUG_READ_NAME, trimmingResult.getCallableRegion().getReads(), "Present in trimming result"); + + final AssemblyResultSet assemblyResult = + trimmingResult.needsTrimming() ? untrimmedAssemblyResult.trimTo(trimmingResult.getCallableRegion()) : untrimmedAssemblyResult; + +// final AssemblyResultSet assemblyResult = untrimmedAssemblyResult; + + // after talking to Ryan -- they grab the reads out of the assembly (and trim then) to pass into the PairHMM + // because at one point they were trying error correcting of the reads based on the haplotypes.. but that is not + // working out, so it's safe for us just to take the reads +// + final ActiveRegion regionForGenotyping = assemblyResult.getRegionForGenotyping(); + logReadInfo(DEBUG_READ_NAME, regionForGenotyping.getReads(), "Present in region for genotyping"); + +// +// final ActiveRegion regionForGenotyping = trimmingResult.getCallableRegion(); + +// final ActiveRegion regionForGenotyping = originalActiveRegion; + + // filter out reads from genotyping which fail mapping quality based criteria + //TODO - why don't do this before any assembly is done? Why not just once at the beginning of this method + //TODO - on the originalActiveRegion? + //TODO - if you move this up you might have to consider to change referenceModelForNoVariation + //TODO - that does also filter reads. + final Collection filteredReads = filterNonPassingReads(regionForGenotyping); + final Map> perSampleFilteredReadList = splitReadsBySample(filteredReads); + + logReadInfo(DEBUG_READ_NAME, regionForGenotyping.getReads(), "Present in region for genotyping after filtering reads"); + + // abort early if something is out of the acceptable range + // TODO is this ever true at this point??? perhaps GGA. Need to check. + if( ! assemblyResult.isVariationPresent() ) + return referenceModelForNoVariation(originalActiveRegion, false); + + // TODO is this ever true at this point??? perhaps GGA. Need to check. + if( regionForGenotyping.size() == 0 ) { + // no reads remain after filtering so nothing else to do! + return referenceModelForNoVariation(originalActiveRegion, false); + } + + // evaluate each sample's reads against all haplotypes + + final List haplotypes = assemblyResult.getHaplotypeList(); + final Map> reads = splitReadsBySample( regionForGenotyping.getReads() ); + for (List rec : reads.values()) { + logReadInfo(DEBUG_READ_NAME, rec, "Present after splitting assemblyResult by sample"); + } + + final HashMap ARreads_origNormalMQ = new HashMap<>(); + for (GATKSAMRecord read : regionForGenotyping.getReads()) { + ARreads_origNormalMQ.put(read.getReadName(), read.getMappingQuality()); + } + + // modify MAPQ scores in normal to be high so that we don't do any base quality score capping + for(GATKSAMRecord rec : regionForGenotyping.getReads()) { + if (isReadFromNormal(rec)) { + rec.setMappingQuality(60); + } + } + + logger.debug("Computing read likelihoods with " + regionForGenotyping.getReads().size() + " reads against " + haplotypes.size() + " haplotypes across region " + assemblyResult.getRegionForGenotyping().toString()); + + + // Calculate the likelihoods: CPU intensive part. + final ReadLikelihoods readLikelihoods = + likelihoodCalculationEngine.computeReadLikelihoods(assemblyResult,samplesList,reads); + + // Realign reads to their best haplotype. + // KCIBUL: this is new stuff -- review it! + final Map readRealignments = realignReadsToTheirBestHaplotype(readLikelihoods, assemblyResult.getReferenceHaplotype(), assemblyResult.getPaddedReferenceLoc()); + readLikelihoods.changeReads(readRealignments); + + for (GATKSAMRecord rec : readRealignments.keySet()) { + logReadInfo(DEBUG_READ_NAME, rec, "Present after computing read likelihoods"); + } + + // Note: we used to subset down at this point to only the "best" haplotypes in all samples for genotyping, but there + // was a bad interaction between that selection and the marginalization that happens over each event when computing + // GLs. In particular, for samples that are heterozygous non-reference (B/C) the marginalization for B treats the + // haplotype containing C as reference (and vice versa). Now this is fine if all possible haplotypes are included + // in the genotyping, but we lose information if we select down to a few haplotypes. [EB] + + final HaplotypeCallerGenotypingEngine.CalledHaplotypes calledHaplotypes = ((SomaticGenotypingEngine)genotypingEngine).callMutations( + haplotypes, + readLikelihoods, + ARreads_origNormalMQ, + perSampleFilteredReadList, + assemblyResult.getFullReferenceWithPadding(), + assemblyResult.getPaddedReferenceLoc(), + regionForGenotyping.getLocation(), + getToolkit().getGenomeLocParser(), + metaDataTracker, + givenAlleles, false , + tumorSampleName, + normalSampleName, + dbsnp.dbsnp, + cosmicRod, + DEBUG_READ_NAME + ); + + if ( MTAC.bamWriter != null ) { + final Set calledHaplotypeSet = new HashSet<>(calledHaplotypes.getCalledHaplotypes()); + if (MTAC.disableOptimizations) + calledHaplotypeSet.add(assemblyResult.getReferenceHaplotype()); + haplotypeBAMWriter.writeReadsAlignedToHaplotypes( + haplotypes, + assemblyResult.getPaddedReferenceLoc(), + haplotypes, + calledHaplotypeSet, + readLikelihoods); + } + + if( MTAC.DEBUG ) { logger.info("----------------------------------------------------------------------------------"); } + + + List annotatedCalls = new ArrayList<>(); + int eventCount = calledHaplotypes.getCalls().size(); + Integer minEventDistance = null; + Integer maxEventDistance = null; + Integer lastPosition = null; + for (VariantContext vc : calledHaplotypes.getCalls()) { + if (lastPosition == null) { + lastPosition = vc.getStart(); + } else { + int dist = Math.abs(vc.getStart() - lastPosition); + if (maxEventDistance == null || dist > maxEventDistance) { + maxEventDistance = dist; + } + if (minEventDistance == null || dist < minEventDistance) { + minEventDistance = dist; + } + } + } + Map eventDistanceAttributes = new HashMap<>(); + eventDistanceAttributes.put(GATKVCFConstants.EVENT_COUNT_IN_HAPLOTYPE_KEY, eventCount); + eventDistanceAttributes.put(GATKVCFConstants.EVENT_DISTANCE_MIN_KEY, minEventDistance); + eventDistanceAttributes.put(GATKVCFConstants.EVENT_DISTANCE_MAX_KEY, maxEventDistance); + + + // can we do this with the Annotation classes instead? + for (VariantContext originalVC : calledHaplotypes.getCalls()) { + VariantContextBuilder vcb = new VariantContextBuilder(originalVC); + + Map attributes = new HashMap<>(originalVC.getAttributes()); + attributes.putAll(eventDistanceAttributes); + vcb.attributes(attributes); + + Set filters = new HashSet<>(originalVC.getFilters()); + + double tumorLod = originalVC.getAttributeAsDouble(GATKVCFConstants.TUMOR_LOD_KEY, -1); + if (tumorLod < MTAC.TUMOR_LOD_THRESHOLD) { + filters.add(GATKVCFConstants.TUMOR_LOD_FILTER_NAME); + } + + // if we are in artifact detection mode, apply the thresholds for the LOD scores + if (!MTAC.ARTIFACT_DETECTION_MODE) { + filters.addAll(calculateFilters(metaDataTracker, originalVC, eventDistanceAttributes)); + } + + if (filters.size() > 0) { + vcb.filters(filters); + } else { + vcb.passFilters(); + } + + if (printTCGAsampleHeader) { + GenotypesContext genotypesWithBamSampleNames = originalVC.getGenotypes(); + List renamedGenotypes = new ArrayList<>(); + GenotypeBuilder GTbuilder = new GenotypeBuilder(genotypesWithBamSampleNames.get(tumorSampleName)); + GTbuilder.name("TUMOR"); + renamedGenotypes.add(GTbuilder.make()); + GTbuilder = new GenotypeBuilder(genotypesWithBamSampleNames.get(normalSampleName)); + GTbuilder.name("NORMAL"); + renamedGenotypes.add(GTbuilder.make()); + vcb.genotypes(renamedGenotypes); + } + + annotatedCalls.add(vcb.make()); + } + + + + + + + return annotatedCalls; + } + + private Set calculateFilters(RefMetaDataTracker metaDataTracker, VariantContext vc, Map eventDistanceAttributes) { + Set filters = new HashSet<>(); + + Integer eventCount = (Integer) eventDistanceAttributes.get(GATKVCFConstants.EVENT_COUNT_IN_HAPLOTYPE_KEY); + Integer maxEventDistance = (Integer) eventDistanceAttributes.get(GATKVCFConstants.EVENT_DISTANCE_MAX_KEY); + + Collection panelOfNormalsVC = metaDataTracker.getValues(normalPanelRod, + getToolkit().getGenomeLocParser().createGenomeLoc(vc.getChr(), vc.getStart())); + VariantContext ponVc = panelOfNormalsVC.isEmpty()?null:panelOfNormalsVC.iterator().next(); + + if (ponVc != null) { + filters.add(GATKVCFConstants.PON_FILTER_NAME); + } + + // FIXME: how do we sum qscores here? + // FIXME: parameterize thresholds + // && sum of alt likelihood scores > 20 + + // TODO: make the change to have only a single normal sample (but multiple tumors is ok...) + int normalAltCounts = 0; + double normalF = 0; + int normalAltQualityScoreSum = 0; + if (hasNormal()) { + Genotype normalGenotype = vc.getGenotype(normalSampleName); + + // NOTE: how do we get the non-ref depth here? + normalAltCounts = normalGenotype.getAD()[1]; + normalF = (Double) normalGenotype.getExtendedAttribute(GATKVCFConstants.ALLELE_FRACTION_KEY); + + Object qss = normalGenotype.getExtendedAttribute(GATKVCFConstants.QUALITY_SCORE_SUM_KEY); + if (qss != null) { + normalAltQualityScoreSum = (Integer) ((Object[]) qss)[1]; + } else { + logger.error("Null qss at " + vc.getStart()); + } + } + + if ( (normalAltCounts >= MTAC.MAX_ALT_ALLELES_IN_NORMAL_COUNT || normalF >= MTAC.MAX_ALT_ALLELE_IN_NORMAL_FRACTION ) && normalAltQualityScoreSum > MTAC.MAX_ALT_ALLELES_IN_NORMAL_QSCORE_SUM) { + filters.add(GATKVCFConstants.ALT_ALLELE_IN_NORMAL_FILTER_NAME); + } else { + + // NOTE: does normal alt counts presume the normal had all these events in CIS? + if ( eventCount > 1 && normalAltCounts >= 1) { + filters.add(GATKVCFConstants.MULTI_EVENT_ALT_ALLELE_IN_NORMAL_FILTER_NAME); + } else if (eventCount >= 3) { + filters.add(GATKVCFConstants.HOMOLOGOUS_MAPPING_EVENT_FILTER_NAME); + } + + } + + // STR contractions, that is the deletion of one repeat unit of a short repeat (>1bp repeat unit) + // such as ACTACTACT -> ACTACT, are overwhelmingly false positives so we + // hard filter them out by default + if (vc.isIndel()) { + ArrayList rpa = (ArrayList) vc.getAttribute(GATKVCFConstants.REPEATS_PER_ALLELE_KEY); + String ru = vc.getAttributeAsString(GATKVCFConstants.REPEAT_UNIT_KEY, ""); + if (rpa != null && rpa.size() > 1 && ru.length() > 1) { + int refCount = (Integer) rpa.get(0); + int altCount = (Integer) rpa.get(1); + + if (refCount - altCount == 1) { + filters.add(GATKVCFConstants.STR_CONTRACTION_FILTER_NAME); + } + } + } + + // NOTE: what if there is a 3bp indel followed by a snp... we are comparing starts + // so it would be thrown but it's really an adjacent event + if ( eventCount >= 2 && maxEventDistance >= 3) { + filters.add(GATKVCFConstants.CLUSTERED_EVENTS_FILTER_NAME); + } + + return filters; + } + + + private final static byte REF_MODEL_DELETION_QUAL = (byte) 30; + /** + * Calculate the genotype likelihoods for the sample in pileup for being hom-ref contrasted with being ref vs. alt + * + * @param pileup the read backed pileup containing the data we want to evaluate + * @param refBase the reference base at this pileup position + * @param minBaseQual the min base quality for a read in the pileup at the pileup position to be included in the calculation + * @return genotype likelihoods of [AA,AB] + */ + protected double[] calcGenotypeLikelihoodsOfRefVsAny(final ReadBackedPileup pileup, final byte refBase, final byte minBaseQual, final double f) { + final double[] genotypeLikelihoods = new double[2]; + int AA = 0, AB=1; + for( final PileupElement p : pileup ) { + final byte qual = (p.isDeletion() ? REF_MODEL_DELETION_QUAL : p.getQual()); + if( p.isDeletion() || qual > minBaseQual ) { + + // TODO: why not use base qualities here? + //double pobs = QualityUtils.qualToErrorProbLog10(qual); + double pobs = 1.0d - pow(10, (30 / -10.0)); + if( isNonRef(refBase, p)) { + genotypeLikelihoods[AB] += Math.log10(f*pobs + (1-f)*pobs/3.0d); + genotypeLikelihoods[AA] += Math.log10((1-pobs)/3); + } else { + genotypeLikelihoods[AB] += Math.log10(f*(1-pobs)/3.0d + (1-f)*pobs); + genotypeLikelihoods[AA] += Math.log10(pobs); + } + } + } + + return genotypeLikelihoods; + } + + private boolean hasNormal() { + return (normalSampleName != null); + } + + protected int getCountOfNonRefEvents(final ReadBackedPileup pileup, final byte refBase, final byte minBaseQual) { + int i=0; + for( final PileupElement p : pileup ) { + final byte qual = (p.isDeletion() ? REF_MODEL_DELETION_QUAL : p.getQual()); + if( p.isDeletion() || qual > minBaseQual ) { + if( isNonRef(refBase, p)) { + i++; + } + } + } + return i; + } + + protected double[] calcGenotypeLikelihoodsOfRefVsAny(final ReadBackedPileup pileup, final byte refBase, final byte minBaseQual) { + double f = calculateF(pileup, refBase, minBaseQual); + return calcGenotypeLikelihoodsOfRefVsAny(pileup, refBase, minBaseQual, f); + } + + private double calculateF(final ReadBackedPileup pileup, final byte refBase, final byte minBaseQual) { + int refCount = 0, altCount = 0; + for( final PileupElement p : pileup ) { + final byte qual = (p.isDeletion() ? REF_MODEL_DELETION_QUAL : p.getQual()); + + // only consider deletions AND sites of sufficient quality + if( p.isDeletion() || qual > minBaseQual ) { + if( isNonRef(refBase, p)) { + altCount++; + } else { + refCount++; + } + } + } + double f = (double) altCount / ((double) refCount + (double) altCount); + return f; + } + + private boolean isNonRef(byte refBase, PileupElement p) { + return p.getBase() != refBase || p.isDeletion() || p.isBeforeDeletionStart() || p.isAfterDeletionEnd() || p.isBeforeInsertion() || p.isAfterInsertion() || p.isNextToSoftClip(); + } + + int MIN_READ_LENGTH = 30; // private in superclass + + protected Set filterNonPassingReads( final ActiveRegion activeRegion) { + final Set readsToRemove = new LinkedHashSet<>(); + for( final GATKSAMRecord rec : activeRegion.getReads() ) { + + // KCIBUL: only perform read quality filtering on tumor reads... + if (isReadFromNormal(rec)) { + + if( rec.getReadLength() < MIN_READ_LENGTH ) { + readsToRemove.add(rec); + } + + } else { + + + if( rec.getReadLength() < MIN_READ_LENGTH || + rec.getMappingQuality() < MQthreshold || + BadMateFilter.hasBadMate(rec) || + + (keepRG != null && !rec.getReadGroup().getId().equals(keepRG)) ) { + readsToRemove.add(rec); + } + } + } + activeRegion.removeAll(readsToRemove); + return readsToRemove; + } + + private static GATKSAMRecord findReadByName(Collection reads, String name) { + for(GATKSAMRecord read : reads) { + if (name.equals(read.getReadName())) return read; + } + return null; + } + + /** + * Instantiates the appropriate likelihood calculation engine. + * + * @return never {@code null}. + */ + private ReadLikelihoodCalculationEngine createLikelihoodCalculationEngine() { + return new PairHMMLikelihoodCalculationEngine( (byte)LEAC.gcpHMM, LEAC.pairHMM, LEAC.pairHMMSub, LEAC.alwaysLoadVectorLoglessPairHMMLib, log10GlobalReadMismappingRate, LEAC.noFpga, pcrErrorModel ); + } + + /** + * FROM HC + * + * Create an ref model result (ref model or no calls depending on mode) for an active region without any variation + * (not is active, or assembled to just ref) + * + * @param region the region to return a no-variation result + * @param needsToBeFinalized should the region be finalized before computing the ref model (should be false if already done) + * @return a list of variant contexts (can be empty) to emit for this ref region + */ + protected List referenceModelForNoVariation(final ActiveRegion region, final boolean needsToBeFinalized) { + return NO_CALLS; + } + + protected Map> splitReadsBySample( final Collection reads ) { + return HaplotypeCaller.splitReadsBySample(samplesList, reads); + } + + // enable deletions in the pileup + @Override + public boolean includeReadsWithDeletionAtLoci() { return true; } + + // enable non primary and extended reads in the active region + @Override + public EnumSet desiredReadStates() { +// if ( includeUnmappedReads ) +// throw new UserException.BadArgumentValue("includeUnmappedReads", "is not yet functional"); +// else + return EnumSet.of( + ActiveRegionReadState.PRIMARY, + ActiveRegionReadState.NONPRIMARY, + ActiveRegionReadState.EXTENDED); + } + + //--------------------------------------------------------------------------------------------------------------- + // + // reduce + // + //--------------------------------------------------------------------------------------------------------------- + + @Override + public Integer reduceInit() { + return 0; + } + + @Override + public Integer reduce(List callsInRegion, Integer numCalledRegions) { + for( final VariantContext call : callsInRegion ) { + vcfWriter.add( call ); + } + return (callsInRegion.isEmpty() ? 0 : 1) + numCalledRegions; + } + + @Override + public void onTraversalDone(Integer result) { +// if ( SCAC.emitReferenceConfidence == ReferenceConfidenceMode.GVCF ) ((GVCFWriter)vcfWriter).close(false); // GROSS -- engine forces us to close our own VCF writer since we wrapped it +// referenceConfidenceModel.close(); + //TODO remove the need to call close here for debugging, the likelihood output stream should be managed + //TODO (open & close) at the walker, not the engine. + likelihoodCalculationEngine.close(); + logger.info("Ran local assembly on " + result + " active regions"); + } + + + // The following are not used but are required by the AnnotatorCompatible interface + public RodBinding getSnpEffRodBinding() { return null; } + public List> getResourceRodBindings() { return Collections.emptyList(); } + public boolean alwaysAppendDbsnpId() { return false; } + + /** + * rsIDs from this file are used to populate the ID column of the output. Also, the DB INFO flag will be set when appropriate. + * dbSNP overlap is only used to require more evidence of absence in the normal if the variant in question has been seen before in germline. + */ + @ArgumentCollection + protected DbsnpArgumentCollection dbsnp = new DbsnpArgumentCollection(); + public RodBinding getDbsnpRodBinding() { return dbsnp.dbsnp; } + + /** + * If a call overlaps with a record from the provided comp track, the INFO field will be annotated + * as such in the output with the track name (e.g. -comp:FOO will have 'FOO' in the INFO field). + * Records that are filtered in the comp track will be ignored. + * Note that 'dbSNP' has been special-cased (see the --dbsnp argument). + */ + @Advanced + @Input(fullName="comp", shortName = "comp", doc="comparison VCF file", required=false) + public List> comps = Collections.emptyList(); + public List> getCompRodBindings() { return comps; } + + + + /** + * Which annotations to add to the output VCF file. See the VariantAnnotator -list argument to view available annotations. + */ + @Advanced + @Argument(fullName="annotation", shortName="A", doc="One or more specific annotations to apply to variant calls", required=false) +// protected List annotationsToUse = new ArrayList<>(Arrays.asList(new String[]{"ClippingRankSumTest", "DepthPerSampleHC"})); + //protected List annotationsToUse = new ArrayList<>(Arrays.asList(new String[]{"DepthPerAlleleBySample", "BaseQualitySumPerAlleleBySample", "TandemRepeatAnnotator", + // "RMSMappingQuality","MappingQualityRankSumTest","FisherStrand","StrandOddsRatio","ReadPosRankSumTest","QualByDepth", "Coverage"})); + protected List annotationsToUse = new ArrayList<>(Arrays.asList(new String[]{"DepthPerAlleleBySample", "BaseQualitySumPerAlleleBySample", "TandemRepeatAnnotator", "OxoGReadCounts"})); + + /** + * Which annotations to exclude from output in the VCF file. Note that this argument has higher priority than the -A or -G arguments, + * so annotations will be excluded even if they are explicitly included with the other options. + */ + @Advanced + @Argument(fullName="excludeAnnotation", shortName="XA", doc="One or more specific annotations to exclude", required=false) + protected List annotationsToExclude = new ArrayList<>(Arrays.asList(new String[]{"SpanningDeletions"})); + + /** + * Which groups of annotations to add to the output VCF file. See the VariantAnnotator -list argument to view available groups. + */ + @Argument(fullName="group", shortName="G", doc="One or more classes/groups of annotations to apply to variant calls", required=false) + //protected String[] annotationGroupsToUse = { "Standard" }; + protected String[] annotationClassesToUse = { }; + + /** + * A raw, unfiltered, highly sensitive callset in VCF format. + */ + @Output(doc="File to which variants should be written") + protected VariantContextWriter vcfWriter = null; + + /** + * Active region trimmer reference. + */ + @ArgumentCollection + protected ActiveRegionTrimmer trimmer = new ActiveRegionTrimmer(); + + @Hidden + @Argument(fullName="keepRG", shortName="keepRG", doc="Only use read from this read group when making calls (but use all reads to build the assembly)", required = false) + protected String keepRG = null; + + + + + /** + * The minimum confidence needed for a given base for it to be used in variant calling. + */ + @Argument(fullName = "min_base_quality_score", shortName = "mbq", doc = "Minimum base quality required to consider a base for calling", required = false) + public byte MIN_BASE_QUALTY_SCORE = 10; + + + +// PAIR-HMM-Related Goodness + +// public PairHMMLikelihoodCalculationEngine.PCR_ERROR_MODEL pcrErrorModel = PairHMMLikelihoodCalculationEngine.PCR_ERROR_MODEL.CONSERVATIVE; +// public PairHMMLikelihoodCalculationEngine.PCR_ERROR_MODEL pcrErrorModel = PairHMMLikelihoodCalculationEngine.PCR_ERROR_MODEL.AGGRESSIVE; + public PairHMMLikelihoodCalculationEngine.PCR_ERROR_MODEL pcrErrorModel = PairHMMLikelihoodCalculationEngine.PCR_ERROR_MODEL.HOSTILE; + + // Parameters to control read error correction + @Hidden + @Argument(fullName="errorCorrectReads", shortName="errorCorrectReads", doc = "Use an exploratory algorithm to error correct the kmers used during assembly. May cause fundamental problems with the assembly graph itself", required=false) + protected boolean errorCorrectReads = false; + + @Hidden + @Argument(fullName="captureAssemblyFailureBAM", shortName="captureAssemblyFailureBAM", doc="If specified, we will write a BAM called assemblyFailure.bam capturing all of the reads that were in the active region when the assembler failed for any reason", required = false) + protected boolean captureAssemblyFailureBAM = false; + + @Advanced + @Argument(fullName="dontUseSoftClippedBases", shortName="dontUseSoftClippedBases", doc="If specified, we will not analyze soft clipped bases in the reads", required = false) + protected boolean dontUseSoftClippedBases = false; + + @Hidden + @Argument(fullName="justDetermineActiveRegions", shortName="justDetermineActiveRegions", doc = "If specified, the HC won't actually do any assembly or calling, it'll just run the upfront active region determination code. Useful for benchmarking and scalability testing", required=false) + protected boolean justDetermineActiveRegions = false; + + + + + // reference base padding size + private static final int REFERENCE_PADDING = 500; + + private static final byte MIN_TAIL_QUALITY_WITH_ERROR_CORRECTION = 6; + private final static int maxReadsInRegionPerSample = 1000; // TODO -- should be an argument + private final static int minReadsPerAlignmentStart = 5; // TODO -- should be an argument + + + + /** + * High-level function that runs the assembler on the active region reads, + * returning a data structure with the resulting information needed + * for further HC steps + * + * @param activeRegion the region we should assemble + * @param giveAlleles additional alleles we might need to genotype (can be empty) + * @return the AssemblyResult describing how to proceed with genotyping + */ + protected AssemblyResultSet assembleReads(final ActiveRegion activeRegion, final List giveAlleles) { + // Create the reference haplotype which is the bases from the reference that make up the active region + finalizeActiveRegion(activeRegion); // handle overlapping fragments, clip adapter and low qual tails + + final byte[] fullReferenceWithPadding = activeRegion.getActiveRegionReference(referenceReader, REFERENCE_PADDING); + final GenomeLoc paddedReferenceLoc = getPaddedLoc(activeRegion); + final Haplotype referenceHaplotype = createReferenceHaplotype(activeRegion, paddedReferenceLoc); + + // Create ReadErrorCorrector object if requested - will be used within assembly engine. + ReadErrorCorrector readErrorCorrector = null; + if (errorCorrectReads) + readErrorCorrector = new ReadErrorCorrector(RTAC.kmerLengthForReadErrorCorrection, MIN_TAIL_QUALITY_WITH_ERROR_CORRECTION, RTAC.minObservationsForKmerToBeSolid, MTAC.DEBUG, fullReferenceWithPadding); + + try { + final AssemblyResultSet assemblyResultSet = assemblyEngine.runLocalAssembly( activeRegion, referenceHaplotype, fullReferenceWithPadding, paddedReferenceLoc, giveAlleles,readErrorCorrector ); + assemblyResultSet.debugDump(logger); + return assemblyResultSet; + + } catch ( final Exception e ) { + // Capture any exception that might be thrown, and write out the assembly failure BAM if requested + if ( captureAssemblyFailureBAM ) { + final SAMFileWriter writer = SAMFileWriterStub.createSAMFileWriter("assemblyFailure.bam", getToolkit()); + new DirectOutputTracker().addOutput((SAMFileWriterStub) writer); + for ( final GATKSAMRecord read : activeRegion.getReads() ) { + writer.addAlignment(read); + } + writer.close(); + } + throw e; + } + } + + private void finalizeActiveRegion( final ActiveRegion activeRegion ) { + if (activeRegion.isFinalized()) return; + + if( MTAC.DEBUG ) { logger.info("Assembling " + activeRegion.getLocation() + " with " + activeRegion.size() + " reads: (with overlap region = " + activeRegion.getExtendedLoc() + ")"); } + + // Loop through the reads hard clipping the adaptor and low quality tails + final List readsToUse = new ArrayList<>(activeRegion.getReads().size()); + for( final GATKSAMRecord myRead : activeRegion.getReads() ) { + GATKSAMRecord clippedRead; + if (errorCorrectReads) + clippedRead = ReadClipper.hardClipLowQualEnds( myRead, MIN_TAIL_QUALITY_WITH_ERROR_CORRECTION ); + else // default case: clip low qual ends of reads + clippedRead= ReadClipper.hardClipLowQualEnds( myRead, MIN_TAIL_QUALITY ); + + if ( dontUseSoftClippedBases || ! ReadUtils.hasWellDefinedFragmentSize(clippedRead) ) { + // remove soft clips if we cannot reliably clip off adapter sequence or if the user doesn't want to use soft clips at all + clippedRead = ReadClipper.hardClipSoftClippedBases(clippedRead); + } else { + // revert soft clips so that we see the alignment start and end assuming the soft clips are all matches + // TODO -- WARNING -- still possibility that unclipping the soft clips will introduce bases that aren't + // TODO -- truly in the extended region, as the unclipped bases might actually include a deletion + // TODO -- w.r.t. the reference. What really needs to happen is that kmers that occur before the + // TODO -- reference haplotype start must be removed + clippedRead = ReadClipper.revertSoftClippedBases(clippedRead); + } + + clippedRead = ( clippedRead.getReadUnmappedFlag() ? clippedRead : ReadClipper.hardClipAdaptorSequence( clippedRead ) ); + if( !clippedRead.isEmpty() && clippedRead.getCigar().getReadLength() > 0 ) { + clippedRead = ReadClipper.hardClipToRegion(clippedRead, activeRegion.getExtendedLoc().getStart(), activeRegion.getExtendedLoc().getStop()); + if( activeRegion.readOverlapsRegion(clippedRead) && clippedRead.getReadLength() > 0 ) { + //logger.info("Keeping read " + clippedRead + " start " + clippedRead.getAlignmentStart() + " end " + clippedRead.getAlignmentEnd()); + readsToUse.add(clippedRead); + } + } + } + + // TODO -- Performance optimization: we partition the reads by sample 4 times right now; let's unify that code. + + final List downsampledReads = DownsamplingUtils.levelCoverageByPosition(ReadUtils.sortReadsByCoordinate(readsToUse), maxReadsInRegionPerSample, minReadsPerAlignmentStart); + + // handle overlapping read pairs from the same fragment + // KC: commented out as we handle overlapping read pairs in a different way... + //cleanOverlappingReadPairs(downsampledReads, normalSampleNames); + + activeRegion.clearReads(); + activeRegion.addAll(downsampledReads); + activeRegion.setFinalized(true); + } + + private GenomeLoc getPaddedLoc( final ActiveRegion activeRegion ) { + final int padLeft = Math.max(activeRegion.getExtendedLoc().getStart()-REFERENCE_PADDING, 1); + final int padRight = Math.min(activeRegion.getExtendedLoc().getStop()+REFERENCE_PADDING, referenceReader.getSequenceDictionary().getSequence(activeRegion.getExtendedLoc().getContig()).getSequenceLength()); + return getToolkit().getGenomeLocParser().createGenomeLoc(activeRegion.getExtendedLoc().getContig(), padLeft, padRight); + } + + /** + * Helper function to create the reference haplotype out of the active region and a padded loc + * @param activeRegion the active region from which to generate the reference haplotype + * @param paddedReferenceLoc the GenomeLoc which includes padding and shows how big the reference haplotype should be + * @return a non-null haplotype + */ + private Haplotype createReferenceHaplotype(final ActiveRegion activeRegion, final GenomeLoc paddedReferenceLoc) { + return ReferenceConfidenceModel.createReferenceHaplotype(activeRegion, activeRegion.getActiveRegionReference(referenceReader), paddedReferenceLoc); + } + + /** + * Clean up reads/bases that overlap within read pairs + * + * @param reads the list of reads to consider + */ + private void cleanOverlappingReadPairs(final List reads, Set normalSampleNames) { + Map> data = splitReadsBySample(reads); + for ( String sampleName : data.keySet() ) { + final boolean isTumor = !normalSampleNames.contains(sampleName); + final List perSampleReadList = data.get(sampleName); + + final FragmentCollection fragmentCollection = FragmentUtils.create(perSampleReadList); + for ( final List overlappingPair : fragmentCollection.getOverlappingPairs() ) + + // in MuTect -- right now we compare the + FragmentUtils.adjustQualsOfOverlappingPairedFragments(overlappingPair); + + + } + } + + public static void logReadInfo(String readName, Collection records, String message) { + if (readName != null) { + for (GATKSAMRecord rec : records) { + logReadInfo(readName, rec, message); + } + + } + } + + public static void logReadInfo(String readName, GATKSAMRecord rec, String message) { + if (readName != null && rec != null && readName.equals(rec.getReadName())) { + logger.info("Found " + rec.toString() + " - " + message); + } + } + + /** + * Returns a map with the original read as a key and the realigned read as the value. + *

+ * Missing keys or equivalent key and value pairs mean that the read was not realigned. + *

+ * @return never {@code null} + */ + // TODO: migrate from HC -> HCUtils Class and share it! + private Map realignReadsToTheirBestHaplotype(final ReadLikelihoods originalReadLikelihoods, final Haplotype refHaplotype, final GenomeLoc paddedReferenceLoc) { + + final Collection.BestAllele> bestAlleles = originalReadLikelihoods.bestAlleles(); + final Map result = new HashMap<>(bestAlleles.size()); + + for (final ReadLikelihoods.BestAllele bestAllele : bestAlleles) { + final GATKSAMRecord originalRead = bestAllele.read; + final Haplotype bestHaplotype = bestAllele.allele; + final boolean isInformative = bestAllele.isInformative(); + final GATKSAMRecord realignedRead = AlignmentUtils.createReadAlignedToRef(originalRead, bestHaplotype, refHaplotype, paddedReferenceLoc.getStart(), isInformative); + result.put(originalRead,realignedRead); + } + return result; + } + + private boolean isReadFromNormal(GATKSAMRecord rec) { + return normalSampleName != null && normalSampleName.equals(rec.getReadGroup().getSample()); + + } + // KCIBUL: new stuff -- read up on this!! + /** + * As of GATK 3.3, HaplotypeCaller outputs physical (read-based) information (see version 3.3 release notes and documentation for details). This argument disables that behavior. + */ + @Advanced + @Argument(fullName="doNotRunPhysicalPhasing", shortName="doNotRunPhysicalPhasing", doc="Disable physical phasing", required = false) + protected boolean doNotRunPhysicalPhasing = false; + +} + + diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/NA12878_Evaluations.md b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/NA12878_Evaluations.md new file mode 100644 index 000000000..b7d0c58f2 --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/NA12878_Evaluations.md @@ -0,0 +1,81 @@ +# CRSP ICE NA12878 Specificity Evaluation + +In order to evaluate the specificity of M2, we sequenced replicates of NA12878 using ICE (Illumina Content Exomes) all call all pairwise combinations as tumor-normals. By definition, everything called is a false positive. + +The target territory is ```/dsde/working/mutect/crsp_nn/whole_exome_illumina_coding_v1.Homo_sapiens_assembly19.targets.no_empty.interval_list``` + +All scripts referenced here are relative to the current working directory of ``` +/dsde/working/mutect/crsp_nn``` + +### Current M2 Performance + +(gsa-unstable 7/13/15, commit:9e93a70) + +| type | # of false positives | +|------|----------------------| +|SNP|99| +|INDEL|15| + + +TODO: write a simple tool to do this more easily + +To calculate per pair-counts, run: +``` +# for SNPs +for vcf in *.bam.vcf +do + cat $vcf | grep PASS | awk '{ if ( length($4) + length($5) == 2) print $0 }' | wc -l +done + +# for INDELs +for vcf in *.bam.vcf +do + cat $vcf | grep PASS | awk '{ if ( length($4) + length($5) != 2) print $0 }' | wc -l +done +``` + +### Current M1 and Indelocator Performance +For comparison, the M1 & Indelocator calls have been made on this same data set in the Firehose workspace ```CRSP_ICE_NA1878_Production_Analysis``` in the pair set ```NA12878_Replicate_Pairs``` which contains 4 samples and 12 pairwise combinations. + +| type | # of false positives | +|------|----------------------| +|SNP|181| +|INDEL|106| + +These results can be obtained (from a LSF / CGA node running the FuSE daemon) + +``` +SNP: +cat /local/cga-fh/cga/CRSP_ICE_NA1878_Production_Analysis/Pair_Set/NA12878_Replicate_Pairs/Pair/*/jobs/capture/mut/calls/latest/*.call_stats.txt | grep KEEP | wc -l + +INDEL (need to restrict to target territory): +reuse BEDTools +cat /dsde/working/mutect/crsp_nn/whole_exome_illumina_coding_v1.Homo_sapiens_assembly19.targets.no_empty.interval_list | grep -v "@" | awk '{ print $1 "\t" $2-1 "\t" $3 }' > ice.bed +cat /local/cga-fh/cga/CRSP_ICE_NA1878_Production_Analysis/Pair_Set/NA12878_Replicate_Pairs/Pair/*/jobs/capture/indel/maflite/latest/*.full.maf | grep KEEP | cut -f2-4 | awk '{ print $1 "\t" $2-1 "\t" $3 }' > /tmp/indels.bed +bedtools intersect -wa -a /tmp/ice.bed -b /tmp/indels.bed | wc -l +``` + + +### How To Run +The SCALA script for running M2 can be found in the gsa-unstable repository under ```private/gatk-tools-private/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2``` + +First, chose the appropriate settings (runnable as environment variables here) +``` +QUEUE_JAR= +OUT_VCF= +GSA_UNSTABLE_HOME= +TEMPDIR=/broad/hptmp/$USER +``` + +and then run the following Queue command +``` +java \ + -Djava.io.tmpdir=$TEMPDIR \ + -jar $QUEUE_JAR \ + -S $GSA_UNSTABLE_HOME/private/gatk-tools-private/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/run_M2_ICE_NN.scala \ + -sc 50 \ + --job_queue gsa -qsub -jobResReq virtual_free=5G -startFromScratch \ + --allbams /humgen/gsa-hpprojects/NA12878Collection/bams/crsp_ice_validation//NA12878.intra.flowcell.replicate.bam_list \ + -o + -run +``` diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/SomaticGenotypingEngine.java b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/SomaticGenotypingEngine.java new file mode 100644 index 000000000..2c05ce825 --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/SomaticGenotypingEngine.java @@ -0,0 +1,558 @@ +/* +* By downloading the PROGRAM you agree to the following terms of use: +* +* BROAD INSTITUTE +* SOFTWARE LICENSE AGREEMENT +* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY +* +* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”). +* +* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and +* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions. +* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows: +* +* 1. DEFINITIONS +* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE. +* +* 2. LICENSE +* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute. LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation. +* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement. +* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement. +* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM. +* +* 3. PHONE-HOME FEATURE +* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM. Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time. Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation. +* +* 4. OWNERSHIP OF INTELLECTUAL PROPERTY +* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication. +* Copyright 2012-2015 Broad Institute, Inc. +* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc. +* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes. +* +* 5. INDEMNIFICATION +* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement. +* +* 6. NO REPRESENTATIONS OR WARRANTIES +* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME. +* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING. +* +* 7. ASSIGNMENT +* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void. +* +* 8. MISCELLANEOUS +* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries. +* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes. +* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4. +* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt. +* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter. +* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement. +* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles. +*/ + +package org.broadinstitute.gatk.tools.walkers.cancer.m2; + +import com.google.java.contract.Ensures; +import htsjdk.samtools.util.StringUtil; +import htsjdk.variant.variantcontext.*; +import org.apache.log4j.Logger; +import org.broadinstitute.gatk.tools.walkers.genotyper.GenotypeLikelihoodsCalculationModel; +import org.broadinstitute.gatk.tools.walkers.genotyper.afcalc.AFCalculatorProvider; +import org.broadinstitute.gatk.tools.walkers.haplotypecaller.HaplotypeCallerGenotypingEngine; +import org.broadinstitute.gatk.utils.GenomeLoc; +import org.broadinstitute.gatk.utils.GenomeLocParser; +import org.broadinstitute.gatk.utils.commandline.RodBinding; +import org.broadinstitute.gatk.utils.contexts.ReferenceContext; +import org.broadinstitute.gatk.utils.genotyper.MostLikelyAllele; +import org.broadinstitute.gatk.utils.genotyper.PerReadAlleleLikelihoodMap; +import org.broadinstitute.gatk.utils.genotyper.ReadLikelihoods; +import org.broadinstitute.gatk.utils.genotyper.SampleList; +import org.broadinstitute.gatk.utils.haplotype.Haplotype; +import org.broadinstitute.gatk.utils.refdata.RefMetaDataTracker; +import org.broadinstitute.gatk.utils.sam.GATKSAMRecord; +import org.broadinstitute.gatk.utils.sam.ReadUtils; +import org.broadinstitute.gatk.utils.variant.GATKVCFConstants; +import org.broadinstitute.gatk.utils.variant.GATKVariantContextUtils; + +import java.util.*; + +public class SomaticGenotypingEngine extends HaplotypeCallerGenotypingEngine { + + protected M2ArgumentCollection MTAC; + + private final static Logger logger = Logger.getLogger(SomaticGenotypingEngine.class); + + public SomaticGenotypingEngine(final M2ArgumentCollection configuration, final SampleList samples, final GenomeLocParser genomeLocParser, final AFCalculatorProvider afCalculatorProvider, final boolean doPhysicalPhasing, final M2ArgumentCollection MTAC) { + super(configuration, samples, genomeLocParser, afCalculatorProvider, doPhysicalPhasing); + this.MTAC = MTAC; + } + + /** + * Main entry point of class - given a particular set of haplotypes, samples and reference context, compute + * genotype likelihoods and assemble into a list of variant contexts and genomic events ready for calling + * + * The list of samples we're working with is obtained from the readLikelihoods + * + * @param haplotypes Haplotypes to assign likelihoods to + * @param readLikelihoods Map from reads->(haplotypes,likelihoods) + * @param perSampleFilteredReadList Map from sample to reads that were filtered after assembly and before calculating per-read likelihoods. + * @param ref Reference bytes at active region + * @param refLoc Corresponding active region genome location + * @param activeRegionWindow Active window + * @param genomeLocParser GenomeLocParser + * @param activeAllelesToGenotype Alleles to genotype + * @param emitReferenceConfidence whether we should add a <NON_REF> alternative allele to the result variation contexts. + * + * @return A CalledHaplotypes object containing a list of VC's with genotyped events and called haplotypes + * + */ +// @Requires({"refLoc.containsP(activeRegionWindow)", "haplotypes.size() > 0"}) + @Ensures("result != null") + // TODO - can this be refactored? this is hard to follow! + public HaplotypeCallerGenotypingEngine.CalledHaplotypes callMutations ( + final List haplotypes, + //final Map haplotypeReadMap, + final ReadLikelihoods readLikelihoods, + final Map originalNormalReadQualities, + final Map> perSampleFilteredReadList, + final byte[] ref, + final GenomeLoc refLoc, + final GenomeLoc activeRegionWindow, + final GenomeLocParser genomeLocParser, + final RefMetaDataTracker tracker, + final List activeAllelesToGenotype, + final boolean emitReferenceConfidence, + final String tumorSampleName, + final String matchedNormalSampleName, + final RodBinding dbsnpRod, + final List> cosmicRod, + final String DEBUG_READ_NAME + + ) { + + // sanity check input arguments + if (haplotypes == null || haplotypes.isEmpty()) throw new IllegalArgumentException("haplotypes input should be non-empty and non-null, got "+haplotypes); + if (readLikelihoods == null || readLikelihoods.sampleCount() == 0) throw new IllegalArgumentException("readLikelihoods input should be non-empty and non-null, got "+readLikelihoods); + if (ref == null || ref.length == 0 ) throw new IllegalArgumentException("ref bytes input should be non-empty and non-null, got "+ref); + if (refLoc == null || refLoc.size() != ref.length) throw new IllegalArgumentException(" refLoc must be non-null and length must match ref bytes, got "+refLoc); + if (activeRegionWindow == null ) throw new IllegalArgumentException("activeRegionWindow must be non-null, got "+activeRegionWindow); + if (activeAllelesToGenotype == null ) throw new IllegalArgumentException("activeAllelesToGenotype must be non-null, got "+activeAllelesToGenotype); + if (genomeLocParser == null ) throw new IllegalArgumentException("genomeLocParser must be non-null, got "+genomeLocParser); + + + // Somatic Tumor/Normal Sample Handling + verifySamplePresence(tumorSampleName, readLikelihoods.samples()); + final boolean hasNormal = (matchedNormalSampleName != null); + + // update the haplotypes so we're ready to call, getting the ordered list of positions on the reference + // that carry events among the haplotypes + final TreeSet startPosKeySet = decomposeHaplotypesIntoVariantContexts(haplotypes, readLikelihoods, ref, refLoc, activeAllelesToGenotype); + + // Walk along each position in the key set and create each event to be outputted + final Set calledHaplotypes = new HashSet<>(); + final List returnCalls = new ArrayList<>(); + + for( final int loc : startPosKeySet ) { + if( loc >= activeRegionWindow.getStart() && loc <= activeRegionWindow.getStop() ) { // genotyping an event inside this active region + final List eventsAtThisLoc = getVCsAtThisLocation(haplotypes, loc, activeAllelesToGenotype); + + if( eventsAtThisLoc.isEmpty() ) { continue; } + + // Create the event mapping object which maps the original haplotype events to the events present at just this locus + final Map> eventMapper = createEventMapper(loc, eventsAtThisLoc, haplotypes); + + // Sanity check the priority list for mistakes + final List priorityList = makePriorityList(eventsAtThisLoc); + + // Merge the event to find a common reference representation + + VariantContext mergedVC = GATKVariantContextUtils.simpleMerge(eventsAtThisLoc, priorityList, + GATKVariantContextUtils.FilteredRecordMergeType.KEEP_IF_ANY_UNFILTERED, + GATKVariantContextUtils.GenotypeMergeType.PRIORITIZE, false, false, null, false, false); + + if( mergedVC == null ) { continue; } + + final int numAlts = mergedVC.getNAlleles()-1; + +// final VariantContextBuilder vcb = new VariantContextBuilder(mergedVC); + + final GenotypeLikelihoodsCalculationModel.Model calculationModel = mergedVC.isSNP() + ? GenotypeLikelihoodsCalculationModel.Model.SNP : GenotypeLikelihoodsCalculationModel.Model.INDEL; + + if (emitReferenceConfidence) + mergedVC = addNonRefSymbolicAllele(mergedVC); + + final Map mergeMap = new LinkedHashMap<>(); + mergeMap.put(null, mergedVC.getReference()); // the reference event (null) --> the reference allele + for(int iii = 0; iii < eventsAtThisLoc.size(); iii++) { + mergeMap.put(eventsAtThisLoc.get(iii), mergedVC.getAlternateAllele(iii)); // BUGBUG: This is assuming that the order of alleles is the same as the priority list given to simpleMerge function + } + + final Map> alleleMapper = createAlleleMapper(mergeMap, eventMapper); + + if( configuration.DEBUG && logger != null ) { + if (logger != null) logger.info("Genotyping event at " + loc + " with alleles = " + mergedVC.getAlleles()); + } + + ReadLikelihoods readAlleleLikelihoods = readLikelihoods.marginalize(alleleMapper, genomeLocParser.createPaddedGenomeLoc(genomeLocParser.createGenomeLoc(mergedVC), ALLELE_EXTENSION)); + + //LDG: do we want to do this before or after pulling out overlapping reads? + if (MTAC.isSampleContaminationPresent()) + readAlleleLikelihoods.contaminationDownsampling(MTAC.getSampleContamination()); + + //if (!mergedVC.isBiallelic()) { + // logger.info("[UNSUPPORTED] Detected non-Biallelic VC" + mergedVC.toString()); + // continue; + //} + + // TODO: once tests are passing, refactor to use the new data structure (not the deprecated one) + // handle overlapping fragments + // TODO: CONFIRM WITH GSA IF IT IS OK TO REMOVE READS FROM THE PRALM (should be... they do it in filterPoorlyModeledReads!) + PerReadAlleleLikelihoodMap tumorPRALM = readAlleleLikelihoods.toPerReadAlleleLikelihoodMap(readAlleleLikelihoods.sampleIndex(tumorSampleName)); + filterPRALMForOverlappingReads(tumorPRALM, mergedVC.getReference(), loc, false); + MuTect2.logReadInfo(DEBUG_READ_NAME, tumorPRALM.getLikelihoodReadMap().keySet(), "Present after filtering for overlapping reads"); + // extend to multiple samples + + //handle existence of secondary alts + double[] afs = estimateAlleleFraction(mergedVC, tumorPRALM); + + if( configuration.DEBUG && logger != null ) { + String output = "Calculated allelic fraction at " + loc + " = "; + for (int i = 0; i originalNormalMQs, double[] afs) { + double[] genotypeLikelihoods = new double[mergedVC.getNAlleles()]; + for(Map.Entry> e : tumorPRALM.getLikelihoodReadMap().entrySet()) { + Map m = e.getValue(); + Double refLL = m.get(mergedVC.getReference()); + if (originalNormalMQs.get(e.getKey().getReadName()) != 0) { + genotypeLikelihoods[0] += Math.log10(Math.pow(10, refLL)); + + for (int altInd = 0; altInd < mergedVC.getNAlleles()-1; altInd++) { + Double altLL = m.get(mergedVC.getAlternateAllele(altInd)); + genotypeLikelihoods[altInd+1] += Math.log10(Math.pow(10, refLL) * (1 - afs[altInd]) + Math.pow(10, altLL) * afs[altInd]); + } + } + } + return genotypeLikelihoods; + } + + /** + * Find the allele fractions for each alternate allele + * + * @param vc input VC, for alleles + * @param map read likelihoods + * @return estimated AF for each alt + */ + // FIXME: calculate using the uncertainty rather than this cheap approach + private double[] estimateAlleleFraction(VariantContext vc, PerReadAlleleLikelihoodMap map) { + int[] counts = getRefAltCount(vc, map); + int numAlts = vc.getNAlleles()-1; + double[] afs = new double[numAlts]; + int refCount = counts[0]; + int altCount; + + for(int altInd = 0; altInd < numAlts; altInd++) { + altCount = counts[altInd+1]; + afs[altInd] = (double) altCount / ((double) refCount + (double) altCount); + //logger.info("Counted " + refCount + " ref and " + altCount + " alt " ); + } + + return afs; + } + + /** + * Evalutate the most likely allele for each read, if it is in fact informative + * + * @param mergedVC input VC, for alleles + * @param afMap read likelihoods + * @return an array giving the read counts for the ref and each alt allele + */ + // TODO: ensure there are only two alleles in the VC + private int[] getRefAltCount(VariantContext mergedVC, PerReadAlleleLikelihoodMap afMap) { + int counts[] = new int[mergedVC.getNAlleles()]; + int REF = 0; + + for(Map.Entry> e : afMap.getLikelihoodReadMap().entrySet()) { + Map m = e.getValue(); + Double rl = m.get(mergedVC.getReference()); + for(int altInd=0; altInd= rl) logger.info("Alt found in " + e.getKey().getReadName()); + } + return counts; + } + + + private void logM2Debug(String s) { + if (MTAC.M2_DEBUG) { + logger.info(s); + } + } + + // would have used org.broadinstitute.sting.utils.genotyper.PerReadAlleleLikelihoodMap.getMostLikelyAllele but we have this case where + // there is a read that doesn't overlap the variant site, and thus supports both alleles equally. + private boolean arePairHMMLikelihoodsInformative(double l1, double l2) { + // TODO: should this be parameterized, or simply encoded + double EPSILON = 0.1; + return (Math.abs(l1 - l2) >= EPSILON); + } + + private void filterPRALMForOverlappingReads(PerReadAlleleLikelihoodMap pralm, Allele ref, int location, boolean retainMismatches) { + + Map> m = pralm.getLikelihoodReadMap(); + + + // iterate through the reads, if the name has been seen before we have overlapping (potentially) fragments, so handle them + Map nameToRead = new HashMap<>(); + Set readsToKeep = new HashSet<>(); + + for(GATKSAMRecord rec : m.keySet()) { + // if we haven't seen it... just record the name and add it to the list of reads to keep + GATKSAMRecord existing = nameToRead.get(rec.getReadName()); + if (existing == null) { + nameToRead.put(rec.getReadName(), rec); + readsToKeep.add(rec); + } else { + logM2Debug("Found a paired read for " + rec.getReadName()); + + // NOTE: Can we use FragmentUtils to do all of this processing (to find overlapping pairs?) + // seems like maybe, but it has some requirements about the order of the reads supplied which may be painful to meet + // TODO: CHECK IF THE READS BOTH OVERLAP THE POSITION!!!! + if ( ReadUtils.isInsideRead(existing, location) && ReadUtils.isInsideRead(rec, location) ) { + + MostLikelyAllele existingMLA = pralm.getMostLikelyAllele(pralm.getLikelihoodReadMap().get(existing)); + Allele existingAllele = existingMLA.getMostLikelyAllele(); + + MostLikelyAllele recMLA = pralm.getMostLikelyAllele(pralm.getLikelihoodReadMap().get(rec)); + Allele recAllele = recMLA.getMostLikelyAllele(); + + // if the reads disagree at this position... + if (!existingAllele.equals(recAllele)) { + //... and we're not retaining mismatches, throw them both out + if (!retainMismatches) { + logM2Debug("Discarding read-pair due to disagreement" + rec.getReadName() + " and allele " + existingAllele); + readsToKeep.remove(existing); + + //... and we are retaining mismatches, keep the mismatching one + } else { + if (existingAllele.equals(ref)) { + logM2Debug("Discarding read to keep mismatching " + rec.getReadName() + " and allele " + existingAllele); + readsToKeep.remove(existing); + readsToKeep.add(rec); + } + } + // Otherwise, keep the element with the higher quality score + } else { + logM2Debug("Discarding lower quality read of overlapping pair " + rec.getReadName() + " and allele " + existingAllele); + if (existingMLA.getLog10LikelihoodOfMostLikely() < recMLA.getLog10LikelihoodOfMostLikely()) { + readsToKeep.remove(existing); + readsToKeep.add(rec); + } + } + } else { + // although these are overlapping fragments, they don't overlap at the position in question + // so keep the read + readsToKeep.add(rec); + } + } + + } + + // perhaps moved into PRALM + final Iterator>> it = m.entrySet().iterator(); + while ( it.hasNext() ) { + final Map.Entry> record = it.next(); + if(!readsToKeep.contains(record.getKey())) { + it.remove(); + logM2Debug("Dropping read " + record.getKey() + " due to overlapping read fragment rules"); + } + } + } + + // Move to utility class so we can use one shared with HaplotypeCallerGenotypingEngine + private VariantContext addNonRefSymbolicAllele(final VariantContext mergedVC) { + final VariantContextBuilder vcb = new VariantContextBuilder(mergedVC); + final List originalList = mergedVC.getAlleles(); + final List alleleList = new ArrayList<>(originalList.size() + 1); + alleleList.addAll(mergedVC.getAlleles()); + alleleList.add(GATKVCFConstants.NON_REF_SYMBOLIC_ALLELE); + vcb.alleles(alleleList); + return vcb.make(); + } + +} diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/create_M2_pon.scala b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/create_M2_pon.scala new file mode 100644 index 000000000..009eebaca --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/create_M2_pon.scala @@ -0,0 +1,140 @@ +/* +* By downloading the PROGRAM you agree to the following terms of use: +* +* BROAD INSTITUTE +* SOFTWARE LICENSE AGREEMENT +* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY +* +* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”). +* +* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and +* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions. +* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows: +* +* 1. DEFINITIONS +* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE. +* +* 2. LICENSE +* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute. LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation. +* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement. +* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement. +* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM. +* +* 3. PHONE-HOME FEATURE +* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM. Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time. Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation. +* +* 4. OWNERSHIP OF INTELLECTUAL PROPERTY +* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication. +* Copyright 2012-2015 Broad Institute, Inc. +* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc. +* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes. +* +* 5. INDEMNIFICATION +* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement. +* +* 6. NO REPRESENTATIONS OR WARRANTIES +* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME. +* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING. +* +* 7. ASSIGNMENT +* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void. +* +* 8. MISCELLANEOUS +* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries. +* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes. +* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4. +* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt. +* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter. +* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement. +* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles. +*/ + +package org.broadinstitute.gatk.tools.walkers.cancer.m2 + +import java.io.File + +import org.broadinstitute.gatk.queue.QScript +import org.broadinstitute.gatk.queue.extensions.gatk._ +import org.broadinstitute.gatk.queue.function.CommandLineFunction +import org.broadinstitute.gatk.queue.util.QScriptUtils +import org.broadinstitute.gatk.utils.commandline.{Input, Output} +import org.broadinstitute.gatk.utils.variant.GATKVariantContextUtils.FilteredRecordMergeType + +import scala.collection.mutable.ListBuffer + +class create_M2_pon extends QScript { + + @Argument(shortName = "bams", required = true, doc = "file of all BAM files") + var allBams: String = "" + + @Argument(shortName = "o", required = true, doc = "Output prefix") + var outputPrefix: String = "" + + @Argument(shortName = "minN", required = false, doc = "minimum number of sample observations to include in PON") + var minN: Int = 2 + + @Argument(doc="Reference fasta file to process with", fullName="reference", shortName="R", required=false) + var reference = new File("/seq/references/Homo_sapiens_assembly19/v1/Homo_sapiens_assembly19.fasta") + + @Argument(doc="Intervals file to process with", fullName="intervals", shortName="L", required=true) + var intervals : File = "" + + @Argument(shortName = "sc", required = false, doc = "base scatter count") + var scatter: Int = 10 + + + def script() { + val bams = QScriptUtils.createSeqFromFile(allBams) + val genotypesVcf = outputPrefix + ".genotypes.vcf" + val finalVcf = outputPrefix + ".vcf" + + val perSampleVcfs = new ListBuffer[File]() + for (bam <- bams) { + val outputVcf = "sample-vcfs/" + bam.getName + ".vcf" + add( createM2Config(bam, outputVcf)) + perSampleVcfs += outputVcf + } + + val cv = new CombineVariants() + cv.reference_sequence = reference + cv.memoryLimit = 2 + cv.setKey = "null" + cv.minimumN = minN + cv.memoryLimit = 16 + cv.filteredrecordsmergetype = FilteredRecordMergeType.KEEP_IF_ANY_UNFILTERED + cv.filteredAreUncalled = true + cv.variant = perSampleVcfs + cv.out = genotypesVcf + + // using this instead of "sites_only" because we want to keep the AC info + val vc = new VcfCutter() + vc.inVcf = genotypesVcf + vc.outVcf = finalVcf + + add (cv, vc) + + } + + + def createM2Config(bam : File, outputVcf : File): org.broadinstitute.gatk.queue.extensions.gatk.MuTect2 = { + val mutect2 = new org.broadinstitute.gatk.queue.extensions.gatk.MuTect2 + + mutect2.reference_sequence = reference + mutect2.artifact_detection_mode = true + mutect2.intervalsString :+= intervals + mutect2.memoryLimit = 2 + mutect2.input_file = List(new TaggedFile(bam, "tumor")) + + mutect2.scatterCount = scatter + mutect2.out = outputVcf + + mutect2 + } +} + +class VcfCutter extends CommandLineFunction { + @Input(doc = "vcf to cut") var inVcf: File = _ + @Output(doc = "output vcf") var outVcf: File = _ + + def commandLine = "cat %s | cut -f1-8 > %s".format(inVcf, outVcf) +} \ No newline at end of file diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/run_M2_ICE_NN.scala b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/run_M2_ICE_NN.scala new file mode 100644 index 000000000..46d31c461 --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/run_M2_ICE_NN.scala @@ -0,0 +1,102 @@ +/* +* By downloading the PROGRAM you agree to the following terms of use: +* +* BROAD INSTITUTE +* SOFTWARE LICENSE AGREEMENT +* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY +* +* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”). +* +* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and +* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions. +* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows: +* +* 1. DEFINITIONS +* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE. +* +* 2. LICENSE +* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute. LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation. +* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement. +* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement. +* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM. +* +* 3. PHONE-HOME FEATURE +* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM. Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time. Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation. +* +* 4. OWNERSHIP OF INTELLECTUAL PROPERTY +* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication. +* Copyright 2012-2015 Broad Institute, Inc. +* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc. +* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes. +* +* 5. INDEMNIFICATION +* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement. +* +* 6. NO REPRESENTATIONS OR WARRANTIES +* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME. +* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING. +* +* 7. ASSIGNMENT +* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void. +* +* 8. MISCELLANEOUS +* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries. +* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes. +* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4. +* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt. +* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter. +* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement. +* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles. +*/ + +package org.broadinstitute.gatk.queue.qscripts.dev + +import org.broadinstitute.gatk.queue.QScript +import org.broadinstitute.gatk.queue.extensions.gatk._ +import org.broadinstitute.gatk.queue.util.QScriptUtils + +class run_M2_ICE_NN extends QScript { + + @Argument(shortName = "bams", required = true, doc = "file of all BAM files") + var allBams: String = "" + + @Argument(shortName = "o", required = false, doc = "Output prefix") + var outputPrefix: String = "" + + @Argument(shortName = "pon", required = false, doc = "Normal PON") + var panelOfNormals: String = "/dsde/working/mutect/panel_of_normals/panel_of_normals_m2_ice_wgs_territory/m2_406_ice_normals_wgs_calling_regions.vcf"; + + @Argument(shortName = "sc", required = false, doc = "base scatter count") + var scatter: Int = 10 + + + def script() { + val bams = QScriptUtils.createSeqFromFile(allBams) + + for (tumor <- bams) { + for (normal <- bams) { + if (tumor != normal) add( createM2Config(tumor, normal, new File(panelOfNormals), outputPrefix)) + } + } + } + + + def createM2Config(tumorBAM : File, normalBAM : File, panelOfNormals : File, outputPrefix : String): M2 = { + val mutect2 = new MuTect2 + + mutect2.reference_sequence = new File("/seq/references/Homo_sapiens_assembly19/v1/Homo_sapiens_assembly19.fasta") + mutect2.cosmic :+= new File("/xchip/cga/reference/hg19/hg19_cosmic_v54_120711.vcf") + mutect2.dbsnp = new File("/humgen/gsa-hpprojects/GATK/bundle/current/b37/dbsnp_138.b37.vcf") + mutect2.normal_panel :+= panelOfNormals + + mutect2.intervalsString :+= new File("/dsde/working/mutect/crsp_nn/whole_exome_illumina_coding_v1.Homo_sapiens_assembly19.targets.no_empty.interval_list") + mutect2.memoryLimit = 2 + mutect2.input_file = List(new TaggedFile(normalBAM, "normal"), new TaggedFile(tumorBAM, "tumor")) + + mutect2.scatterCount = scatter + mutect2.out = outputPrefix + tumorBAM.getName + "-vs-" + normalBAM.getName + ".vcf" + + println("Adding " + tumorBAM + " vs " + normalBAM + " as " + mutect2.out) + mutect2 + } +} diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/run_M2_dream.scala b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/run_M2_dream.scala new file mode 100644 index 000000000..95b5c0043 --- /dev/null +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/run_M2_dream.scala @@ -0,0 +1,89 @@ +/* +* By downloading the PROGRAM you agree to the following terms of use: +* +* BROAD INSTITUTE +* SOFTWARE LICENSE AGREEMENT +* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY +* +* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”). +* +* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and +* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions. +* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows: +* +* 1. DEFINITIONS +* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE. +* +* 2. LICENSE +* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute. LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation. +* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement. +* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement. +* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM. +* +* 3. PHONE-HOME FEATURE +* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM. Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time. Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation. +* +* 4. OWNERSHIP OF INTELLECTUAL PROPERTY +* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication. +* Copyright 2012-2015 Broad Institute, Inc. +* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc. +* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes. +* +* 5. INDEMNIFICATION +* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement. +* +* 6. NO REPRESENTATIONS OR WARRANTIES +* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME. +* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING. +* +* 7. ASSIGNMENT +* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void. +* +* 8. MISCELLANEOUS +* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries. +* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes. +* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4. +* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt. +* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter. +* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement. +* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles. +*/ + +package org.broadinstitute.gatk.queue.qscripts.dev + +import org.broadinstitute.gatk.queue.QScript +import org.broadinstitute.gatk.queue.extensions.gatk._ + +class run_M2_dream extends QScript { + + @Argument(shortName = "L", required=false, doc = "Intervals file") + var intervalsFile: List[File] = Nil + @Argument(shortName = "normal", required=true, doc = "Normal sample BAM") + var normalBAM: String = "" + @Argument(shortName = "tumor", required=true, doc = "Tumor sample BAM") + var tumorBAM: String = "" + @Argument(shortName = "o", required=true, doc = "Output file") + var outputFile: String = "" + @Argument(shortName = "sc", required=false, doc = "base scatter count") + var scatter: Int = 10 + + + def script() { + + val mutect2 = new MuTect2 + + mutect2.reference_sequence = new File("/seq/references/Homo_sapiens_assembly19/v1/Homo_sapiens_assembly19.fasta") + mutect2.cosmic :+= new File("/xchip/cga/reference/hg19/hg19_cosmic_v54_120711.vcf") + mutect2.dbsnp = new File("/humgen/gsa-hpprojects/GATK/bundle/current/b37/dbsnp_138.b37.vcf") + mutect2.normal_panel :+= new File("/xchip/cga/reference/hg19/wgs_hg19_125_cancer_blood_normal_panel.vcf") + + mutect2.intervalsString = intervalsFile + mutect2.memoryLimit = 2 + mutect2.input_file = List(new TaggedFile(normalBAM, "normal"), new TaggedFile(tumorBAM, "tumor")) + + mutect2.scatterCount = scatter + mutect2.out = outputFile + add(mutect2) + } + +} diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/haplotypecaller/HaplotypeCaller.java b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/haplotypecaller/HaplotypeCaller.java index fe7d7c3ae..7b4399068 100644 --- a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/haplotypecaller/HaplotypeCaller.java +++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/tools/walkers/haplotypecaller/HaplotypeCaller.java @@ -108,9 +108,17 @@ import java.io.FileNotFoundException; import java.util.*; /** - * Call SNPs and indels simultaneously via local assembly of haplotypes in an active region + * Call germline SNPs and indels via local re-assembly of haplotypes * - *

The basic operation of the HaplotypeCaller proceeds as follows:

+ *

The HaplotypeCaller is capable of calling SNPs and indels simultaneously via local de-novo assembly of haplotypes in an active region. In other words, whenever the program encounters a region showing signs of variation, it discards the existing mapping information and completely reassembles the reads in that region. This allows the HaplotypeCaller to be more accurate when calling regions that are traditionally difficult to call, for example when they contain different types of variants close to each other. It also makes the HaplotypeCaller much better at calling indels than position-based callers like UnifiedGenotyper.

+ +

In the so-called GVCF mode used for scalable variant calling in DNA sequence data, HaplotypeCaller runs per-sample to generate an intermediate genomic gVCF (gVCF), which can then be used for joint genotyping of multiple samples in a very efficient way, which enables rapid incremental processing of samples as they roll off the sequencer, as well as scaling to very large cohort sizes (e.g. the 92K exomes of ExAC).

+ +

In addition, HaplotypeCaller is able to handle non-diploid organisms as well as pooled experiment data. Note however that the algorithms used to calculate variant likelihoods is not well suited to extreme allele frequencies (relative to ploidy) so its use is not recommended for somatic (cancer) variant discovery. For that purpose, use MuTect2 instead.

+ +

Finally, HaplotypeCaller is also able to correctly handle the splice junctions that make RNAseq a challenge for most variant callers.

+ * + *

How HaplotypeCaller works

* *
*

1. Define active regions

@@ -159,16 +167,13 @@ import java.util.*; * Best Practices documentation for detailed recommendations.

* *
- *

Single-sample all-sites calling on DNAseq (for `-ERC GVCF` cohort analysis workflow)

+ *

Single-sample GVCF calling on DNAseq (for `-ERC GVCF` cohort analysis workflow)

* 
- *   java
- *     -jar GenomeAnalysisTK.jar
- *     -T HaplotypeCaller
- *     -R reference.fasta
+ *   java -jar GenomeAnalysisTK.jar \
+ *     -R reference.fasta \
+ *     -T HaplotypeCaller \
  *     -I sample1.bam \
  *     --emitRefConfidence GVCF \
- *     --variant_index_type LINEAR \
- *     --variant_index_parameter 128000
  *     [--dbsnp dbSNP.vcf] \
  *     [-L targets.interval_list] \
  *     -o output.raw.snps.indels.g.vcf
@@ -176,10 +181,9 @@ import java.util.*;
  *
  * 
Variant-only calling on DNAseq

  * 
- *   java
- *     -jar GenomeAnalysisTK.jar
- *     -T HaplotypeCaller
- *     -R reference.fasta
+ *   java -jar GenomeAnalysisTK.jar \
+ *     -R reference.fasta \
+ *     -T HaplotypeCaller \
  *     -I sample1.bam [-I sample2.bam ...] \
  *     [--dbsnp dbSNP.vcf] \
  *     [-stand_call_conf 30] \
@@ -190,10 +194,9 @@ import java.util.*;
  *
  * 
Variant-only calling on RNAseq

  * 
- *   java
- *     -jar GenomeAnalysisTK.jar
- *     -T HaplotypeCaller
- *     -R reference.fasta
+ *   java -jar GenomeAnalysisTK.jar \
+ *     -R reference.fasta \
+ *     -T HaplotypeCaller \
  *     -I sample1.bam \
  *     [--dbsnp dbSNP.vcf] \
  *     -stand_call_conf 20 \
@@ -210,8 +213,7 @@ import java.util.*;
  * 
  *
  * 
Special note on ploidy

- * 
This tool is able to handle almost any ploidy (except very high ploidies in large pooled experiments); the ploidy
- * can be specified using the -ploidy argument for non-diploid organisms.

+ * 
This tool is able to handle almost any ploidy (except very high ploidies in large pooled experiments); the ploidy can be specified using the -ploidy argument for non-diploid organisms.

  *
  * 
Additional Notes

  * 
    
diff --git a/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/utils/cancer/TestingReadUtils.java b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/utils/cancer/TestingReadUtils.java
new file mode 100644
index 000000000..bb8105ec1
--- /dev/null
+++ b/protected/gatk-tools-protected/src/main/java/org/broadinstitute/gatk/utils/cancer/TestingReadUtils.java
@@ -0,0 +1,110 @@
+/*
+* By downloading the PROGRAM you agree to the following terms of use:
+* 
+* BROAD INSTITUTE
+* SOFTWARE LICENSE AGREEMENT
+* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY
+* 
+* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”).
+* 
+* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and
+* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions.
+* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows:
+* 
+* 1. DEFINITIONS
+* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE.
+* 
+* 2. LICENSE
+* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute.  LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation.
+* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement.
+* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement.
+* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM.
+* 
+* 3. PHONE-HOME FEATURE
+* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM.  Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time.  Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation.
+* 
+* 4. OWNERSHIP OF INTELLECTUAL PROPERTY
+* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication.
+* Copyright 2012-2015 Broad Institute, Inc.
+* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc.
+* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes.
+* 
+* 5. INDEMNIFICATION
+* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement.
+* 
+* 6. NO REPRESENTATIONS OR WARRANTIES
+* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME.
+* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING.
+* 
+* 7. ASSIGNMENT
+* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void.
+* 
+* 8. MISCELLANEOUS
+* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries.
+* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes.
+* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4.
+* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt.
+* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter.
+* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement.
+* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles.
+*/
+
+package org.broadinstitute.gatk.utils.cancer;
+
+import htsjdk.samtools.SAMFileHeader;
+import htsjdk.samtools.SAMReadGroupRecord;
+import org.broadinstitute.gatk.utils.contexts.AlignmentContext;
+import org.broadinstitute.gatk.utils.GenomeLoc;
+import org.broadinstitute.gatk.utils.GenomeLocParser;
+import org.broadinstitute.gatk.utils.pileup.ReadBackedPileup;
+import org.broadinstitute.gatk.utils.pileup.ReadBackedPileupImpl;
+import org.broadinstitute.gatk.utils.sam.ArtificialSAMUtils;
+import org.broadinstitute.gatk.utils.sam.GATKSAMReadGroupRecord;
+import org.broadinstitute.gatk.utils.sam.GATKSAMRecord;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Created by IntelliJ IDEA.
+ * User: aaron
+ * Date: 2/18/12
+ * Time: 9:33 PM
+ * To change this template use File | Settings | File Templates.
+ */
+public class TestingReadUtils {
+    /**
+     * generates an alignment context with the following parameters:
+     * @param totalReads the total number of reads.  total - targetReads with have all 'A' bases
+     * @param targetReads number of reads in the pileup to contain 'T'
+     * @param firstReadGroup the read group for the 'A' reads
+     * @param secondReadGroup the read group for the 'T' reads
+     * @return an alignment context with the reads filled in according to the pattern above.  The pileup will be located at the
+     * first base of the first sequence entry
+     */
+    public static final AlignmentContext generateAlignmentContext(int totalReads, int targetReads, SAMFileHeader header, GenomeLocParser parser, String firstReadGroup, String secondReadGroup) {
+        List reads = new ArrayList();
+
+        // filled with A bases
+        for (int i = 0; i < totalReads - targetReads; i++) {
+            reads.add(addReadGroup(ArtificialSAMUtils.createArtificialRead(header, "Read" + i, 0, 1, 50),firstReadGroup));
+        }
+        byte bases[] = new byte[50];
+        byte quals[] = new byte[50];
+        for (int i = 0; i < 50; i++) {bases[i] = 'T'; quals[i] = 30;}
+        for (int i = totalReads - targetReads; i < totalReads; i++) {
+            reads.add(addReadGroup(ArtificialSAMUtils.createArtificialRead(header, "Read" + i, 0, 1, bases, quals),secondReadGroup));
+        }
+        GenomeLoc loc = parser.createGenomeLoc(header.getSequenceDictionary().getSequence(0).getSequenceName(),1,1);
+        ReadBackedPileup rbp = new ReadBackedPileupImpl(loc, reads, 0);
+        return new AlignmentContext(loc,rbp);
+    }
+
+    public static final GATKSAMRecord addReadGroup(GATKSAMRecord rec, String readGroupName) {
+        GATKSAMReadGroupRecord rgRec = new GATKSAMReadGroupRecord(readGroupName);
+        rgRec.setSample("sample");
+        rec.setReadGroup(rgRec);
+        return rec;
+    }
+    
+}
diff --git a/protected/gatk-tools-protected/src/test/java/org/broadinstitute/gatk/tools/walkers/cancer/BaseQualitySumPerAlleleBySampleUnitTest.java b/protected/gatk-tools-protected/src/test/java/org/broadinstitute/gatk/tools/walkers/cancer/BaseQualitySumPerAlleleBySampleUnitTest.java
new file mode 100644
index 000000000..f507c8b47
--- /dev/null
+++ b/protected/gatk-tools-protected/src/test/java/org/broadinstitute/gatk/tools/walkers/cancer/BaseQualitySumPerAlleleBySampleUnitTest.java
@@ -0,0 +1,80 @@
+/*
+* By downloading the PROGRAM you agree to the following terms of use:
+* 
+* BROAD INSTITUTE
+* SOFTWARE LICENSE AGREEMENT
+* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY
+* 
+* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”).
+* 
+* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and
+* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions.
+* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows:
+* 
+* 1. DEFINITIONS
+* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE.
+* 
+* 2. LICENSE
+* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute.  LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation.
+* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement.
+* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement.
+* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM.
+* 
+* 3. PHONE-HOME FEATURE
+* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM.  Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time.  Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation.
+* 
+* 4. OWNERSHIP OF INTELLECTUAL PROPERTY
+* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication.
+* Copyright 2012-2015 Broad Institute, Inc.
+* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc.
+* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes.
+* 
+* 5. INDEMNIFICATION
+* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement.
+* 
+* 6. NO REPRESENTATIONS OR WARRANTIES
+* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME.
+* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING.
+* 
+* 7. ASSIGNMENT
+* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void.
+* 
+* 8. MISCELLANEOUS
+* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries.
+* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes.
+* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4.
+* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt.
+* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter.
+* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement.
+* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles.
+*/
+
+package org.broadinstitute.gatk.tools.walkers.cancer;
+
+import htsjdk.samtools.SAMFileHeader;
+import org.broadinstitute.gatk.utils.QualityUtils;
+import org.broadinstitute.gatk.utils.sam.ArtificialSAMUtils;
+import org.broadinstitute.gatk.utils.sam.GATKSAMRecord;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+public class BaseQualitySumPerAlleleBySampleUnitTest {
+    @Test
+    public void BasicTest() {
+        BaseQualitySumPerAlleleBySample a = new BaseQualitySumPerAlleleBySample();
+
+        final SAMFileHeader header = ArtificialSAMUtils.createArtificialSamHeader(5, 1, 10000);
+        final GATKSAMRecord read = ArtificialSAMUtils.createArtificialRead(header, "myRead", 0, 1, 76);
+
+        read.setMappingQuality(60);
+        Assert.assertTrue(a.isUsableRead(read));
+
+        read.setMappingQuality(0);
+        Assert.assertFalse(a.isUsableRead(read));
+
+        read.setMappingQuality(QualityUtils.MAPPING_QUALITY_UNAVAILABLE);
+        Assert.assertFalse(a.isUsableRead(read));
+
+    }
+    
+}
diff --git a/protected/gatk-tools-protected/src/test/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContEstIntegrationTest.java b/protected/gatk-tools-protected/src/test/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContEstIntegrationTest.java
new file mode 100644
index 000000000..ac1edfaa9
--- /dev/null
+++ b/protected/gatk-tools-protected/src/test/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContEstIntegrationTest.java
@@ -0,0 +1,103 @@
+/*
+* By downloading the PROGRAM you agree to the following terms of use:
+* 
+* BROAD INSTITUTE
+* SOFTWARE LICENSE AGREEMENT
+* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY
+* 
+* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”).
+* 
+* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and
+* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions.
+* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows:
+* 
+* 1. DEFINITIONS
+* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE.
+* 
+* 2. LICENSE
+* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute.  LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation.
+* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement.
+* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement.
+* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM.
+* 
+* 3. PHONE-HOME FEATURE
+* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM.  Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time.  Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation.
+* 
+* 4. OWNERSHIP OF INTELLECTUAL PROPERTY
+* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication.
+* Copyright 2012-2015 Broad Institute, Inc.
+* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc.
+* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes.
+* 
+* 5. INDEMNIFICATION
+* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement.
+* 
+* 6. NO REPRESENTATIONS OR WARRANTIES
+* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME.
+* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING.
+* 
+* 7. ASSIGNMENT
+* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void.
+* 
+* 8. MISCELLANEOUS
+* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries.
+* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes.
+* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4.
+* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt.
+* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter.
+* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement.
+* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles.
+*/
+
+package org.broadinstitute.gatk.tools.walkers.cancer.contamination;
+
+import org.broadinstitute.gatk.engine.walkers.WalkerTest;
+import org.testng.annotations.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Test ContEst with and without a "genotyping array" VCF
+ *
+ * @author gauthier
+ */
+public class ContEstIntegrationTest extends WalkerTest {
+    public static final String DREAMBamsDirectory = new String("/dsde/working/mutect/dream_smc/bams/");
+    public static final String ContaminatedBamsDirectory = new String("/dsde/working/mutect/contamination/bams/");
+    public static final String NormalBamsDirectory = new String("/humgen/gsa-hpprojects/NA12878Collection/bams/crsp_ice_validation/");
+    public static final String ICEexomeIntervals = new String("/seq/references/HybSelOligos/HybSelOligos/whole_exome_illumina_coding_v1/whole_exome_illumina_coding_v1.Homo_sapiens_assembly19.targets.interval_list");
+
+    @Test(enabled = true)
+    public void testWithArray() {
+        List md5sums = Arrays.asList("d41d8cd98f00b204e9800998ecf8427e");
+        WalkerTest.WalkerTestSpec spec = new WalkerTest.WalkerTestSpec(
+                "-T ContEst" +
+                " -I " + DREAMBamsDirectory + "synthetic.challenge.set4.tumor.bam" +
+                " -R " + b37KGReference +
+                " --popfile " + validationDataLocation + "cancer/hg19_population_stratified_af_hapmap_3.3.fixed.vcf" +
+                " --genotypes " + validationDataLocation + "cancer/Dream.set3.tumorGTs.vcf" +
+                " -L " + validationDataLocation + "cancer/SNP6.hg19.interval_list" +
+                " -L " + ICEexomeIntervals +
+                " -L 1" +
+                " -isr INTERSECTION",md5sums);
+        executeTest("testWithArray",spec);
+    }
+
+    @Test(enabled = true)
+    public void testArrayFree(){
+        List md5sums = Arrays.asList("d41d8cd98f00b204e9800998ecf8427e");
+        WalkerTest.WalkerTestSpec spec = new WalkerTest.WalkerTestSpec(
+                "-T ContEst" +
+                        " -I:eval " + ContaminatedBamsDirectory + "HCC1143_BL.small.0.05.contaminated.with.SM-612V3.small.0.95.bam" +
+                        " -I:genotype " + NormalBamsDirectory + "SM-612V4.bam" +
+                        " -R " + b37KGReference +
+                        " --popfile " + validationDataLocation + "hg19_population_stratified_af_hapmap_3.3.fixed.vcf" +
+                        " -L " + validationDataLocation + "SNP6.hg19.interval_list" +
+                        " -L " + ICEexomeIntervals +
+                        " -L 1" +
+                        " -isr INTERSECTION",md5sums);
+        executeTest("testArrayFree",spec);
+    }
+}
diff --git a/protected/gatk-tools-protected/src/test/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContEstUnitTest.java b/protected/gatk-tools-protected/src/test/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContEstUnitTest.java
new file mode 100644
index 000000000..9b8990a08
--- /dev/null
+++ b/protected/gatk-tools-protected/src/test/java/org/broadinstitute/gatk/tools/walkers/cancer/contamination/ContEstUnitTest.java
@@ -0,0 +1,134 @@
+/*
+* By downloading the PROGRAM you agree to the following terms of use:
+* 
+* BROAD INSTITUTE
+* SOFTWARE LICENSE AGREEMENT
+* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY
+* 
+* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”).
+* 
+* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and
+* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions.
+* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows:
+* 
+* 1. DEFINITIONS
+* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE.
+* 
+* 2. LICENSE
+* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute.  LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation.
+* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement.
+* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement.
+* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM.
+* 
+* 3. PHONE-HOME FEATURE
+* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM.  Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time.  Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation.
+* 
+* 4. OWNERSHIP OF INTELLECTUAL PROPERTY
+* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication.
+* Copyright 2012-2015 Broad Institute, Inc.
+* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc.
+* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes.
+* 
+* 5. INDEMNIFICATION
+* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement.
+* 
+* 6. NO REPRESENTATIONS OR WARRANTIES
+* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME.
+* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING.
+* 
+* 7. ASSIGNMENT
+* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void.
+* 
+* 8. MISCELLANEOUS
+* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries.
+* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes.
+* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4.
+* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt.
+* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter.
+* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement.
+* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles.
+*/
+
+package org.broadinstitute.gatk.tools.walkers.cancer.contamination;
+
+import htsjdk.samtools.SAMFileHeader;
+import org.broadinstitute.gatk.utils.cancer.TestingReadUtils;
+import org.broadinstitute.gatk.utils.BaseTest;
+import org.broadinstitute.gatk.engine.GenomeAnalysisEngine;
+import org.broadinstitute.gatk.utils.contexts.AlignmentContext;
+import org.broadinstitute.gatk.utils.contexts.ReferenceContext;
+import org.broadinstitute.gatk.utils.GenomeLoc;
+import org.broadinstitute.gatk.utils.GenomeLocParser;
+import org.broadinstitute.gatk.utils.sam.ArtificialSAMUtils;
+import htsjdk.variant.variantcontext.Allele;
+import htsjdk.variant.variantcontext.Genotype;
+import org.testng.annotations.Test;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+import static org.testng.AssertJUnit.assertTrue;
+
+/**
+ * Created by IntelliJ IDEA.
+ * User: aaron
+ * Date: 2/18/12
+ * Time: 4:05 PM
+ * To change this template use File | Settings | File Templates.
+ */
+public class ContEstUnitTest extends BaseTest {
+    SAMFileHeader header = ArtificialSAMUtils.createArtificialSamHeader(1,1,2000);
+    GenomeLocParser parser = new GenomeLocParser(header.getSequenceDictionary());
+    private static final Map alleles = new HashMap();
+    String primaryRG = new String("genotype");
+    static {
+        alleles.put(0,Allele.create((byte) 'A'));
+        alleles.put(1,Allele.create((byte) 'C'));
+        alleles.put(2,Allele.create((byte) 'G'));
+        alleles.put(3,Allele.create((byte) 'T'));
+    }
+    @Test
+    public void testGettingGenotypes() {
+        // test that we can get a
+        Genotype g = testContaminationGenotype(20,80,0.8);
+        assertTrue(g.isHomVar());
+        assertTrue("T".equals(g.getGenotypeString()));
+
+        g = testContaminationGenotype(19,81,0.8);
+        assertTrue(g.isHomVar());
+        assertTrue("T".equals(g.getGenotypeString()));
+
+        g = testContaminationGenotype(21,79,0.8);
+        assertTrue(g == null);
+    }
+
+    private Genotype testContaminationGenotype(int aBases, int tBases, double minGenotypeRatio) {
+        // setup all the parameters
+        AlignmentContext context = TestingReadUtils.generateAlignmentContext(aBases + tBases, tBases, header, parser, primaryRG, primaryRG);
+        GenomeLoc loc = parser.createGenomeLoc(header.getSequenceDictionary().getSequence(0).getSequenceName(),1,1);
+        ReferenceContext referenceContext = new ReferenceContext(parser,loc,(byte)'A');
+        ContEst.SeqGenotypeMode genotypeMode = ContEst.SeqGenotypeMode.HARD_THRESHOLD;
+        int minGenotypingDepth = 50;
+        double minGenotypingLOD = 5;
+        Map> bamReadGroupMapping = new HashMap>();
+        HashSet brgm = new HashSet();
+        brgm.add(primaryRG);
+
+        bamReadGroupMapping.put(ContEst.GENOTYPE_BAM_TAG,brgm);
+        String sampleName = "sample";
+        GenomeAnalysisEngine toolKit = null;
+        return ContEst.getGenotypeFromSeq(context,
+                referenceContext,
+                alleles,
+                genotypeMode,
+                minGenotypeRatio,
+                minGenotypingDepth,
+                minGenotypingLOD,
+                sampleName,
+                sampleName,
+                toolKit);
+    }
+
+
+}
diff --git a/protected/gatk-tools-protected/src/test/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/MuTect2IntegrationTest.java b/protected/gatk-tools-protected/src/test/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/MuTect2IntegrationTest.java
new file mode 100644
index 000000000..f9b18d4b9
--- /dev/null
+++ b/protected/gatk-tools-protected/src/test/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/MuTect2IntegrationTest.java
@@ -0,0 +1,124 @@
+/*
+* By downloading the PROGRAM you agree to the following terms of use:
+* 
+* BROAD INSTITUTE
+* SOFTWARE LICENSE AGREEMENT
+* FOR ACADEMIC NON-COMMERCIAL RESEARCH PURPOSES ONLY
+* 
+* This Agreement is made between the Broad Institute, Inc. with a principal address at 415 Main Street, Cambridge, MA 02142 (“BROAD”) and the LICENSEE and is effective at the date the downloading is completed (“EFFECTIVE DATE”).
+* 
+* WHEREAS, LICENSEE desires to license the PROGRAM, as defined hereinafter, and BROAD wishes to have this PROGRAM utilized in the public interest, subject only to the royalty-free, nonexclusive, nontransferable license rights of the United States Government pursuant to 48 CFR 52.227-14; and
+* WHEREAS, LICENSEE desires to license the PROGRAM and BROAD desires to grant a license on the following terms and conditions.
+* NOW, THEREFORE, in consideration of the promises and covenants made herein, the parties hereto agree as follows:
+* 
+* 1. DEFINITIONS
+* 1.1 PROGRAM shall mean copyright in the object code and source code known as GATK3 and related documentation, if any, as they exist on the EFFECTIVE DATE and can be downloaded from http://www.broadinstitute.org/gatk on the EFFECTIVE DATE.
+* 
+* 2. LICENSE
+* 2.1 Grant. Subject to the terms of this Agreement, BROAD hereby grants to LICENSEE, solely for academic non-commercial research purposes, a non-exclusive, non-transferable license to: (a) download, execute and display the PROGRAM and (b) create bug fixes and modify the PROGRAM. LICENSEE hereby automatically grants to BROAD a non-exclusive, royalty-free, irrevocable license to any LICENSEE bug fixes or modifications to the PROGRAM with unlimited rights to sublicense and/or distribute.  LICENSEE agrees to provide any such modifications and bug fixes to BROAD promptly upon their creation.
+* The LICENSEE may apply the PROGRAM in a pipeline to data owned by users other than the LICENSEE and provide these users the results of the PROGRAM provided LICENSEE does so for academic non-commercial purposes only. For clarification purposes, academic sponsored research is not a commercial use under the terms of this Agreement.
+* 2.2 No Sublicensing or Additional Rights. LICENSEE shall not sublicense or distribute the PROGRAM, in whole or in part, without prior written permission from BROAD. LICENSEE shall ensure that all of its users agree to the terms of this Agreement. LICENSEE further agrees that it shall not put the PROGRAM on a network, server, or other similar technology that may be accessed by anyone other than the LICENSEE and its employees and users who have agreed to the terms of this agreement.
+* 2.3 License Limitations. Nothing in this Agreement shall be construed to confer any rights upon LICENSEE by implication, estoppel, or otherwise to any computer software, trademark, intellectual property, or patent rights of BROAD, or of any other entity, except as expressly granted herein. LICENSEE agrees that the PROGRAM, in whole or part, shall not be used for any commercial purpose, including without limitation, as the basis of a commercial software or hardware product or to provide services. LICENSEE further agrees that the PROGRAM shall not be copied or otherwise adapted in order to circumvent the need for obtaining a license for use of the PROGRAM.
+* 
+* 3. PHONE-HOME FEATURE
+* LICENSEE expressly acknowledges that the PROGRAM contains an embedded automatic reporting system (“PHONE-HOME”) which is enabled by default upon download. Unless LICENSEE requests disablement of PHONE-HOME, LICENSEE agrees that BROAD may collect limited information transmitted by PHONE-HOME regarding LICENSEE and its use of the PROGRAM.  Such information shall include LICENSEE’S user identification, version number of the PROGRAM and tools being run, mode of analysis employed, and any error reports generated during run-time.  Collection of such information is used by BROAD solely to monitor usage rates, fulfill reporting requirements to BROAD funding agencies, drive improvements to the PROGRAM, and facilitate adjustments to PROGRAM-related documentation.
+* 
+* 4. OWNERSHIP OF INTELLECTUAL PROPERTY
+* LICENSEE acknowledges that title to the PROGRAM shall remain with BROAD. The PROGRAM is marked with the following BROAD copyright notice and notice of attribution to contributors. LICENSEE shall retain such notice on all copies. LICENSEE agrees to include appropriate attribution if any results obtained from use of the PROGRAM are included in any publication.
+* Copyright 2012-2015 Broad Institute, Inc.
+* Notice of attribution: The GATK3 program was made available through the generosity of Medical and Population Genetics program at the Broad Institute, Inc.
+* LICENSEE shall not use any trademark or trade name of BROAD, or any variation, adaptation, or abbreviation, of such marks or trade names, or any names of officers, faculty, students, employees, or agents of BROAD except as states above for attribution purposes.
+* 
+* 5. INDEMNIFICATION
+* LICENSEE shall indemnify, defend, and hold harmless BROAD, and their respective officers, faculty, students, employees, associated investigators and agents, and their respective successors, heirs and assigns, (Indemnitees), against any liability, damage, loss, or expense (including reasonable attorneys fees and expenses) incurred by or imposed upon any of the Indemnitees in connection with any claims, suits, actions, demands or judgments arising out of any theory of liability (including, without limitation, actions in the form of tort, warranty, or strict liability and regardless of whether such action has any factual basis) pursuant to any right or license granted under this Agreement.
+* 
+* 6. NO REPRESENTATIONS OR WARRANTIES
+* THE PROGRAM IS DELIVERED AS IS. BROAD MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE PROGRAM OR THE COPYRIGHT, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, WHETHER OR NOT DISCOVERABLE. BROAD EXTENDS NO WARRANTIES OF ANY KIND AS TO PROGRAM CONFORMITY WITH WHATEVER USER MANUALS OR OTHER LITERATURE MAY BE ISSUED FROM TIME TO TIME.
+* IN NO EVENT SHALL BROAD OR ITS RESPECTIVE DIRECTORS, OFFICERS, EMPLOYEES, AFFILIATED INVESTIGATORS AND AFFILIATES BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ECONOMIC DAMAGES OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER BROAD SHALL BE ADVISED, SHALL HAVE OTHER REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY OF THE FOREGOING.
+* 
+* 7. ASSIGNMENT
+* This Agreement is personal to LICENSEE and any rights or obligations assigned by LICENSEE without the prior written consent of BROAD shall be null and void.
+* 
+* 8. MISCELLANEOUS
+* 8.1 Export Control. LICENSEE gives assurance that it will comply with all United States export control laws and regulations controlling the export of the PROGRAM, including, without limitation, all Export Administration Regulations of the United States Department of Commerce. Among other things, these laws and regulations prohibit, or require a license for, the export of certain types of software to specified countries.
+* 8.2 Termination. LICENSEE shall have the right to terminate this Agreement for any reason upon prior written notice to BROAD. If LICENSEE breaches any provision hereunder, and fails to cure such breach within thirty (30) days, BROAD may terminate this Agreement immediately. Upon termination, LICENSEE shall provide BROAD with written assurance that the original and all copies of the PROGRAM have been destroyed, except that, upon prior written authorization from BROAD, LICENSEE may retain a copy for archive purposes.
+* 8.3 Survival. The following provisions shall survive the expiration or termination of this Agreement: Articles 1, 3, 4, 5 and Sections 2.2, 2.3, 7.3, and 7.4.
+* 8.4 Notice. Any notices under this Agreement shall be in writing, shall specifically refer to this Agreement, and shall be sent by hand, recognized national overnight courier, confirmed facsimile transmission, confirmed electronic mail, or registered or certified mail, postage prepaid, return receipt requested. All notices under this Agreement shall be deemed effective upon receipt.
+* 8.5 Amendment and Waiver; Entire Agreement. This Agreement may be amended, supplemented, or otherwise modified only by means of a written instrument signed by all parties. Any waiver of any rights or failure to act in a specific instance shall relate only to such instance and shall not be construed as an agreement to waive any rights or fail to act in any other instance, whether or not similar. This Agreement constitutes the entire agreement among the parties with respect to its subject matter and supersedes prior agreements or understandings between the parties relating to its subject matter.
+* 8.6 Binding Effect; Headings. This Agreement shall be binding upon and inure to the benefit of the parties and their respective permitted successors and assigns. All headings are for convenience only and shall not affect the meaning of any provision of this Agreement.
+* 8.7 Governing Law. This Agreement shall be construed, governed, interpreted and applied in accordance with the internal laws of the Commonwealth of Massachusetts, U.S.A., without regard to conflict of laws principles.
+*/
+
+package org.broadinstitute.gatk.tools.walkers.cancer.m2;
+
+import org.broadinstitute.gatk.engine.walkers.WalkerTest;
+import org.testng.annotations.Test;
+
+import java.util.*;
+
+public class MuTect2IntegrationTest extends WalkerTest {
+    final static String REF = hg19Reference;
+
+    final static String CCLE_MICRO_TUMOR_BAM = privateTestDir + "HCC1143.cghub.ccle.micro.bam";
+    final static String CCLE_MICRO_NORMAL_BAM = privateTestDir + "HCC1143_BL.cghub.ccle.micro.bam";
+    final static String CCLE_MICRO_INTERVALS_FILE = privateTestDir + "HCC1143.cghub.ccle.micro.intervals";
+
+    final static String DBSNP=b37dbSNP132;
+    final static String COSMIC="/xchip/cga/reference/hg19/hg19_cosmic_v54_120711.vcf";
+    final static String PON="/xchip/cga/reference/hg19/refseq_exome_10bp_hg19_300_1kg_normal_panel.vcf";
+
+    final static String DREAM3_TUMOR_BAM = validationDataLocation + "cancer/dream3.integrationtest.tumor.bam";
+    final static String DREAM3_NORMAL_BAM = validationDataLocation + "cancer/dream3.integrationtest.normal.bam";
+    final static String DREAM3_TP_INTERVALS_FILE = privateTestDir + "m2_dream3.tp.intervals";
+    final static String DREAM3_FP_INTERVALS_FILE = privateTestDir + "m2_dream3.fp.intervals";
+
+
+
+    private void M2Test(String tumorBam, String normalBam, String intervals, String args, String md5) {
+                final String base = String.format(
+                "-T MuTect2 --no_cmdline_in_header -dt NONE --disableDithering -alwaysloadVectorHMM -pairHMM LOGLESS_CACHING -ip 50 -R %s --dbsnp %s --cosmic %s --normal_panel %s -I:tumor %s -I:normal %s -L %s",
+                REF, DBSNP, COSMIC, PON, tumorBam, normalBam, intervals) +
+                " -o %s ";
+
+        final WalkerTestSpec spec = new WalkerTestSpec(base + " " + args, Arrays.asList(md5));
+
+        // TODO: do we want to enable this and why?  It explodes with
+        // java.lang.RuntimeException: java.lang.ClassCastException: java.lang.Double cannot be cast to java.lang.String
+        //    at htsjdk.variant.variantcontext.writer.BCF2FieldEncoder$StringOrCharacter.javaStringToBCF2String(BCF2FieldEncoder.java:312)
+        spec.disableShadowBCF();
+        executeTest("testM2: args=" + args, spec);
+    }
+
+    @Test
+    public void testMicroRegression() {
+        M2Test(CCLE_MICRO_TUMOR_BAM, CCLE_MICRO_NORMAL_BAM, CCLE_MICRO_INTERVALS_FILE, "", "617054c6d056cad7448a463cb8d04a55");
+    }
+
+    /**
+     * Tests all the True Positive sites in the DREAM 3 data set.  We don't necessarily call
+     * all of these (e.g. we have some FNs) but it's the full set of things we want to be able
+     * to call, and not regress
+     */
+    @Test
+    public void testTruePositivesDream3() {
+        M2Test(DREAM3_TUMOR_BAM, DREAM3_NORMAL_BAM, DREAM3_TP_INTERVALS_FILE, "", "f856432679e43445d2939772be4326cf");
+    }
+
+    /**
+     * Tests a number of False Positive calls from the DREAM 3 data set.  Some of them are not rejected
+     * (e.g. we have some FPs!) but most are rejected.
+     */
+    @Test
+    public void testFalsePositivesDream3() {
+        M2Test(DREAM3_TUMOR_BAM, DREAM3_NORMAL_BAM, DREAM3_FP_INTERVALS_FILE, "", "11357aa543e7c6b2725cd330adba23a0");
+    }
+
+    /*
+     * Test that contamination downsampling reduces tumor LOD, rejects more variants
+     */
+    @Test
+    public void testContaminationCorrection() {
+        M2Test(CCLE_MICRO_TUMOR_BAM, CCLE_MICRO_NORMAL_BAM, CCLE_MICRO_INTERVALS_FILE, "-contamination 0.1", "d7947ddf0240fe06a44621312831f44c");
+    }
+
+}
diff --git a/protected/gatk-tools-protected/src/test/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/how_to_make_dream3_bams.txt b/protected/gatk-tools-protected/src/test/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/how_to_make_dream3_bams.txt
new file mode 100644
index 000000000..e10ab75ca
--- /dev/null
+++ b/protected/gatk-tools-protected/src/test/java/org/broadinstitute/gatk/tools/walkers/cancer/m2/how_to_make_dream3_bams.txt
@@ -0,0 +1,24 @@
+# intervals file (m2_dream3.intervals) was created by merging the following data sets:
+#     1. all M2 calls (including failures) from chr21 on the full DREAM 3 challenge data set, after being called 
+#        realigned by BWA-MEM.  This is to capture FP, TN and some TP
+#     2. all TP events from the DREAM 3 truth data for chr21
+#
+# Then the following command was run to produce the subsetted BAMs
+
+cat m2_dream3.tp.intervals m2_dream3.fp.intervals > m2_dream3.intervals
+
+export GATK_JAR=/humgen/gsa-hpprojects/GATK/bin/current/GenomeAnalysisTK.jar
+export REF=/seq/references/Homo_sapiens_assembly19/v1/Homo_sapiens_assembly19.fasta 
+
+java -jar $GATK_JAR -T PrintReads -R $REF -ip 250 -L m2_dream3.intervals \
+-I /cga/tcga-gsc/benchmark/data/realignments/synthetic.challenge.set3.tumor/IS3.snv.indel.sv.bam \
+-o dream3.integrationtest.tumor.bam
+
+
+java -jar $GATK_JAR -T PrintReads -R $REF -ip 250 -L m2_dream3.intervals \
+-I /cga/tcga-gsc/benchmark/data/realignments/synthetic.challenge.set3.normal/G15512.prenormal.sorted.bam \
+-o dream3.integrationtest.normal.bam
+
+mv *.bam /humgen/gsa-hpprojects/GATK/data/Validation_Data/cancer
+mv *.bai /humgen/gsa-hpprojects/GATK/data/Validation_Data/cancer
+
diff --git a/public/gatk-tools-public/src/main/java/org/broadinstitute/gatk/tools/walkers/annotator/interfaces/StandardSomaticAnnotation.java b/public/gatk-tools-public/src/main/java/org/broadinstitute/gatk/tools/walkers/annotator/interfaces/StandardSomaticAnnotation.java
new file mode 100644
index 000000000..72eecaa36
--- /dev/null
+++ b/public/gatk-tools-public/src/main/java/org/broadinstitute/gatk/tools/walkers/annotator/interfaces/StandardSomaticAnnotation.java
@@ -0,0 +1,28 @@
+/*
+* Copyright 2012-2015 Broad Institute, Inc.
+* 
+* Permission is hereby granted, free of charge, to any person
+* obtaining a copy of this software and associated documentation
+* files (the "Software"), to deal in the Software without
+* restriction, including without limitation the rights to use,
+* copy, modify, merge, publish, distribute, sublicense, and/or sell
+* copies of the Software, and to permit persons to whom the
+* Software is furnished to do so, subject to the following
+* conditions:
+* 
+* The above copyright notice and this permission notice shall be
+* included in all copies or substantial portions of the Software.
+* 
+* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+* OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+* HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR
+* THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+*/
+
+package org.broadinstitute.gatk.tools.walkers.annotator.interfaces;
+
+public interface StandardSomaticAnnotation extends AnnotationType {}