* The value of this element (default true) determines whether Queue will explicitly run this walker over unmapped reads
* This patch fixes a runtime error when FindCoveredIntervals was used with Queue
* PT 81777160
* TextCigarCodec.decode() is now static, and the getSingleton() method is gone
* MergingSamRecordIterator now wants a Collection<SamReader> rather than Collection<SAMFileReader> in the constructor
* SeekableBufferedStream now correctly reads the requested number of bytes, removed workaround in GATKBAMIndex
* Removed unused annotations (CCC and HWP)
* Renamed one of the two GC annotations to "IGC" (for Interval GC)
* Revved picard & htsjdk (GATK constants are now removed from htsjdk)
* PT 82046038
-- Active Region Traversal was using per sample limits on the number of reads that were too low, especially now that we are running one sample at a time. This caused issues with high confidence variants being dropped in high coverage data.
-- HaplotypeCallerGVCFIntegrationTest PL/annotation changes due to using more reads in those tests
-- Removed a CountReadsInActiveRegionsIntegrationTest test for excessive coverage because the read coverage no longer goes over the limits in ART
Add multi-allele test for info field annotations
Fix to process all types of INFO annotations
roll back to previous version, removes INFO and FORMAT
Correct @return for VariantAnnotatorEngine.getNonReferenceAlleles()
Enhance comments and clean up multi-allelic logic, handle header info number = R
only parse counts of A & R
Add INFO for AC
update MD5
Performance enhancement, only parse multiallelic with a count A or R
Make argument final in getNonReferenceAlleles()
Code cleanup, add exceptions for bad expression/allele size mismatch and missing header info for an expression
Change exception to warning for expression value/number of alleles check
remove adevertised exceptions
* PT 84242218
* Note that FORMAT fields behave the same as INFO fields - if the annotation has a count of A (one entry per Alt Allele), it is split across the multiple output lines. Otherwise, the entire list is output with each field
Add more logging to annotators, change loggers from info to warn
Add comments to testStrandBiasBySample()
Clarify comments in testStrandBiasBySample
remove logic for not prcossing an indel if strand bias (SB) was not computed
remove per variant warnings in annotate()
Log warnings if using the wrong annotator or missing a pedgree file
Log test failures once in annotate(), because HaplotypeCaller does not call initialize(). Avoid using exceptions
Fix so only log once in annotate(), Hardey-Weinberg does not require pedigree files, fix test MD5s so pass
Check if founderIds == null
Update MD5s from HaplotypeCaller integrations tests and clean up code
Change logic so SnpEff does not throw excpetions, change engine to utils in imports
Update test MD5s, return immediately if cannot annotate in SnpEff.initialization()
Post peer review, add more logging warnings
Update MD5 for testHaplotypeCallerMultiSampleComplex1, return null if PossibleDeNovo.annotate() is not called by VariantAnnotator
Story:
-----
https://www.pivotaltracker.com/story/show/80684230
Changes:
-------
- Corrected the bug: AlignmentUtils#createReadAlignedToRef was
not realigning against the reference but the best haplotype for
the read.
Test:
----
- Added integration test in HaplotypeCallerIntegrationTest to check
that the bug has been fixed.
- Fixed md5s modified by this change; these are cause due to small
changes in the state of the random-number generator and read vs
variant site overlapping.
Reading the multiple GATKText files as a single stream, especially with new top level target executable jar files pointing to a lib folder.
Don't dirty the build with a new GATKText.properties if input files are unmodified.
Stop warning on undocumented abstract classes.
Fixed ClassNotFoundException/NoClassDefFoundError by fixing ResourceBundleExtractorDoclet artifact.
Excluding Exceptions from documentation.
Removed custom log4j dependency from ResourceBundleExtractorDoclet.
Stop generating the dependency reduced pom during shade.
Stop regenerating gsalib when the files are already up to date.
Disabled mvn site generation from external-example.
Moved top level target symlinks to package jar files to under target/package.
Executable jar files are placed under target/executable with the new target[/lib] directories.
Under top level target, symlinks to *either* the package *or* the executable jars replace what was a symlink to the package jar path.
Allow disabling of the shade package.
ant-bridge.sh by default only builds executable jars, and doesn't package by default, as did the old ant build.xml.
Added a new package_path.sh utility script for other scripts to use instead of anything in the target folder.
Fixed off by one error in size calculation IntervalUtils.scatterContigIntervals().
In test for fewer files than intervals, adjusted expected intervals.
In test for more files than intervals, adjusted expected exception.
remove final keyword before refMap and altMap, constructHaplotype() changes their values
return ArtificialHaplotype from constructHaplotype instaed of passing as an argument
Add logic so arraycopy does not throw an IndexOutOfBoundsException, add test for a long insert
remove TODO comment after activeProbThreshold
recover static ACTIVE_PROB_THRESHOLD for unit tests
Add min/max values for active_probability_threshold parameter
Move activeProbThreshold parameter to GATKArguemtnCollection
define ACTIVE_PROB_THRESHOLD in unit tests
add construction of argCollection in in ctor
Move arguments from GATKArgumentCollection to ActiveRegionWalker
Throw exception if threshold < 0 or > 1 in ActivityProfile ctor
max propogation distance parameter to ActiveRegionWalker for AcrtivityProfile
Use polymorphic getMaxProbPropagationDistance() so BandPassActivityProfile computes the crrect region size cutoff
Get the maxProbPropagationDistance from the super class's method, instead of directly, this is safer
Removed extraneous command line imports and make maxProbPropagationDistance a hidden argument
remove limit check for activeProbThreshold, not necessary because the check is made when imput as a command line arg
Remove extra 'region' in the doxygen param description for maxProbPropagationDistance
* This argument forces GATK to always write every record in the VCF format field, even if some records at the end are missing and could be removed
* Revved htsjdk and picard
* PT 70993484
Changes:
-------
* Updated current unit and integration test to use the new API components.
* Added unit tests for new classes AFPriorProvider and AFCalculatorProviders.
* Added integration test for mixed ploidy GenotypeGVCFs and CombineGVCFs
Changes:
-------
* GenotypingEngine uses now a AFCalc provider instead of
its own thread-local with one-time initialized and fixed
AF calculator.
* All walkers that use a GenotypingEngine now are passing
the appropiate AF calculator provider. For now most
just use a fix calculator (FixedAFCalculatorProvider)
except GenotypeGVCFs as this one now can cope with
mixture of ploidies failing-over to a general-ploidy
calculator when the preferred implementation is not
capable to handle a site's analysis.
* Arguments involved are --no_cmdline_in_header, --sites_only, and --bcf for VCF files and --bam_compression, --simplifyBAM, --disable_bam_indexing, and --generate_md5 for BAM files
* PT 52740563
* Removed ReadUtils.createSAMFileWriterWithCompression(), replaced with ReadUtils.createSAMFileWriter(), which applies all appropriate engine-level arguments
* Replaced hard-coded field names in ArgumentDefinitionField (Queue extension generator) with a Reflections-based lookup that will fail noisily during extension generation if there's an error
Explicitly including gatk/queue test-jar artifacts in package test classpaths.
SelectVariantsIntegrationTest#testInvalidJexl now resets the JexlEngine silent flag that VariantFiltration.initialize() toggles.
External example no longer tries to unpack nonexistent gatk artifact jars during package tests.
Same changes fixed the problem for GenotypeGVCFs and CombineGVCFs.
Stories:
- https://www.pivotaltracker.com/story/show/77626044
- https://www.pivotaltracker.com/story/show/77626854
Changes:
- Generalized the code for the merging in GATKVariantContextUtils to cope
with ploidy != 2.
- GenotypeGVCFs now check that the input's ploidy conform to the '-ploidy'
argument.
- Moved out Refernce Confidence VC merging code from GATKVariantContextUtils
so that we can keep new code in protected.
Caveats:
- GenotypeGVCFs only can deal with input files that have the same ploidy in
all positions; the one that the user MUST indicate in the -ploidy argument
(if different to the default 2).
- CombineGVCFs won't necessarely complain if its passed mixed ploidy
inputs but you won't be able to genotype it with GenotypeGVCFs.
Test:
- Removed deprecated unit tests for GATKVariantContextUtils.
- Moved unit-tests regarding GVCF merging from GATKVariantContextUtilsUnitTest
to ReferenceConfidenceVariantContextUtilsUnitTest.
- Added unit test for new code for mapping genotype indices between allele
index encoding in GenotypeLikelihoodCalculator.
- GenotypeGVCFs and CombineGVCFs original integration test are unaffected
by the change.
- Added tetraploid run integration tests to check on non-diploid execution
of GenotypeGVCFs and CombineGVCFs.
Changed tests and scripts to use gatkdir full path instead of relative testdata/qscripts symbolic links.
Although symlinks not created, left the symlink deletion script execution with a comment about future removal.
Re-enabled example UG pipeline queue test.
Replaced all hardcoded strings of {public,private}/testdata with BaseTest variables.
Refactored temp list creation method from ListFileUtilsUnitTest to BaseTest.createTempListFile.
Removed list files with hardcoded paths, now using createTempListFile instead with private test dir variable.
Story:
https://www.pivotaltracker.com/story/show/77250524
Changes:
- Remove the annotating code in GeneralPloidyExactAFCalc (GPEAFC) class.
- Added the asAlleleList to GenotypeAlleleCounts class and get (GPEAFC) to use that instead of implementing its own (nicer and more reusable code).
- Removed the explicit addition of AlleleCountBySample fields to the VCF header by the walker initialize
- Added utility methods in Utils to wrap and int[] array into a List<Integer>, and double[] array into a List<Double> efficiently.
Test:
- Added unit-testing for asAlleleList in GenotypeAlleleCountsUnitTest (within testFirst and testNext).
- Added unit-testing for new methods in Utils : asList(int[]) and asList(double[])
- Changed UG General Ploidy test to add explicitly those annotations.
- Non-trivial changes in integration tests involving non-diploid runs (namelly haploid and tetraploid) as they are not showing
those annotations anylonger, so the MD5s have been changed accordingly.
Changes in several walker to use new sample, allele closed lists and new GenotypingEngine constructors signatures
Rebase adoption of new calculation system in walkers
If any pair of variants occurs on all used haplotypes together, then we propagate that information into the gVCF.
Can be enabled with the --tryPhysicalPhasing argument.
- Read groups that are excluded by sample_name, platform, or read_group arguments no longer appear in the header
- The performance penalty associated with filtering by read group has been essentially eliminated
- Partial fulfillment of PT 73075482
Stories:
https://www.pivotaltracker.com/story/show/70222086https://www.pivotaltracker.com/story/show/67961652
Changes:
Done some changes that I missed in relation with making sure that all PairHMM implentations use the same interface; as a consequence we were running always the standard PairHMM.
Fixed some additional bugs detected when running it on full wgs single sample and exom multi sample data set.
Updated some integration test md5s.
Fixing GraphBased bugs with new master code
Fixed ReadLikelihoods.changeReads difficult to spot bug.
Changed PairHMM interface to fix a bug
Fixed missing changes for various PairHMM implementations to get them to use the new structure.
Fixed various bugs only detectable when running with full sample(s).
Believe to have fixed the lack of annotations in UG runs
Fixed integrationt test MD5s
Updating some md5s
Fixed yet another md5 probably left out by mistake
The array structure should be faster to populate and query (no properly benchmarked) and reduce memory footprint considerably.
Nevertheless removing PairHMM factor (using likelihoodEngine Random) it only achieves a speed up of 15% in some example WGS dataset
i.e. there are other bigger bottle necks in the system. Bamboo tests also seem to run significantly faster with this change.
Stories:
https://www.pivotaltracker.com/story/show/70222086https://www.pivotaltracker.com/story/show/67961652
Changes:
- ReadLikelihoods added to substitute Map<String,PerSampleReadLikelihoods>
- Operation that involve changes in full sets of ReadLikelihoods have been moved into that class.
- Simplified a bit the code that handles the downsampling of reads based on contamination
Caveats:
- Still we keep Map<String,PerReadAlleleLikelihoodsMap> around to pass to annotators..., didn't feel like change the interface of so many public classes in this pull-request.
In particular, it was possible to specify arguments for Files or Compound types without values
Added a special "none" value for annotations, since a bare "-A" is no longer allowed
Delivers PT 71792842 and 59360374
ValidationStringency was moved from htsjdk.samtools.SAMFileReader to htsjdk.samtools
samtools find BAM index file method was also moved (and made public!)
- Edited intervals merging docs for correctness & clarity
- Edited VQSR arg docs and made mode required (+added -mode SNP to VQSR tests)
- Moved PaperGenotyper to Toy Walkers to declutter the actually useful docs
- Moved GenotypeGVCFs to Variant Discovery category and clarified a few points
- Clarified that the -resource argument depends on using the -V:tag format
- Clarified how the pcr indel model works
- Added caveat for -U ALLOW_N_CIGAR_READS
- Added MathJax support for displaying equations in GATKDocs
- Updated HC example commands and caveats
In these cases, where the alignment contains multiple indels, we output a single complex
variant instead of the multiple partial indels.
We also re-enable dangling tail recovery by default.
-- AD,DP will now correspond directly to the reads that were used to construct the PLs
-- RankSumTests, etc. will use the bases from the realigned reads instead of the original alignments
-- There is now no additional runtime cost to realign the reads when using bamout or GVCF mode
-- bamout mode no longer sets the mapping quality to zero for uninformative reads, instead the read will not be given an HC tag
-When parental genotypes are available, implements an HMM on genotype observations in the quartet.
-Outputs IBD regions as well as per-site posterior probabilities of being in each IBD state.
-Includes an experimental heuristic based mode for when parental genotypes are not available.
-Made a method in MendelianViolation public static to reuse code.
-Added the mockito library to private/gatk-tools-private/pom.xml
- Fields in the header are delimited with the pipe character
- Header is now split into two lines to improve spacing
- Field width in header and progress lines auto-adjusts to length of "processing units" label (sites, active regions, etc)
- Addresses PT 69725930
-They are now only computed when necessary
-Log10Cache is dynamically resizable, either by calling get() on an out-of-range value or by calling ensureCacheContains
-Log10FactorialCache and JacobianLogTable are initialized to a fixed size on first access and are not resizable
-Addresses PT 69124396
-Make BaseTest.createTempFile() mark any possible corresponding index files for deletion on exit
-Make WalkerTest mark shadow BCF files and auxiliary for deletion on exit
-Make VariantRecalibrationWalkersIntegrationTest mark PDF files for deletion on exit
Ron Levine
biocyberman
Geraldine Van der Auwera
seru71
Phillip Dexheimer
rpoplin
Khalid Shakir
Joel Thibault
Menachem Fromer
Alec Wysoker
Ron Levine
Laura Gauthier
Valentin Ruano-Rubio
Chris Smowton
Sheila Chandran
droazen
Eric Banks
Michael Linderman
kshakir
Nigel Delaney
Nigel Delaney
Chris Whelan