Commit Graph

5365 Commits (31a2575c7bc1485d18ccaa0bc07d608be4d54865)

Author SHA1 Message Date
chartl 31a2575c7b Fixes:
- Don't know how I got the wiggle header so utterly wrong. Fixed.
 - Q-values now have a static maximum of 2000 so IGV averaging won't make everything look spikey and ugly.
 - Changing windows to size 100 for (hopefully) better resolution.



git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5406 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-08 17:16:21 +00:00
chartl 1b310401fe Due to the approximation not being well-founded in this case, (and the non-existence of a pre-computed table at this time), pushing up the cutoff
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5405 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-08 16:24:42 +00:00
delangel 00ac51acc8 Added several integration tests for UG indel caller:
- Basic
- Multiple technology
- Test minIndelCnt parameter

Added also 2 disabled tests:
- Parallelization: issue w/code right now is that if -nt > 1, filter field shows "PASS" instead to ".", cause TBD
- Genotype given alleles mode: code not working yet.



git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5404 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-08 16:21:21 +00:00
chartl 77fe902dbd Testing modules now use wider windows and heftier shift, hopefully this will remove some of the noisiness of the results. Some UStatistics were changed to TStatistics to try and limit noisiness as well. Walker will also additionally write out wiggle files directly (which can be converted into "proper" tdf files via igvtools tile [args] [in].wig [out].tdf [ref]) subject to some restrictions. MWU could get stuck in a long-running recursive regime, it'd be nice to have a table lookup or a good small-n large-m approximation, for now the uniform should work just fine.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5403 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-08 15:26:13 +00:00
carneiro b733cba7c7 re-fixing for a different approach suggested by eric!
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5402 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-08 04:54:49 +00:00
kshakir 5564838087 Email password should be specified on the command line OR the password file, not and.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5401 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-08 03:42:09 +00:00
kiran d0598c7a04 Somehow missed this test when I was updating the md5s
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5400 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 23:53:42 +00:00
kshakir c03341aec1 JobRunner can be specified on the command line. -bsub is currently short form of -jobRunner Lsf706.
Added an empty wrapper for a GridEngine job runner which is only activated when SGE_ROOT is detected.
Refactored a bit more common code into CommandLineJobRunner / JobRunner / FunctionEdge.
Status for analyisNames now includes the number of functions in state.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5399 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 23:52:48 +00:00
kiran b6339967f8 Updated GenomicAnnotator integration tests to include the -NO_HEADER argument so that they tests op yelling about trtrivial differences
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5398 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 23:07:01 +00:00
hanna 85ff983a59 Failed to include some required GenomeLoc utilities in my last commit.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5397 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 23:00:17 +00:00
carneiro 02006954bc UG: small bug fix when creating empty variant contexts in UG for the -EMIT_ALL_SITES to allow indels.
GAV: First version of the walker that validates reads from a BAM file based on an annotated VCF with TP/FP annotations. 


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5396 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 22:51:04 +00:00
hanna 9384b2ff65 A few quick fixes to temporarily make the LowMemorySharder return exactly the
same shards as the previous sharder, so that I can directly compare filespans 
to see where some performance bugs lie.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5395 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 22:43:14 +00:00
depristo 0b4e51317b Now includes project consensus high sensitivity data set
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5394 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 20:52:11 +00:00
kiran 43056d0188 Fixed integration test to reflect changes regarding when comp tracks got subset to fewer samples and whether no-call sites would get pulled in for comp tracks.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5393 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 20:25:57 +00:00
carneiro 73e43d8d2c Added functionality:
-disc (--discordance) parameter together with a ROD track will output a VCF with the variants in the ROD track that are not present in the 'variants' VCF. Useful tool to list the variants from hapmap (for example) that weren't called in a dataset.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5392 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 19:18:15 +00:00
depristo bf2e02f472 Generic, easy-to-use variant evaluation Queue script that tests indel and SNP call sets against standard evaluation data sets for sensitivity and specificity
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5391 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 18:03:29 +00:00
kshakir dc33fbed7c Switched the CVUnitTest broken info from an Integer to a String since as of r5383 Integers are no longer broken when converted to Floats.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5390 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 16:33:14 +00:00
delangel 8c262eb605 Initial commit of new likelihood model to evaluate indel quality. Principle is simple, a plain Pair HMM with affine gap penalties (in log space) that does quasi-local alignment between reads and candidate haplotypes and which in theory should be more solid and more reliable than the older Dindel-based model. It also allows to be easily extensible in the future if we decide to introduce either context-dependent and/or read-dependent gap penalties.
Model is disabled by default and we're still using the old Dindel model until I'm more confident that new model is a definitive improvement, so right now this is enabled by hidden command line arguments, and it's not to be used yet.

In detail:
a) Several refactorings to share softMax() available to other modules, so its now part of MathUtils.
b) Refactored a couple of read utilities and moved from BAQ to ReadUtils.
c) New PairHMMIndelErrorModel class implementing new likelihood model
d) Several new hidden debug arguments in UAC.



git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5389 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 15:31:58 +00:00
kshakir 96fe540d66 Removing .tmp~ file.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5388 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 14:52:38 +00:00
kshakir 92045ecaa6 Finally figured out what data in the LSF C API call lsb_readjobinfo is causing JNA to SIGSEGV with a strlen error:
- LSF recycles memory for C arrays, but sets a separate variable setting the size of the array to zero.
- JNA only sees the non-NULL pointer and starts to auto-access it, sometimes causing a SIGSEGV.
- In the short term neutered the jobInfoEnt structure so that this bad array is not autoRead().

QGraph updates:
- Job status is now checked in bulk every 30 seconds instead of one job at a time, even in the middle of dispatching jobs.
- If there is a hiccup (unexpected but not fatal error) during status check then the the error is ignored and status is checked again 30 seconds later.
- Jobs prefer to dispatch depth vs. breadth first.

More refactoring of SG framework separating the reusable code from the implementations.
The DistributedScatterFunction is still a work in progress and is not enabled yet. Still need to think through how Queue handles when a job dies.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5387 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 04:29:06 +00:00
chartl 60ddc08cdf Added a boatload of new case-control association modules. Switched the U-test to use longs rather than ints (it just so happened that I overflowed and started getting negative U statistics. Not good.) Added the ALL association type for ease of specifying that we want to throw the book at something. Added an svn-commit.tmp~ because i can't get rid of it even with --force. Hopefully I can remove it after.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5386 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 21:58:12 +00:00
depristo 5c979633f0 Due to a problem in the way that dynamic type selection works, I've added an explicit (temporary) ability to restrict VE to specific variant types (SNPs, INDELs, etc), so that calculations will work when a site has a SNP in dbSNP but is called as an indel, causing the SNP site to mysteriously disappear from the comp track, a huge problem for validation report. VEU updated to allow both dynamic type (old) and just returning everything in the track.
Also, created a standard Queue script that calculates a suite of standard indel and SNP assessment results.  Will be the basis for a general evaluation Queue script with standardized data files for SNPs and Indels.

git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5385 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 19:31:12 +00:00
depristo 2f1e249aed A proper validation report, calculating TP, FP, FN, sensitivity, FDR, PPV. Treats comp as a set of sites that have been either filtered (failed in assay), validated (polymorphic among samples), or invalidated (AC=0 or all genotypes = hom-ref). Very useful.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5384 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 19:27:40 +00:00
depristo af71576a07 CalculateChromosomeCounts() now only calculates AC, AF, and AN when there are genotypes. Can now combine variants with headers that differ in only whether a field is a integer or a float. Updated CombineVariants integrationtest, as incorrect AC values where being calculated in the previous GS outputs.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5383 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 19:25:52 +00:00
depristo 5b8fdc5b1f Slightly optimized calculation for ~linear exact model, as well as totally incorrect banded calculation, for future development, if this proves useful.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5382 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 18:47:08 +00:00
chartl fc5a071de2 Output format is 10^6 times better - now uses the multiplexer to write tdf tracks that can (after conversion to binary with igvtools) can be loaded directly into igv.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5381 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 18:23:52 +00:00
chartl a40a8006b5 Added in unit tests for the statistics calculated by the test runner; and bug-fixes to the calculations; so we have some assurance that the statistics coming out the back-end are correct.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5380 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 16:54:02 +00:00
hanna c40efe1dea Fixed exception for BAMs without filenames (unit tests, BAM input streaming,
etc.).


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5379 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 13:43:49 +00:00
depristo ad51f30244 A trivial, but useful, sum of a list of integers
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5378 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-05 06:09:05 +00:00
depristo 9a8356892a Cleaner error (really now just warnings) if you can't reach the S3 for logging
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5377 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-05 06:08:35 +00:00
hanna 10516f5de4 Fixed one low-memory sharder performance culprit: regions with no BAM data
whatsoever were misusing the Picard MergingIterator, triggering a re-traversal 
through the entire contig.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5376 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-04 21:26:22 +00:00
ebanks 337b54136f 2 fixes. For Mark: when insertions can be partially left-aligned, we were reading off the wrong bases. For GS post: the stored VariantContext.REFERENCE_BASE_FOR_INDEL_KEY needs to be updated when left-aligning because it can change.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5375 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-04 21:00:08 +00:00
kiran b42005e7d7 Fixed issue where comp tracks with genotypes that didn't exactly overlap the eval track were getting dropped. Fixed issue where the 'row' column wasn't being output for things implementing TableType. This is an urgent patch for Mark - it'll break tests until I go back and update the md5s.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5374 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-04 16:51:12 +00:00
kiran 1861ca90fc A change to the definition of CpG sites (is now, from 5' to 3' a CG dinucleotide in the reference, and the CpG site is at the C, rather than either at the C or a G).
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5373 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-04 15:36:07 +00:00
chartl 9ca1dd5d62 Miscellaneous changes:
- RefMetaDataTracker: grabbing variant contexts given a prefix (not sure where else this was implemented, if someone can show me I'll remove it)
 - VCFUtils: grabbing VCF headers given a prefix 
 - MathUtils: Useful functions for calculating statistics on collections of Numbers
 - VariantAnnotator: Made isUniqueHeaderLine a public static method -- maybe this should go into a different class. Not sure.
 - Associations: PluginManager now used to propagate classes, implementations for Z,T,U tests, slight alteration to format to make the objects stored
      in the window optionally different from those returned by whatever statistic is run across the window
Added:
 - MannWhitneyU. Started to fix up WilcoxonRankSum but there are comments in there questioning the validity of some of the code, and I'm sure that
    it's actually doing a U test. This implementation includes the direct calculation of p-values for small sample sizes, and a uniform approximation
    for when one of the sample sets is small, and the other large. Unit tests to follow.
 - BootstrapCallsMerger: takes n VCFs which have been called on the same samples; merges them together while averaging the annotations
 - BootstrapCalls.q: qscript for testing the effectiveness of boostrap low-pass calling on the exome
 


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5372 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 22:43:36 +00:00
carneiro 8ae42b70ac give it an annotated VCF and a BAM and it creates a truth table on the validation of the VCF calls. This is just the first version, not ready for primetime.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5371 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 21:45:11 +00:00
carneiro 0daa65b9ef quick and dirty 'close your eyes' solution to run the papuans over the weekend. Will be properly fixed soon.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5370 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 21:42:22 +00:00
rpoplin f7ef35b8f5 Removing untrue comments in the GaussianMixtureModel
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5369 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 18:18:47 +00:00
chartl 9e12cd1312 Gotta include the changes i made to get an init function into the contexts
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5368 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 15:45:51 +00:00
chartl 835a26d145 A pass at a sample-normalized test. I think maybe all of them will simply do their own normalizing.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5367 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 15:42:25 +00:00
chartl ef38fd1e0e Major refactoring of association testing framework. New modules are now beyond trivial to implement. One hurdle remains which is how to deal with statistics that ought to be sample-normalized (e.g. depth, insert-size [when multiple libraries are used], and possibly others).
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5366 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 14:27:16 +00:00
hanna 5d4bbf41fb Behave intelligently in the deepest levels of GATK record filtration when
we find a read flagged as 'mapped' in the unmapped region at the end of the 
file.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5365 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 04:52:55 +00:00
hanna 7a22f19366 More descriptive error when VerifyingSamIterator hits an inconsistent alignment. Also updated
case UserException.MalformedBAM to match case of UserExceptio.MissortedBAM for consistency and
ease-of-use.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5364 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 03:55:24 +00:00
kshakir a0309e7fb0 Want to get this into Ryan's hands asap: First working version of a distributed scatter function.
More refactoring to do so that other new scatter functions can be implemented very easily and annotated on walkers.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5363 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 01:40:08 +00:00
depristo 0181d95fe4 Intermediate optimization checkin. LinearExact model now about 10-20% faster than previous commit, by reorganizing and optimizing the if statements and genotype likelihood calculations. Next commit will include a banded implementation.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5362 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 22:01:35 +00:00
ebanks f0f4bc3363 This was busted because it assumed 1 (and only 1) record at each position. However it's possible to have 0 (which generated a NullPointer) or 2+ records (which dropped records).
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5361 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 21:35:50 +00:00
kshakir 1e68259f5c Stage one of refactoring GATK scatter functions. This intermediate stage should only be used by "rhymes with Shmoplin" until the next refactoring.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5360 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 18:10:19 +00:00
depristo c152ef4339 Better error message for unknown reference file extension.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5359 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 17:52:16 +00:00
hanna bef83b8b09 Bug fix: was tracking state across BAMs that should've been tracked per-BAM.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5358 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 17:32:06 +00:00
depristo bafa61c1fe LINEAR_EXACT now the default model. Passes all integration tests. 2-3x faster in low-pass data. Tests on exome data ongoing, but potentially vastly faster there.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5357 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 17:14:36 +00:00