96fe540d66
Removing .tmp~ file.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5388 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 14:52:38 +00:00
92045ecaa6
Finally figured out what data in the LSF C API call lsb_readjobinfo is causing JNA to SIGSEGV with a strlen error:
...
- LSF recycles memory for C arrays, but sets a separate variable setting the size of the array to zero.
- JNA only sees the non-NULL pointer and starts to auto-access it, sometimes causing a SIGSEGV.
- In the short term neutered the jobInfoEnt structure so that this bad array is not autoRead().
QGraph updates:
- Job status is now checked in bulk every 30 seconds instead of one job at a time, even in the middle of dispatching jobs.
- If there is a hiccup (unexpected but not fatal error) during status check then the the error is ignored and status is checked again 30 seconds later.
- Jobs prefer to dispatch depth vs. breadth first.
More refactoring of SG framework separating the reusable code from the implementations.
The DistributedScatterFunction is still a work in progress and is not enabled yet. Still need to think through how Queue handles when a job dies.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5387 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 04:29:06 +00:00
60ddc08cdf
Added a boatload of new case-control association modules. Switched the U-test to use longs rather than ints (it just so happened that I overflowed and started getting negative U statistics. Not good.) Added the ALL association type for ease of specifying that we want to throw the book at something. Added an svn-commit.tmp~ because i can't get rid of it even with --force. Hopefully I can remove it after.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5386 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 21:58:12 +00:00
5c979633f0
Due to a problem in the way that dynamic type selection works, I've added an explicit (temporary) ability to restrict VE to specific variant types (SNPs, INDELs, etc), so that calculations will work when a site has a SNP in dbSNP but is called as an indel, causing the SNP site to mysteriously disappear from the comp track, a huge problem for validation report. VEU updated to allow both dynamic type (old) and just returning everything in the track.
...
Also, created a standard Queue script that calculates a suite of standard indel and SNP assessment results. Will be the basis for a general evaluation Queue script with standardized data files for SNPs and Indels.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5385 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 19:31:12 +00:00
2f1e249aed
A proper validation report, calculating TP, FP, FN, sensitivity, FDR, PPV. Treats comp as a set of sites that have been either filtered (failed in assay), validated (polymorphic among samples), or invalidated (AC=0 or all genotypes = hom-ref). Very useful.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5384 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 19:27:40 +00:00
af71576a07
CalculateChromosomeCounts() now only calculates AC, AF, and AN when there are genotypes. Can now combine variants with headers that differ in only whether a field is a integer or a float. Updated CombineVariants integrationtest, as incorrect AC values where being calculated in the previous GS outputs.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5383 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 19:25:52 +00:00
5b8fdc5b1f
Slightly optimized calculation for ~linear exact model, as well as totally incorrect banded calculation, for future development, if this proves useful.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5382 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 18:47:08 +00:00
fc5a071de2
Output format is 10^6 times better - now uses the multiplexer to write tdf tracks that can (after conversion to binary with igvtools) can be loaded directly into igv.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5381 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 18:23:52 +00:00
a40a8006b5
Added in unit tests for the statistics calculated by the test runner; and bug-fixes to the calculations; so we have some assurance that the statistics coming out the back-end are correct.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5380 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 16:54:02 +00:00
c40efe1dea
Fixed exception for BAMs without filenames (unit tests, BAM input streaming,
...
etc.).
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5379 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-06 13:43:49 +00:00
ad51f30244
A trivial, but useful, sum of a list of integers
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5378 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-05 06:09:05 +00:00
9a8356892a
Cleaner error (really now just warnings) if you can't reach the S3 for logging
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5377 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-05 06:08:35 +00:00
10516f5de4
Fixed one low-memory sharder performance culprit: regions with no BAM data
...
whatsoever were misusing the Picard MergingIterator, triggering a re-traversal
through the entire contig.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5376 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-04 21:26:22 +00:00
337b54136f
2 fixes. For Mark: when insertions can be partially left-aligned, we were reading off the wrong bases. For GS post: the stored VariantContext.REFERENCE_BASE_FOR_INDEL_KEY needs to be updated when left-aligning because it can change.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5375 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-04 21:00:08 +00:00
b42005e7d7
Fixed issue where comp tracks with genotypes that didn't exactly overlap the eval track were getting dropped. Fixed issue where the 'row' column wasn't being output for things implementing TableType. This is an urgent patch for Mark - it'll break tests until I go back and update the md5s.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5374 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-04 16:51:12 +00:00
1861ca90fc
A change to the definition of CpG sites (is now, from 5' to 3' a CG dinucleotide in the reference, and the CpG site is at the C, rather than either at the C or a G).
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5373 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-04 15:36:07 +00:00
9ca1dd5d62
Miscellaneous changes:
...
- RefMetaDataTracker: grabbing variant contexts given a prefix (not sure where else this was implemented, if someone can show me I'll remove it)
- VCFUtils: grabbing VCF headers given a prefix
- MathUtils: Useful functions for calculating statistics on collections of Numbers
- VariantAnnotator: Made isUniqueHeaderLine a public static method -- maybe this should go into a different class. Not sure.
- Associations: PluginManager now used to propagate classes, implementations for Z,T,U tests, slight alteration to format to make the objects stored
in the window optionally different from those returned by whatever statistic is run across the window
Added:
- MannWhitneyU. Started to fix up WilcoxonRankSum but there are comments in there questioning the validity of some of the code, and I'm sure that
it's actually doing a U test. This implementation includes the direct calculation of p-values for small sample sizes, and a uniform approximation
for when one of the sample sets is small, and the other large. Unit tests to follow.
- BootstrapCallsMerger: takes n VCFs which have been called on the same samples; merges them together while averaging the annotations
- BootstrapCalls.q: qscript for testing the effectiveness of boostrap low-pass calling on the exome
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5372 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 22:43:36 +00:00
8ae42b70ac
give it an annotated VCF and a BAM and it creates a truth table on the validation of the VCF calls. This is just the first version, not ready for primetime.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5371 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 21:45:11 +00:00
f7ef35b8f5
Removing untrue comments in the GaussianMixtureModel
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5369 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 18:18:47 +00:00
9e12cd1312
Gotta include the changes i made to get an init function into the contexts
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5368 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 15:45:51 +00:00
835a26d145
A pass at a sample-normalized test. I think maybe all of them will simply do their own normalizing.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5367 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 15:42:25 +00:00
ef38fd1e0e
Major refactoring of association testing framework. New modules are now beyond trivial to implement. One hurdle remains which is how to deal with statistics that ought to be sample-normalized (e.g. depth, insert-size [when multiple libraries are used], and possibly others).
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5366 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 14:27:16 +00:00
5d4bbf41fb
Behave intelligently in the deepest levels of GATK record filtration when
...
we find a read flagged as 'mapped' in the unmapped region at the end of the
file.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5365 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 04:52:55 +00:00
7a22f19366
More descriptive error when VerifyingSamIterator hits an inconsistent alignment. Also updated
...
case UserException.MalformedBAM to match case of UserExceptio.MissortedBAM for consistency and
ease-of-use.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5364 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-03 03:55:24 +00:00
0181d95fe4
Intermediate optimization checkin. LinearExact model now about 10-20% faster than previous commit, by reorganizing and optimizing the if statements and genotype likelihood calculations. Next commit will include a banded implementation.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5362 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 22:01:35 +00:00
f0f4bc3363
This was busted because it assumed 1 (and only 1) record at each position. However it's possible to have 0 (which generated a NullPointer) or 2+ records (which dropped records).
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5361 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 21:35:50 +00:00
c152ef4339
Better error message for unknown reference file extension.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5359 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 17:52:16 +00:00
bef83b8b09
Bug fix: was tracking state across BAMs that should've been tracked per-BAM.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5358 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 17:32:06 +00:00
bafa61c1fe
LINEAR_EXACT now the default model. Passes all integration tests. 2-3x faster in low-pass data. Tests on exome data ongoing, but potentially vastly faster there.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5357 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 17:14:36 +00:00
8e1aa6059a
New mode for CombineVariants to assume the incoming VCFs have the same samples and disjoint calls. Drastically reduces the runtime for routine combining operations. Very useful with Queue.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5356 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 15:52:17 +00:00
5e4b321f86
Add hidden command-line argument for low-memory sharding.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5355 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 15:13:16 +00:00
ae42c0c7da
Bug fix based on GATK run report
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5354 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 14:18:12 +00:00
660998065b
'Okay, now I'm absolutely certain that there are no more bugs in the constrained writer.'
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5353 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 03:48:40 +00:00
880c607d79
Disable validation of linear index against original linear index process.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5352 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 01:51:26 +00:00
dc62685a2f
For Ryan: force creation of BAM index when no reads are present in the BAM
...
file. Temporary fix until Picard changes the behavior of indexing.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5351 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 01:50:42 +00:00
570186fa42
Added (deep) clone() and merge() to the RunningAverage utility class
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5350 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 00:35:23 +00:00
43567b7fe3
Load the linear index without forcing the index for the entire contig to be
...
loaded into memory.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5349 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-02 00:08:39 +00:00
a20ce1436d
A temporary @hidden hack to get indel calling done for Phase I: don't try to call if there's too much coverage. Do not use this unless your last name rhymes with Shmoplin.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5348 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-01 19:22:27 +00:00
3c7ae0d1a6
Special case handling of unmapped region in low memory sharder.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5346 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-01 17:38:30 +00:00
dd30ad751a
Fix bug in low memory sharder's interval accumulator.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5345 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-01 17:11:22 +00:00
d6145de970
More comprehensive tracking of position when bin trees are sparse.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5344 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-01 15:53:43 +00:00
bb969cd3a2
EMIT_ALL_SITES now does exactly that - even when there's no coverage or too many deletions
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5343 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-01 05:05:00 +00:00
0723b0f44c
Generalized association is now working. Output is in a horrific format. Implementation of T-testing. Improvements are to look for classes dynamically (a la VariantEval/VariantAnnotator), beautify output, and do optimizations where they exist.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5341 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-01 01:23:37 +00:00
ce34a8a918
New hidden option in VQSR to not parse the genotypes of the incoming training data. Updated VQSR training in methods development pipeline to be more in line with best practices.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5340 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-28 23:19:51 +00:00
e7089f9870
Fix for particularly small, isolated intervals: make sure the bounds of the
...
bin tree are dictated by the lowest bin level, whether it exists or not.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5339 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-28 22:35:53 +00:00
c869d1c9cf
Fix misc issues in new protosharder regarding proper iterator termination when
...
an unexpectedly small amount of data is present.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5338 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-28 21:14:18 +00:00
5ac9af472c
Adding performance test for case with very high coverage (> 600,000x) over an interval
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5336 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-28 19:48:56 +00:00
e75366f738
Fixed performance issue in protosharding code -- turns out that the index
...
optimizer was mutating the data stored in the indices. Protosharding still
disabled by default.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5334 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-28 17:32:12 +00:00
8de83725f9
Simple walker to randomly break VCF files into (potentially unequal) subsets. Useful for e.g. cutting hapmap into training and evaluation sets.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5333 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-28 16:51:46 +00:00
d059d89a9d
Fixes and cleanups for indel eval module. Also outputs AT/CG ratio in dedicated column in IndelStatistics.
...
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5332 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-28 12:07:50 +00:00
kshakir
chartl
depristo
hanna
ebanks
kiran
carneiro
rpoplin
asivache
delangel