c6d8df8639
queueJobReport is a public feature of Queue
2011-08-29 17:20:54 -04:00
1e5001b447
Merge branch 'master' of ssh://gsa1/humgen/gsa-scr1/gsa-engineering/git/unstable
2011-08-29 17:04:21 -04:00
3af001fff2
Bugfix for file that must not exist on disk
2011-08-29 17:00:10 -04:00
3b09d42ed6
Now only prints 1 warning message about duplicate headers in simpleMerge
2011-08-29 14:41:29 -04:00
c2f0db969b
Don't use the default deletion value from UG if not asking to have it set
2011-08-29 13:48:10 -04:00
bb7a37e8f2
We need to allow reference calls in the input VCF for the GenotypeAndValidate walker when using the BAM as truth so that we can test supposed monomorphic calls against the truth.
2011-08-29 13:19:35 -04:00
bc252a0d62
misc minor bug fixes in assembly. Increasing the minimum number of bad variants to be used in negative model training in the VQSR
2011-08-29 08:11:31 -04:00
61633c95a8
Default jobreport is now jobPrefix, so you see logs like Q-2508.jobreport.txt
2011-08-28 19:19:45 -04:00
a5c65fc133
Debugging information to print out the Query tracks
2011-08-28 18:54:49 -04:00
b38de1fa35
Now captures the exechost in the job report
...
-- Works for in process, shell, and LSF runners
-- Cleanup of debugging output
2011-08-28 12:05:56 -04:00
7bf006278d
Moved ResolveHostname to general utils as a static function
2011-08-28 12:04:16 -04:00
ccec0b4d73
AnalyzeCovariates uses the general RScript system now
...
-- Convenience constructor for collection for testing
-- callRScript() now accepts Objects not Strings, for convenience
2011-08-27 12:54:13 -04:00
1ceb020fae
UnitTests for RScript
2011-08-27 10:50:05 -04:00
e37a638e09
Fix for disallowed characters in GATKReportTable
...
-- Illegal characters are automatically replaced with _
2011-08-26 13:24:06 -04:00
0cb1605df0
Clean documentation for JobRunInfo
2011-08-26 09:22:58 -04:00
415d5d5301
LSF long times are in seconds, convert to milliseconds to meet standard
2011-08-26 09:18:28 -04:00
c0503283df
Spelling fix requires md5 updates
2011-08-26 07:40:44 -04:00
eef1ac415a
Merge branch 'master' into rodTesting
...
Conflicts:
public/java/src/org/broadinstitute/sting/gatk/walkers/variantutils/VariantsToTable.java
2011-08-26 00:35:41 -04:00
9b7512fd94
Just because there's a ref base doesn't mean the VC needs to be padded
2011-08-25 22:42:14 -04:00
e03dfdb0ab
Automatic iteration field addition works properly.
2011-08-25 16:59:02 -04:00
e01273ca7c
Queue now writes out queueJobReport.pdf
...
-- General purpose RScript executor in java (please use when invoking RScripts)
-- Removed groupName. This is now analysisName
-- Explicitly added capability to enable/disable individual QFunction
2011-08-25 16:57:11 -04:00
09a729da3a
Removing incorrect comment
2011-08-25 15:42:52 -04:00
8bbef79fc2
Create clipped alleles during allele parsing instead of creating a full VC, clipping alleles, and regenerating the VC from scratch.
2011-08-25 15:37:26 -04:00
0f4be2c4a4
Argument to disable queueJobReport entirely
...
-- Minor improvements to RodPerformanceGoals
2011-08-25 13:32:03 -04:00
d65faf509c
Default output name for Queue JobReport is queue_jobreport.gatkreport.txt
2011-08-25 13:15:20 -04:00
a7d6946b22
Refactored QJobReport and QFunction, which is now automatically tracked
...
-- All QFunctions, including sg ones, are tracked
-- Removed memory information
2011-08-25 13:13:55 -04:00
16caca0822
BLASR BAMs and new BWA parameters
...
*Added the functions to turn a BLASR generated BAM file into a usable BAM file.
*Modified the bwa parameters according to test results from NA12878 pb2k dataset.
2011-08-24 17:04:07 -04:00
e3f5d7067a
Added ReorderSam queue binding
2011-08-24 17:03:11 -04:00
08fb21f127
Removing hostname
2011-08-24 16:45:50 -04:00
dc8398e165
fixing bai output for indel cleaning.
2011-08-24 15:58:34 -04:00
06e30a81d1
Fixes throughout for getting job information
...
-- no more hostname -- it's just not going to be important
2011-08-24 15:30:09 -04:00
29c7b10f7b
Merge branch 'master' of ssh://nickel.broadinstitute.org/humgen/gsa-scr1/gsa-engineering/git/unstable
2011-08-24 15:18:58 -04:00
e5008aba00
Output the top two haplotypes as a variant call by running smith-waterman alignment against the reference and calling any difference as variation. This is the first verion that runs end-to-end by taking in reads as bam file and writing out variant calls in VCF.
2011-08-24 15:18:44 -04:00
4918519a58
No more NPE in getRuntime() when you cntr-c out of Queue
2011-08-24 14:14:01 -04:00
16d8360592
QJobReport is now the official capability name
2011-08-24 13:59:14 -04:00
d047c19ad1
Writes output to file
2011-08-24 13:52:05 -04:00
3ae68e2397
JobLogging trait now writes out GATKReport log of jobs
2011-08-24 13:36:39 -04:00
e618cb1e79
a) Renamed/expanded SelectVariants arguments that choose particular kinds of variants and particular allelic types, now instead of -Indels or -SNPs we can specify for example -selectType [MIXED|INDEL|SNP|MNP|SYMBOLIC]. To select biallelic, multiallelic variants, use -restrictAllelesTo [BIALLELIC|MULTIALLELIC]. Corresponding gatkdocs changes.
...
b) More useful AC,AF logging in VariantsToTable with multiallelic sites: instead of logging comma-separated values, log max value by default. Hidden, experimental argument -logACSum to log sum of ACs instead. This is due to extreme slowness of R in parsing strings to tokens and computing max/sum itself (~100x slower than gatk).
c) Added integrationtest for new SelectVariants commands
2011-08-24 12:25:50 -04:00
cd12f7f286
Fixed list dependency
...
Instead of creating a bam list file, I dynamically create a scala list and pass as parameters. This way the intermediate bam files don't get deleted before they should.
2011-08-24 11:12:46 -04:00
219252a566
Adapting to the new RodBinding framework
2011-08-24 11:12:46 -04:00
28ee6dac41
Fixed spelling mistake
2011-08-24 10:14:45 -04:00
f37875600a
Merge branch 'master' of ssh://nickel.broadinstitute.org/humgen/gsa-scr1/gsa-engineering/git/unstable
2011-08-24 09:02:44 -04:00
1ecbf05aae
Avoid segfaults due to out of date and possibly abandonded LSF DRMAA implementation when use'ing LSF instead of .combined_LSF_SGE
2011-08-23 23:49:36 -04:00
b8bc03bb42
JobRunInfo improvements
...
-- dry-run now adds some info, for testing
-- InProcessRunner adds some, but not all, of the information we want
2011-08-23 17:11:22 -04:00
569e1a1089
Walker.isDone() aborts execution early
...
-- Useful if you want to have a parameter like MAX_RECORDS that wants the walker to stop after some number of map calls without having to resort to the old System.exit() call directly.
2011-08-23 16:53:06 -04:00
31ec6e316c
First implementation of JobRunInfo
...
-- onExecutionDone(Map(QFunction, JobRunInfo)) is the new signature, so that you can walk over your jobs and inspect their success/failure and runtime characteristics
2011-08-23 16:51:54 -04:00
a1a1fac9e4
Likelihood engine now gives non-zero likelihoods. Using HMM function that can handle context specific gap open and gap continuation penalties
2011-08-23 13:43:07 -04:00
6e2552a9ef
Merge fix
2011-08-23 12:40:43 -04:00
8b7a0b3b62
Two new arguments to SelectVariants to exclude either multiallelic or biallelic sites from input vcf
2011-08-23 12:40:01 -04:00
ac36271457
Fixed extra reads showing up in Variable Sites
...
Reads that were not hard clipped for the variable site no longer show up in output file
Walker now uses unclippedStart of Read to determine position in the sliding Window
2011-08-23 11:26:00 -04:00
6d6feb5540
Better error message when you cannot determine a ROD type because the file doesn't exist or cannot be read
2011-08-23 10:56:37 -04:00
a9ba945595
onExecutionDone(jobs, successFlag) added to QScript.
...
-- This function is called when the Qscript ends, so scripts can overload this function if they want to run some code after all of the jobs have completed
2011-08-23 10:09:51 -04:00
feeab6075f
Merging ReduceReads development with unstable repo
...
It is time to bring the ReadClipper class to the main repo. Read Clipper has tested functionality for soft and hard clipping reads. I will prepare thorough documentation for it as it will be very useful for the assembler and the GATK in general.
2011-08-22 23:03:03 -04:00
ee68713267
Further Bug fixes to CountVariants: stratifications were wrong in case genotypes had no-calls, for example if we stratified by sample and a sample had a no-call, this no-call was considered a true variant and counts were incorrectly increased
2011-08-22 20:42:47 -04:00
c270384b2e
Merge branch 'master' of ssh://nickel.broadinstitute.org/humgen/gsa-scr1/gsa-engineering/git/unstable
2011-08-22 20:39:32 -04:00
8ae24912f4
a) Misc fixes in Phase1 indel vqsr script,
...
b) More R-friendly VariantsToTable printing of AC in case of multiple alt alleles
c) Rename FixPLOrderingWalker to FixGenotypesWalker and rewrote: no longer need older code, replaced with code to replace genotypes with all-zero PL's with a no-call.
2011-08-22 20:39:06 -04:00
136f0eb685
Creating sample-bam list instead of joining
...
This should save us at least one day in the trio decoy processing.
2011-08-22 18:03:39 -04:00
04d8bcaf19
Fixed bai removal on picard tools
...
BAM index files were not being deleted because picard replaces the name of the file with bai instead of appending to it.
2011-08-22 18:03:39 -04:00
8aed151a71
Created RevertSam queue class
...
Class for the picard tool RevertSam with all the options for queue scripts.
2011-08-22 18:03:39 -04:00
caebc88e9a
Consensus mode and new RodBinding framework.
...
The DPP was not using the parameter correctly. It didn't matter for the default option (which is the only one we have been testing) but it would not work for knowns only or smith waterman. It is fixed now.
It now complies with the new rod binding framework.
2011-08-22 18:03:39 -04:00
85c5a6f890
Merge branch 'rodTesting'
...
Conflicts:
private/java/src/org/broadinstitute/sting/gatk/walkers/performance/ProfileRodSystem.java
2011-08-22 17:43:47 -04:00
1eab9be35d
Now with accurate javadoc
2011-08-22 17:25:15 -04:00
3612a3501d
info, not warn, about dynamic type determination
2011-08-22 17:24:51 -04:00
dc42571dd9
Only create the genotype map when necessary
2011-08-22 15:40:36 -04:00
c4c90c8826
Updates to JobRunners from the Queue developer community and from running the WholeGenomePipeline:
...
- Ability to pass a different resident memory reservation and limits. Useful for large pileups of low pass genome data that sometimes need high -Xmx6g but usually don't exceed 2-3g in actual heap size.
- Fixed jobPriority to work for all job runners. Now must be a integer between 0 and 100- even for GridEngine- and will be mapped to the correct values.
- Passing parallel environment and job resource requests to LSF and GridEngine. Useful for passing tokens like iodine_io=1 and -pe pe_slots 8
- Refactored GridEngine JobRunner to also provide basic support for other job dispatchers with DRMAA implementations such as Torque/PBS. Should work for basic running but advanced users must pass their own jobNativeArgs from the command line or in customized QScripts until someone maps properties like jobQueue, jobPriority, residentRequest, etc. into a Torque/PBS/etc. dispatcher.
2011-08-22 15:13:27 -04:00
2c24b68a96
Working implementation of DecodeLoc for VCF parsing. Makes indexing 3x faster.
2011-08-22 15:11:21 -04:00
518b3dd291
Don't let the genotypes map be null
2011-08-22 15:10:30 -04:00
f93a554b01
updating exome specific parameters in MDCP
2011-08-21 10:25:36 -04:00
dbff84c54e
Merge branch 'master' of ssh://nickel.broadinstitute.org/humgen/gsa-scr1/gsa-engineering/git/unstable
2011-08-21 10:09:19 -04:00
22ca44c015
Fixed Queue's tagging of RodBindings.
...
Fixed argument definition names.
2011-08-21 02:34:20 -04:00
a8cbced71b
Bug fix for Ryan: check for no context
2011-08-20 22:49:51 -04:00
0ccd173967
Fixing the recent SelectVariants fix
2011-08-20 21:30:08 -04:00
b008676878
fixing the previous fix
2011-08-20 21:21:55 -04:00
782453235a
Updated VariantEvalIntegrationTest since there's a new column separating nMixed and nComplex in CountVariants
...
Misc updates to WholeGenomeIndelCalling.scala
Bug fix in VariantEval (may be temporary, need more investigation): if -disc option is used in sites-only vcf's then a null pointer exception is produced, caused by recent introduction of -xl_sf options.
2011-08-20 12:24:22 -04:00
539e157ecd
Fixing misc parameters in MDCP. The pipeline now does VariantEval of output by default. Fix for NaN vqslod values in VQSR
2011-08-20 11:28:48 -04:00
4939648fd4
Merge branch 'master' of ssh://nickel.broadinstitute.org/humgen/gsa-scr1/gsa-engineering/git/unstable
2011-08-20 08:50:43 -04:00
a96ecbab71
Merge branch 'master' of ssh://nickel.broadinstitute.org/humgen/gsa-scr1/gsa-engineering/git/unstable
2011-08-19 19:30:05 -04:00
ddb5045e14
Updating the methods development calling pipeline for the new rod binding syntax and the new best practices.
2011-08-19 19:29:51 -04:00
ff018c7964
Swapped argument order but not MD5 order
2011-08-19 16:55:56 -04:00
8b3cfb2f1c
Final documented version of GATKDoclet and associated classes
...
-- Docs on everything.
-- Feature complete. At this point only minor improvements and bugfixes are anticipated
2011-08-19 16:52:17 -04:00
b08d63a6b8
Documentation and code cleanup for ClipReads, CallableLoci, and VariantsToTable
...
-- Swapped -o [summary] and -ob [bam] for more standard -o [bam] and -os [summary] arguments.
-- @Advanced arguments
2011-08-19 15:06:37 -04:00
49e831a13b
Should have checked in
2011-08-19 14:35:16 -04:00
7b5fa4486d
GenotypeAndValidate - Added docs to the @Arguments
2011-08-19 13:35:11 -04:00
9f7d4beb89
Merge branch 'help'
2011-08-19 13:14:02 -04:00
4d1fd17a97
GATKDoclet cleanup and documentation
...
-- Fixed bug in the way ArgumentCollections were handled that lead to failure in handling the dbsnp argument collection.
2011-08-19 13:13:41 -04:00
0f25167efd
minor fix in VariantEval docs
2011-08-19 11:01:04 -04:00
198955f752
GATKDoc descriptions for all standard codecs, or TODO for their owners
...
-- Also added vcf.gz support in the VCF codec. This wasn't committed in the last round, because it was missed by the parallel documentation effort.
2011-08-19 09:57:21 -04:00
269ed1206c
Merge branch 'master' of ssh://nickel.broadinstitute.org/humgen/gsa-scr1/gsa-engineering/git/unstable
2011-08-19 09:32:20 -04:00
bc902e8421
GZIP VCF for testing
2011-08-19 09:05:39 -04:00
a5e279d697
Dynamic typing of vcf.gz files
...
-- CombineVariantsIntegrationTests now use dynamic typing of vcf.gz files
-- FeatureManagerUnitTests tests for correctness.
2011-08-19 09:05:11 -04:00
40e67cff1b
I like the @Advanced annotation
2011-08-18 22:27:34 -04:00
2457c7b8f5
Merge branch 'master' into help
2011-08-18 22:20:43 -04:00
5fbdf968f7
ArgumentSource no longer comparable. Arguments sorted by GATKDoclet
2011-08-18 22:20:14 -04:00
77fa2c1546
Renaming read filters with a superfluous 'Read' in their names. Kept the ones that made sense to have it (e.g. MalformedReadFilter).
2011-08-18 22:01:33 -04:00
1d3799ddf7
Merge branch 'master' into help
2011-08-18 22:00:29 -04:00
d1892cd0d7
Bug fixes
...
-- Sorting of ArgumentSources now done in GATKDoclet, not in the ParsingEngine, as the system depends on the LinkedTreeMap
-- Fixed broken exception throwing in the case where a file's type could not be determined
2011-08-18 21:58:36 -04:00
c5efb6f40e
Usability improvements to GATKDocs
...
-- ArgumentSources are now sorted by case insensitive names, so arguments are shown in alphabetical order (Ryan)
-- @Advanced annotation can be used to indicate that an argument is an advanced option and should be visually deemphasized in the GATKs. There's now an advanced section. Mauricio or Ryan -- could you figure out how to make this section less prominent in the style.css?
2011-08-18 21:39:11 -04:00
d94da0b1cf
Moved CG and SOAP codecs to private
2011-08-18 21:20:26 -04:00
f7414e39bc
Improvements to GATKDocs
...
-- Allowed values for RodBinding<T> are displayed in the GATKDocs
-- Longest name up to 30 characters is chosen for main argument list (suggested by Ryan/Mauricio)
-- Features are listed in alphabetical order
-- Moved useful getParameterizedType() function to JVMUtils
-- Tests of these features in the Documentation Test
2011-08-18 21:20:09 -04:00
09d099cada
Added GATKDocs to the UnifiedGenotyper.
2011-08-18 20:57:02 -04:00
Mark DePristo
Eric Banks
Ryan Poplin
Mauricio Carneiro
Guillermo del Angel
Khalid Shakir
Roger Zurawicki