Commit Graph

54 Commits (90aef66ec5f485b647bef4afc9aa63809eebcfb2)

Author SHA1 Message Date
kshakir 3aedd0055e Updated firehose clean bam pipeline to pull firehose info and push back firehose clean bam.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4088 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-23 20:38:42 +00:00
kshakir 51678d48e4 Using job ids instead of job names for LSF dependency tracking.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4071 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-19 23:42:06 +00:00
kshakir 88ca1fb22c Lazy loading reflections so Queue can hack the classpath before the PluginManager looks for classes.
Removed extra quotes from 'cd' pre-exec command.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4067 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-19 20:29:52 +00:00
chartl 0028b884d8 Reformatting and tweaks to the end-to-end pipeline
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4066 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-19 20:29:48 +00:00
aaron 3dc4d3c3a9 removing the custom reflections library from the libs, and adding a release version. Hopefully this will fix the problem Menachem has been seeing with random JVM crashes. Also
removed the auto-deletion of the reflections jar, and removed the very old OmniPlan document we had checked-in.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4056 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-19 00:42:37 +00:00
kshakir 618c69f8dc More updates to the CleanBamFile pipeline.
Added the a CommandLineFunction.jobDependencies that will explicitly force a function to wait for a file, even if the value isn't otherwise listed on an @Input.
More bug fixes and refactoring of functions.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4048 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-17 14:59:42 +00:00
chartl 3a4977c75e Re-add the 1KG trigger as a comp as well
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4045 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-16 18:19:47 +00:00
depristo c85ab9db37 functional recalibrate script
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4034 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-14 16:01:37 +00:00
kshakir 307c8ca027 Created a new playground script for cleaning bams in Firehose.
Some refactoring of Queue extensions for reusability in scripts.
Putting the extensions into the Queue.jar after building them.
More updates to GATK walker arguments specifying @Input and @Output for Queue.

git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4032 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-13 23:52:24 +00:00
kshakir 8e46d5de04 Printing to INFO where to find the job output files.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4029 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-13 19:26:53 +00:00
kshakir 542d394e09 Cleaning up Queue debugging output.
-l DEBUG with local programs now prints out the stdout/stderr of the programs as they are run.
More documentation in the examples with a new even simpler CountReads example.
Took out unused option to build Queue GATK extensions separately.

git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4025 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-13 15:54:08 +00:00
kshakir f39dce1082 Exposed CommandLineFunction defaults to the Queue.jar command line (see -help).
Added ability to skip up-to-date jobs where the outputs are older than the inputs.
Changed -T CountDuplicates --quiet to --quietLocus so that Queue GATK extensions can use both short and full argument names.
Short names can be used to set values on Queue GATK extensions, for example: vf.XL :+= myFile
Moved Hidden from the GATK to StingUtils.
Updated ivy from 2.0.0 to 2.2.0-rc1 to fix sha1 issue: http://bit.ly/aX72w7
Added Queue to javadoc and testing build targets.
Added first Queue unit test.
Another pass at avoiding cycles in the DAG thanks to all function I/O being files.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4017 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-11 21:58:26 +00:00
depristo cd2d051209 full path to Rscript
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3999 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-10 12:02:38 +00:00
depristo 9b432d0801 1kg script now works
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3998 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-10 12:01:18 +00:00
kshakir 4f51a02dea Changed logging level to default at INFO instead of WARN.
Changes to StingUtils command line for use in Queue, replacing Queue's use of property files.
Updates to walkers used in existing QScripts to add @Input/@Output.
RMD used in @Required/@Allows now has a new default equal to "any" type.
New QueueGATKExtensions.jar generator for auto wrapping walkers as Queue CommandLineFunctions.
Added hooks to modify the functions that perform the Scattering and Gathering (setting their jar files, other arguments, etc.)
Removed dependency on BroadCore by porting LSF job submitter to scala.
Ivy now pulls down module dependencies from maven.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3984 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-09 16:42:48 +00:00
chartl 5815348ebc Switch to newer version of comp tracks (and make the trigger track a comp as well). Indel cleaning should override the interval list and only use the contig interval list; and also force jobs to go to long.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3941 348d0f76-0448-11de-a6fe-93d51630548a
2010-08-04 20:05:27 +00:00
chartl 9132c98eec Slightly smarter interval list dealing (whole exome intervals are .interval_list, whole genome are .interval.list). Also use BTI with the Genomic Annotator
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3904 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-29 22:04:02 +00:00
chartl 54d93f63d2 Hacky fix for LSF confusion -- submitted jobs check to see if their directory exists, despite depending on the job which creates said directory. Filter strings now have escaped quotes.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3903 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-29 21:35:50 +00:00
chartl 0f9baa2e94 Ha ha ha ha ha
:(



git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3902 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-29 20:48:35 +00:00
kshakir 735ef19dc8 Added option to sleep after creating temporary directories.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3900 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-29 19:53:17 +00:00
kshakir 82c37fceb5 Create intermediate directories and don't error if the directory already exists.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3899 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-29 19:45:12 +00:00
chartl 7a5ee485d2 Full pipeline now works through DAG creation. First draft; more work to do to make it cleaner and better command-line input handling (and properties handling); but the DAG is rendered and looks good.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3898 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-29 19:36:17 +00:00
chartl 4d4cf6e1dc Updates to calling pipeline
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3896 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-29 18:37:20 +00:00
chartl 62a9217a61 A brute-force exome/genome independent end-to-end cleaning/calling pipeline using Queue
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3894 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-29 13:17:14 +00:00
chartl f35e6d73b4 Actually name the class the name of the file. (Clearly created by cp)
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3892 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-28 15:18:04 +00:00
chartl cd9395fa14 Since Picard's FixMateInformation merges, fixes mates, and sorts, allow it to be used as a gather function.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3891 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-28 15:09:19 +00:00
depristo 25a27b78bc 1KG Table 1 counting pipeline. Useful example
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3819 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-17 22:30:56 +00:00
depristo b0fc42906e Better DOT support and updated recalibration pipeline
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3811 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-16 20:54:51 +00:00
depristo 81eef0d993 DOT visualization with Queue. More sophisticated recalibation queue script with scatter/gather
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3799 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-15 22:32:48 +00:00
depristo 6bf5df4eb5 Better merge command
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3797 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-15 17:02:47 +00:00
depristo 530a320f28 Intermediate commit of scatter/gather recalibation pipeline
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3785 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-13 22:46:08 +00:00
kshakir 1d399aa2f3 Added a temporary gatkLoggingLevel field to the soon to be obsolete GatkFunction while finishing up the delayed generic gatk walker utility.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3757 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-11 03:27:32 +00:00
kshakir 7be8c35eb2 Workaround for scala trait erasing parameterized types:
- Requiring explicit @ClassType on parameterized fields in traits.
- Scatter / Gather functions are now abstract classes since @ClassType can't be used on parameterized fields with type parameters.

git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3726 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-07 03:15:10 +00:00
rpoplin 87470d5fe5 Checking in a simplistic VR qscript file for posterity's sake
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3705 348d0f76-0448-11de-a6fe-93d51630548a
2010-07-01 18:53:17 +00:00
kshakir a46e22ed13 Refactored ArgumentDefinition to absorb functionality from ArgumentDefinition and ArgumentTypeDescriptor.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3689 348d0f76-0448-11de-a6fe-93d51630548a
2010-06-30 18:55:57 +00:00
kshakir 178cf64a0c Refactored ArgumentDefinition to absorb functionality from ArgumentDefinition and ArgumentTypeDescriptor.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3688 348d0f76-0448-11de-a6fe-93d51630548a
2010-06-30 18:37:58 +00:00
kshakir dce2c17404 Added "-bsubWait" where Queue waits for all the jobs to exit before exiting.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3661 348d0f76-0448-11de-a6fe-93d51630548a
2010-06-28 19:52:17 +00:00
kshakir c047232b18 Using picard for bam merging.
Properties now propagate to scatter/gather functions.

git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3659 348d0f76-0448-11de-a6fe-93d51630548a
2010-06-28 17:59:09 +00:00
kshakir 894ad354fa Fixed typo in the name of the shell directory.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3644 348d0f76-0448-11de-a6fe-93d51630548a
2010-06-25 20:59:40 +00:00
kshakir 75c98c42b8 Started path of deprecation of Sting's @Argument by splitting the annotation into @Output and @Input. Anything that's not an @Output should be an @Input.
Checked in example qscripts that are basically todo integration tests.
Replaced use of queue @Input/@Output with Sting's new @Input/@Output.  This means you'll now have to doc-ument the annotations.
More work on dependency resolution cycles being created in the graph during scatter/gather.
Filtering nulls to avoid NPE exceptions in scala's 'Collection'.hashCode.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3643 348d0f76-0448-11de-a6fe-93d51630548a
2010-06-25 20:51:13 +00:00
kshakir ce27ed0d60 Added missing @ClassType to memory limits.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3641 348d0f76-0448-11de-a6fe-93d51630548a
2010-06-25 18:56:35 +00:00
kshakir 30cf78fdc0 Refactoring for a first version of scatter gather api with basic shell script implementations.
Modified build script so that queue is cleaned during "ant clean".



git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3611 348d0f76-0448-11de-a6fe-93d51630548a
2010-06-22 18:39:20 +00:00
kshakir c44fd05aa1 Fix for a reflection issue with generic types.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3565 348d0f76-0448-11de-a6fe-93d51630548a
2010-06-16 15:58:38 +00:00
kshakir 32fc221ffe Replaced pattern matched pipeline spec with annotated objects.
Old version is no longer available.

git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3558 348d0f76-0448-11de-a6fe-93d51630548a
2010-06-15 04:43:46 +00:00
kshakir beb8a83ca4 Multiple inputs / outputs for rules.
Cleanup.

git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3464 348d0f76-0448-11de-a6fe-93d51630548a
2010-05-30 08:28:16 +00:00
kshakir e96bcf2128 GSA-333 Replaced all accidental usages of getCanonicalPath (which should never be used) with getAbsolutePath.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3463 348d0f76-0448-11de-a6fe-93d51630548a
2010-05-28 22:27:05 +00:00
kshakir 0e3c76ab84 GSA-333 Meant to use getAbsolutePath and not deref symlinks.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3449 348d0f76-0448-11de-a6fe-93d51630548a
2010-05-27 21:23:02 +00:00
depristo 99aae4ac0b ended not done for LSF queue submission.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3446 348d0f76-0448-11de-a6fe-93d51630548a
2010-05-26 22:16:48 +00:00
kshakir ed4d8ddd05 Refactoring and sugar to give lower level access to the job graph.
Will add more sugar / glue depending on how much of a graph the python generator outputs.

git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3435 348d0f76-0448-11de-a6fe-93d51630548a
2010-05-25 22:52:29 +00:00
kshakir e9ee55d7dd A cleaned up functioning early, early access version of Queue for others to play with and provide feedback about next steps.
Current version only has syntatic sugar for accessing the graph via rules ex. "bam" -> "bam.bai", "samtools index ${bam}" and DOES NOT have sugar for constructing your own graph.
Usage info on the internal wiki at https://iwww.broadinstitute.org/gsa/wiki/index.php/Queue


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3420 348d0f76-0448-11de-a6fe-93d51630548a
2010-05-23 20:21:09 +00:00