Commit Graph

151 Commits (311dfa09989b2e98a0b1b6f4c063158e2e202d87)

Author SHA1 Message Date
depristo 311dfa0998 Now builds examples, as I expected. GATKPaperGenotyper lives again.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5953 348d0f76-0448-11de-a6fe-93d51630548a
2011-06-07 00:13:44 +00:00
kshakir 77cae39c8e Step towards tribble precompiled jar, support in build.xml for source with fallback to the checked in jar.
Current tribble-129M.jar in SVN does not work with current version of GATK code.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5897 348d0f76-0448-11de-a6fe-93d51630548a
2011-05-27 21:04:27 +00:00
depristo a02ab4028b Build for tests now includes path to contracts
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5840 348d0f76-0448-11de-a6fe-93d51630548a
2011-05-22 19:44:49 +00:00
depristo 0095aa2627 Contracts for java now enabled by default in GATK build. The contract checking is automatically enabled when running tests and integrationtests. If you want to run the GATK with Contract checking enabled, add -javaagent:lib/cofoja.jar to your jvm args
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5826 348d0f76-0448-11de-a6fe-93d51630548a
2011-05-20 02:53:42 +00:00
depristo 218354e338 Contracts for Java (http://code.google.com/p/cofoja/) infrastructure enabled. No piece of code actually uses this, so it's possible to remove easily. Does not build by default (you must modify build.xml). Really an intermediate commit so I can play around with the system in my java classes and revert safely. Very much looking forward to DVCS
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5815 348d0f76-0448-11de-a6fe-93d51630548a
2011-05-18 18:05:59 +00:00
kshakir ec443e89cf Added pass-throughs for -Djava.io.tmpdir to javac and testng.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5791 348d0f76-0448-11de-a6fe-93d51630548a
2011-05-11 20:56:35 +00:00
hanna c2e8c460cb Factor out all testing dependencies into a separate test configuration and
only download that test configuration when running unit/integration tests.
This means that the build will (hopefully) never break because it can't
fetch a file that isn't required for the GATK to run.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5775 348d0f76-0448-11de-a6fe-93d51630548a
2011-05-05 22:42:11 +00:00
kshakir f7d9f0a1f3 Removing QPipeline directory as there's no one to support it at the moment.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5757 348d0f76-0448-11de-a6fe-93d51630548a
2011-05-04 18:36:02 +00:00
kshakir 08f0509a5c Disabling the queue/pipeline package by default so that scala code can build. If it's not going to be fixed the package should be removed. If it is going to be fixed this patch to build.xml should be reverted.
Also added the old model of indel calling to the FCP.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5749 348d0f76-0448-11de-a6fe-93d51630548a
2011-05-04 12:17:33 +00:00
kshakir 0a58d7aa1a Marked boolean SAMFileWriterATD arguments as flags so scala generator maps them to Boolean instead of Option[Boolean].
Using the VCFWriterATD isCompressed to check if the VCF index will be auto generated.
Tracking BAM and Tribble indexes as @Inputs and @Outputs in generated QFunctions.
Updates to the BamGatherFunction to disable the index during merge when disable_bam_indexing = true.
Made a shortcut for live-running pipelinetest, pipelinetestrun.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5606 348d0f76-0448-11de-a6fe-93d51630548a
2011-04-08 18:44:32 +00:00
droazen cb3e8aec5e Modified the buildfile and help extractor doclet so that help text is only
extracted from source files that have been modified since the help resource
file was last generated. This significantly speeds up builds where only a few 
source files have been modified, at the expense of making clean builds take 
slightly longer. Here's some performance data gathered by testing the old and 
new versions of extracthelp in isolation and averaging across 10 runs:
old extracthelp, 1 modified source file: 20.1 seconds
new extracthelp, 1 modified source file: 7.2 seconds <-- woohoo! :)
old extracthelp, clean build: 17.8 seconds
new extracthelp, clean build: 20.5 seconds


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5590 348d0f76-0448-11de-a6fe-93d51630548a
2011-04-07 18:40:53 +00:00
depristo 11822da578 Stand alone, GATK dependent tool that Reads a list of BAM files and slices all of them into a single merged BAM file containing reads in overlapping chr:start-stop interval. Highly efficient when working with thousands of BAM files. Can merge 1MB of sequence of 1600 4x BAMs in 4g in only 2 hours.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5558 348d0f76-0448-11de-a6fe-93d51630548a
2011-04-02 13:41:29 +00:00
kshakir d5ac822e97 When @Gather annotation is missing (probably due to an unclean build) printing out the full field+class name for debugging purposes.
Custom gatherer prints out the class name in the logs.
Try to retrieve mail domain from /etc/mailname before falling back to the hostname.
Building oneoff jars during ant oneoffs.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5540 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-30 21:43:37 +00:00
depristo 18002abb1f Build.xml contained references to tools now in picard
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5463 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-17 18:29:46 +00:00
kshakir 272fab582f Added another erroronmissingdir="false" for packging external dirs.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5445 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-14 21:26:28 +00:00
kshakir 93de326066 Added a new @PartitionBy for walkers to specify how to cut up their inputs.
Now building all javadoc.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5428 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-12 01:33:08 +00:00
kshakir 2058fc12bc Small bug fixes:
Added a property to allow generating Queue extensions around an external version of the GATK.
Updated argument order in -help.
Restored the ability to use QScript trait imported annotations in constructors.
Removing line feeds from email password files.

git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5416 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-10 19:54:14 +00:00
kshakir 63e1625cc5 Instead of just looking for $SGE_ROOT specifically hunting for $SGE_ROOT/lib/drmaa.jar to enable early grid engine builds.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5407 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-08 18:27:08 +00:00
kshakir c03341aec1 JobRunner can be specified on the command line. -bsub is currently short form of -jobRunner Lsf706.
Added an empty wrapper for a GridEngine job runner which is only activated when SGE_ROOT is detected.
Refactored a bit more common code into CommandLineJobRunner / JobRunner / FunctionEdge.
Status for analyisNames now includes the number of functions in state.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5399 348d0f76-0448-11de-a6fe-93d51630548a
2011-03-07 23:52:48 +00:00
kcibul d9ea7daa73 fixed problem with Matt re: packaging commands from external walker codebases
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5311 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-24 21:11:38 +00:00
depristo 1a5d296737 ReplaceReadGroups. Fixes BAM files without read group info. MissingReadGroup points people to this tool now. Please point users on the forum to this tool now. Will migrate to Picard.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5284 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-21 14:02:41 +00:00
depristo 7323b05dc1 A simple utility for reordering a BAM file based on a new reference sequence. This tool can be used to efficiently correct a lexicographically sorted BAM file
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5280 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-19 23:24:42 +00:00
kshakir 49931b12fb Ignore missing external directory, take two.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5263 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-17 20:00:00 +00:00
kshakir 188c4f67b0 Ignore missing external directory.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5262 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-17 19:22:21 +00:00
hanna fb9f92d09c For Kristian...bug fixes for mechanism allowing external source
directory to live anywhere on the filesystem.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5259 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-17 18:35:27 +00:00
hanna b992abb6eb A few more unit tests plus some extra
functionality for BAM index visualization.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5222 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-09 01:51:34 +00:00
kshakir b1ff371c8f Peek inside the StingText.properties to make sure the version in the properties matches the build version.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5185 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-03 18:07:55 +00:00
kshakir 5279ef24a3 Fixed typo that kept scala tests from recompiling.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5153 348d0f76-0448-11de-a6fe-93d51630548a
2011-02-01 02:41:07 +00:00
depristo 91824f478e FASTQ directory is gone
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4986 348d0f76-0448-11de-a6fe-93d51630548a
2011-01-13 15:16:06 +00:00
kshakir 8ba3a5a43f Command lines for locally run Queue jobs no longer have to be escaped differently than bsub'ed jobs.
GSA-410 Local job runs now can run command lines longer than than 4096 on our linux machines.
When determining if the help text and Queue extensions need to be rebuilt, use the .class files not the .java so that GATK oneoffs are picked up correctly.
Added the most basic of all example QScripts for debugging, Hello World.
Minor updates to copy/pasted LSF code to reduce ant javadoc warnings by a third.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4970 348d0f76-0448-11de-a6fe-93d51630548a
2011-01-10 21:07:29 +00:00
kshakir b34e2f733f Removed stochasticity from IndelRealigner by random sampling using and seed based on the read list.
Updated the Queue scatter/gather for read walkers to include -L unmapped on the last scatter job when intervals aren't specified, and to map it correctly when it is explicitly set.
Simplified the build.xml/ivy.xml to fix a bug reported with "ant clean dist test" where the scalac target wasn't found.
Now building all scala code at the same time, just like all java code is compiled at the same time.
Sped up the build for everyone by uncommenting a small bit of classes so that javac/scalac will not constantly launch trying to build .class files that will never compile.
Moved some source files to their expected location so that the .java/.scala -> .class is a one-to-one match, again keeping the compilers from wasting cycles.
Used <uptodate> and <touch> to skip extracting the help text and generating the GATK Queue extensions when the source files haven't been modified.
Fixed a couple errors when the <javadoc> task is run.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4963 348d0f76-0448-11de-a6fe-93d51630548a
2011-01-07 22:03:36 +00:00
kshakir 56433ebf6b Switched from LSF command line wrappers to JNA wrappers around the C API. Side effects:
- bsub command line is no longer fully printed out.
- extraBsubArgs hack is now a callback function updateJobRun.
Updated FullCallingPipelineTest to reflect latest changes to fullCallingPipeline.q.
Added a pipeline that tests the UGv2 runtimes at different bam counts and memory limits.
Updated VE packages that live in oneoffs to compile to oneoffs.
Added a hack to replace the deprecated symbol environ in Mac OS X 10.5+ which is needed by LSF7 on Mac.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4816 348d0f76-0448-11de-a6fe-93d51630548a
2010-12-10 04:36:06 +00:00
kshakir 787e5d85e9 Added the ability to test pipelines in dry or live mode via 'ant pipelinetest' and 'ant pipelinetest -Dpipeline.run=run'.
Added an initial test for genotyping chr20 on ten 1000G bams.
Since tribble needs logging support too, for now setting the logging level and appending the console logger to the root logger, not just to "org.broadinstitute.sting".
Updated IntervalUtilsUnitTest to output to a temp directory and not the SVN controlled testdata directory.
Added refseq tables and dbsnps to validation data in BaseTest.
Now waiting up to two minutes for gather parts to propagate over NFS before attempting to merge the files.
Setting scatter/gather directories relative to the -run directory instead of the current directory that queue is running.
Fixed a bug where escaping test expressions didn't handle delimiters at the beginning or end of the String.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4717 348d0f76-0448-11de-a6fe-93d51630548a
2010-11-22 22:59:42 +00:00
kshakir 79725f2d9c Excluding the QFunction log files from the set of files to delete on completion.
When a QGraph is empty displaying a warning instead of crashing with an JGraph internal assertion error.
Cleaned up code using the Log4J root logger and explicitly talking to a logger for Sting.
When integration tests are run detecting that the logger has already been setup so that messages aren't logged twice.
Updated from Ivy 2.2.0-rc1 to 2.2.0.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4707 348d0f76-0448-11de-a6fe-93d51630548a
2010-11-18 20:22:01 +00:00
hanna 9942f436b4 Support dist target for externally written walkers.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4701 348d0f76-0448-11de-a6fe-93d51630548a
2010-11-17 20:41:04 +00:00
hanna 24ec35deaf - Reintroduce test dependency so that the tests passing / failing is not
dependent on the contents of the integrationtest directory.  Will figure
  out how to better manage the integrationtest directory at some point in
  the future.
- Up the max heap size for tests.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4691 348d0f76-0448-11de-a6fe-93d51630548a
2010-11-16 19:55:20 +00:00
hanna 8ff4e4cb25 Cleanup testng listener configuration.
- Add StingTextReporter, which provides a text dump of the errors to the
  console.  Had to create our own reporter (inheriting from the standard
  TestNG TextReporter) to work around a configuration issue with the
  TextReporter.  In an ideal world, I'd report this on the TestNG mailing
  list and help them resolve the issue, but this solution is relatively
  robust at the moment and life is too short.
- Added back the failed test listener, which generates the testng-failed.xml
  file.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4686 348d0f76-0448-11de-a6fe-93d51630548a
2010-11-15 23:43:14 +00:00
kshakir a2c160da2d Explicitly set TestNG to run headless with no UI.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4662 348d0f76-0448-11de-a6fe-93d51630548a
2010-11-12 22:02:02 +00:00
kshakir 33b3f2adaa GSA-413 Explicitly set all compilers to use 512m memory maximum instead of 64m default maximum and running them out of the ant process.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4639 348d0f76-0448-11de-a6fe-93d51630548a
2010-11-09 19:32:19 +00:00
hanna 8ceb18eea9 Adding packaging system support for external directories.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4618 348d0f76-0448-11de-a6fe-93d51630548a
2010-11-02 20:13:13 +00:00
hanna 39c2247150 Capture failed test results in Bamboo as well as successful test results.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4615 348d0f76-0448-11de-a6fe-93d51630548a
2010-11-02 16:57:15 +00:00
hanna 14e992ab90 Test: trying to make Bamboo aggregate all test results instead of only
most recently run suite.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4614 348d0f76-0448-11de-a6fe-93d51630548a
2010-11-02 15:33:42 +00:00
hanna d496f2afde Switch from JUnitReportReporter (sic) to JUnitXMLReporter for output.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4612 348d0f76-0448-11de-a6fe-93d51630548a
2010-11-02 07:14:21 +00:00
hanna af2313de45 Another TestNG fix: we were generating JUnit html reports to the wrong
directory.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4611 348d0f76-0448-11de-a6fe-93d51630548a
2010-11-02 05:30:50 +00:00
hanna ebc01648af Update output directory for TestNG reports.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4610 348d0f76-0448-11de-a6fe-93d51630548a
2010-11-01 23:11:33 +00:00
hanna 861ee3e37a Changing testing framework from junit -> testng, for its enhanced configurability.
Initial test to see how Bamboo will respond.  More detailed email to follow.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4609 348d0f76-0448-11de-a6fe-93d51630548a
2010-11-01 21:31:44 +00:00
hanna 2f8057bf24 Cleanup for multithreading memory leak during integration tests...unregister MXBean at end
of traversal to avoid holding a reference to the microscheduler, which holds a reference to
the engine, which in turn holds a reference to the walker, which itself holds a reference to
all the data aggregated during the course of the traversal.


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4594 348d0f76-0448-11de-a6fe-93d51630548a
2010-10-28 18:37:42 +00:00
kshakir db47230dd9 Wrapping ScatterGatherableFunctions with a facade instead of using slower clone library. Will require keeping Clone's facade code in sync with CommandLineFunction but runs *much* faster.
Shell invoking scripts so that even really long shell scripts make it through LSF.
Using the truncated (up to 1000 characters) of the command line for the job name for use with bjobs.
Switched the default from re-running everything to re-running only files that need to be regenerated.  --skip_up_to_date replaced with --start_clean for those who want to regenerate everything.
Updated logging to let users know when the scatter gather generator is running, which still takes a while but is orders of magnatudes faster for large lists of functions.  (40s for a 100 function graph exploding to a 2500 function graph)


git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4448 348d0f76-0448-11de-a6fe-93d51630548a
2010-10-07 01:19:18 +00:00
kshakir bb44044ce0 Fixed re-builds of queue so that previously compiled classes are included. Fixes redundant case of "ant queue test" vs. "ant test".
Refactored temp directory utils.



git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4426 348d0f76-0448-11de-a6fe-93d51630548a
2010-10-04 21:12:07 +00:00
kshakir 4dfed62e7d Generating the Queue GATK extensions using java, then compiling all the Queue scala code at once to allow circular dependencies between existing and generated scala code.
Will see how this behaves for those using IntelliJ as generated source code will disappear during an ant clean.

git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@4425 348d0f76-0448-11de-a6fe-93d51630548a
2010-10-04 19:38:29 +00:00