Packaging from a non-clean working directory can result in an incorrect
jar. Now that we have external collaborators packaging and distributing
the GATK, not enforcing the clean requirement has become far too dangerous.
At the same time, invoking "clean" automatically through a direct
dependency would also be dangerous -- instead, it's better to error out
if a packaging target is invoked from a non-clean working dir.
-- Closes GSA-494 / Add maximum runtime for integration tests, running them in timeout thread
-- Needed to debug locking issues
-- Needed to debug excessively long running integrationtests
-- Added build.xml maximum runtime for all testng tests of 10 hours. We will ultimately fail the build if it goes on for more than 10 hours
Added targets to build.xml to effectively 'mvn install' packaged GATK/Queue from ant.
TODO: Versions during 'mvn install' are hardcoded at 0.0.1 until a better versioning scheme that works with maven dependencies has been identified.
-- Renamed TraversalErrorManager to the more general MultiThreadedErrorTracker
-- ErrorTracker is now used throughout the NanoScheduler. In order to properly handle errors, the work previously done by main thread (submit jobs, block on reduce) is now handled in a separate thread. The main thread simply wakes up peroidically and checks whether the reduce result is available or if an error has occurred, and handles each appropriately.
-- EngineFeaturesIntegrationTest checks that -nt and -nct properly throw errors in Walkers
-- Added NanoSchedulerUnitTest for input errors
-- ThreadEfficiencyMonitoring is now disabled by default, and can be enabled with a GATK command line option. This is because the monitoring doesn't differentiate between threads that are supposed to do work, and those that are supposed to wait, and therefore gives misleading results.
-- Build.xml no longer copies the unittest results verbosely
The previous push fixed the external classpath issue but broke external
builds in a new way by changing the above from paths to properties. This
was a mistake, since external builds require absolute, not relative, paths
Thanks to akiezun for the bug report and patch
Use "path" instead of "pathconvert" to construct the external.gatk.classpath.
This allows the path to evolve as the build progresses, instead of being
fixed early on to a value that (in some cases) could be incorrect.
Cuts major corners for speed. Tests start in SECONDS instead of minutes.
SIGNIFICANT limitations (see below!)
Usage: ant fasttest -Dsingle=TestClass
The idea is that you do a regular "ant test -Dsingle=TestClass" (or "ant committests")
FIRST, then do "ant fasttest -Dsingle=TestClass" for all subsequent runs until
satisfied.
LIMITATIONS:
-REQUIRES that a full test build has already been done (using one of the
test targets like committests, or a manual "ant test.compile").
-Java only
-Single test class only
-No contracts
-Build jars in dist/ not updated, only classes in build/
-Version number output at runtime may be incorrect
-Classpaths to test the release jars were being constructed prematurely,
before all needed properties had been defined
-Added reportng as a TestNG dependency for testing purposes
* Targets to package and release lite/protected versions of the GATK/Queue
* Still TODO: -determine the actual directories where the protected releases should go
-update the Bamboo release plan
-fix a bug in the binary release test targets
-- Fully working version
-- Use -generateShadowBCF to write out foo.bcf as well as foo.vcf anywhere you use -o foo.vcf
-- Moved MedianUnitTest to its proper home in Utils
-- Added reportng to ivy and testng, so build/report/X/html/ is a nicely formatted output for Unit and Integration tests. From this website it's easy to see md5 diffs, etc. This is a vastly better way to manage unit and integration test output
-- Added VCFHeader() constructor that makes an empty header, and updated VariantRecalibrator to use it
-- Update build.xml to build vcf.jar with updated paths and bcf2 support.
* Added support for a protected directory whose contents are only made public in binary form
* Simplified and reorganized build.xml to improve readability and maintainability
* build.xml now autodetects most build properties:
-Includes private/protected if they exist
-No more STING_BUILD_TYPE or specialized targets for public-only, etc.
* Build targets have changed! There are now two main build options:
"ant" build everything (GATK and Queue)
"ant gatk" build just the GATK
It was too hard to build everything before -- now it is the default.
* To run tests with debugging, use -Dtest.debug=true -Dtest.debug.port=XXXX on the command line.
Much better than the old comment/uncomment method!
-- Other tribble contributors did major refactoring / simplification of tribble, which required some changes to GATK code
-- Integrationtests pass without modification, though some very old index files (callable loci beds) were apparently corrupt and no longer tolerated by the newer tribble codebase
-Running the GATK with the -et NO_ET or -et STDOUT options now
requires a key issued by us. Our reasons for doing this, and the
procedure for our users to request keys, are documented here:
http://www.broadinstitute.org/gsa/wiki/index.php/Phone_home
-A GATK user key is an email address plus a cryptographic signature
signed using our private key, all wrapped in a GZIP container.
User keys are validated using the public key we now distribute with
the GATK. Our private key is kept in a secure location.
-Keys are cryptographically secure in that valid keys definitely
came from us and keys cannot be fabricated, however keys are not
"copy-protected" in any way.
-Includes private, standalone utilities to create a new GATK user key
(GenerateGATKUserKey) and to create a new master public/private key
pair (GenerateKeyPair). Usage of these tools will be documented on
the internal wiki shortly.
-Comprehensive unit/integration tests, including tests to ensure the
continued integrity of the GATK master public/private key pair.
-Generation of new user keys and the new unit/integration tests both
require access to the GATK private key, which can only be read by
members of the group "gsagit".
"ant alltests" is now "ant committests"
"ant alltests.public" is now "ant committests.public"
"ant alltests.gatk.packagejar" is now "ant releasetests.gatk.packagejar"
"ant alltests.queue.packagejar" is now "ant releasetests.queue.packagejar"
This is going into both Stable + Unstable so that all Bamboo
plans can be properly updated at the same time.
Changes to the build script that allow the external directory to have tests.
This means groups like CGA don't have to reinvent the wheel on testing, and
can instead use the GATKs unit and integration tests.
Signed-off-by: David Roazen <droazen@broadinstitute.org>
QScript accessor to QSettings to specify a default runName and other default function settings.
Because log files are no longer pseudo-random their presense can be used to tell if a job without other file outputs is "done". For now still using the log's .done file in addition to original outputs.
Gathered log files concatenate all log files together into the stdout.
InProcessFunctions now have PrintStreams for stdout and stderr.
Updated ivy to use commons-io 2.1 for copying logs to the stdout PrintStream. Removed snakeyaml.
During graph tracking of outputs the Index files, and now BAM MD5s, are tracked with the gathering of the original file.
In Queue generated wrappers for the GATK the Index and MD5s used for tracking are switched to private scope.
Added more detailed output when running with -l DEBUG.
Simplified graphviz visualization for additional debugging.
Switched usage of the scala class 'List' to the trait 'Seq' (think java.util.ArrayList vs. using the interface java.util.List)
Minor cleanup to build including sending ant gsalib to R's default libloc.
Previously, the initial release of a new GATK version had a version
number with only one part (eg., "1.4"). This could potentially mislead
people into thinking it's the most recent revision of a release, instead
of the least recent.
Now, initial releases will have full, three-part version numbers
(eg., "1.4-0-g472fc94") like everything else.
Moved gsalib and queueJobReport.R to embeddable namespaced locations.
Updated packager dependencies/dir to add an @includes which filters the embedded fileset.
RScriptExecutor can now JIT compiles the gsalib.
RScriptExecutor uses ProcessController and sends the Rscript output to java's stdout when run under -l DEBUG.
Refactored ProcessController and IOUtils from Queue to Sting Utils.
Added more unit tests to ProcessController along with a utility class to hard stop OutputStreams at a specified byte count.
Replaced uses of some IOUtils with Apache Commons IO.
ShellJobRunner refactored to use direct ProcessController and now kills jobs on shutdown.
Better QGraph responsiveness on shutdown by using Object.wait() instead of Thread.sleep().
The test was successful -- the packaging error was detected and
prevented from propagating out into the binary/source releases.
This reverts commit 8f2a1462d94ce50fb7c1b1d0c40142b109e9c38e.
This is to test the ability of the new release system
to detect the kinds of packaging errors we couldn't
detect before (in this case, missing codecs). If all goes
as expected, "GSA-Stable: Release" will fail, preventing
the binary release and github from getting updated, while
"GSA-Stable: All Tests" and all of the other old test plans
will pass.
Will revert this in a bit -- if the system works as it should,
our users will never see it until after it's been reverted.
-Added targets to run unit and integration tests on the fully-packaged GATK jar,
and pipeline tests on the fully-packaged Queue jar. Once enabled in Bamboo,
these will provide greatly-enhanced protection against breakage in the binary
release.
-Unconditionally include all of the subset of org.broadinstitute.sting
included in the intermediate jars GenomeAnalysisTK.jar, StingUtils.jar,
etc. in the final, fully-packaged jar. This:
* is necessary to get tests to run on the fully-packaged jar
* decreases the chances of a class that is a runtime-only
dependency getting left out of the binary release
* only slightly increases the size of the binary release
(before: 9352465 bytes, after: 10985482 bytes)
David Roazen
Mark DePristo
kshakir
Guillermo del Angel
Aaron McKenna
Matt Hanna