-Running the GATK with the -et NO_ET or -et STDOUT options now
requires a key issued by us. Our reasons for doing this, and the
procedure for our users to request keys, are documented here:
http://www.broadinstitute.org/gsa/wiki/index.php/Phone_home
-A GATK user key is an email address plus a cryptographic signature
signed using our private key, all wrapped in a GZIP container.
User keys are validated using the public key we now distribute with
the GATK. Our private key is kept in a secure location.
-Keys are cryptographically secure in that valid keys definitely
came from us and keys cannot be fabricated, however keys are not
"copy-protected" in any way.
-Includes private, standalone utilities to create a new GATK user key
(GenerateGATKUserKey) and to create a new master public/private key
pair (GenerateKeyPair). Usage of these tools will be documented on
the internal wiki shortly.
-Comprehensive unit/integration tests, including tests to ensure the
continued integrity of the GATK master public/private key pair.
-Generation of new user keys and the new unit/integration tests both
require access to the GATK private key, which can only be read by
members of the group "gsagit".
"ant alltests" is now "ant committests"
"ant alltests.public" is now "ant committests.public"
"ant alltests.gatk.packagejar" is now "ant releasetests.gatk.packagejar"
"ant alltests.queue.packagejar" is now "ant releasetests.queue.packagejar"
This is going into both Stable + Unstable so that all Bamboo
plans can be properly updated at the same time.
Changes to the build script that allow the external directory to have tests.
This means groups like CGA don't have to reinvent the wheel on testing, and
can instead use the GATKs unit and integration tests.
Signed-off-by: David Roazen <droazen@broadinstitute.org>
QScript accessor to QSettings to specify a default runName and other default function settings.
Because log files are no longer pseudo-random their presense can be used to tell if a job without other file outputs is "done". For now still using the log's .done file in addition to original outputs.
Gathered log files concatenate all log files together into the stdout.
InProcessFunctions now have PrintStreams for stdout and stderr.
Updated ivy to use commons-io 2.1 for copying logs to the stdout PrintStream. Removed snakeyaml.
During graph tracking of outputs the Index files, and now BAM MD5s, are tracked with the gathering of the original file.
In Queue generated wrappers for the GATK the Index and MD5s used for tracking are switched to private scope.
Added more detailed output when running with -l DEBUG.
Simplified graphviz visualization for additional debugging.
Switched usage of the scala class 'List' to the trait 'Seq' (think java.util.ArrayList vs. using the interface java.util.List)
Minor cleanup to build including sending ant gsalib to R's default libloc.
Previously, the initial release of a new GATK version had a version
number with only one part (eg., "1.4"). This could potentially mislead
people into thinking it's the most recent revision of a release, instead
of the least recent.
Now, initial releases will have full, three-part version numbers
(eg., "1.4-0-g472fc94") like everything else.
Moved gsalib and queueJobReport.R to embeddable namespaced locations.
Updated packager dependencies/dir to add an @includes which filters the embedded fileset.
RScriptExecutor can now JIT compiles the gsalib.
RScriptExecutor uses ProcessController and sends the Rscript output to java's stdout when run under -l DEBUG.
Refactored ProcessController and IOUtils from Queue to Sting Utils.
Added more unit tests to ProcessController along with a utility class to hard stop OutputStreams at a specified byte count.
Replaced uses of some IOUtils with Apache Commons IO.
ShellJobRunner refactored to use direct ProcessController and now kills jobs on shutdown.
Better QGraph responsiveness on shutdown by using Object.wait() instead of Thread.sleep().
The test was successful -- the packaging error was detected and
prevented from propagating out into the binary/source releases.
This reverts commit 8f2a1462d94ce50fb7c1b1d0c40142b109e9c38e.
This is to test the ability of the new release system
to detect the kinds of packaging errors we couldn't
detect before (in this case, missing codecs). If all goes
as expected, "GSA-Stable: Release" will fail, preventing
the binary release and github from getting updated, while
"GSA-Stable: All Tests" and all of the other old test plans
will pass.
Will revert this in a bit -- if the system works as it should,
our users will never see it until after it's been reverted.
-Added targets to run unit and integration tests on the fully-packaged GATK jar,
and pipeline tests on the fully-packaged Queue jar. Once enabled in Bamboo,
these will provide greatly-enhanced protection against breakage in the binary
release.
-Unconditionally include all of the subset of org.broadinstitute.sting
included in the intermediate jars GenomeAnalysisTK.jar, StingUtils.jar,
etc. in the final, fully-packaged jar. This:
* is necessary to get tests to run on the fully-packaged jar
* decreases the chances of a class that is a runtime-only
dependency getting left out of the binary release
* only slightly increases the size of the binary release
(before: 9352465 bytes, after: 10985482 bytes)
classloading of bcel*.jar/ant-apache-bcel*.jar. Switching instead to manually
specifying a minimal set of packages/classes to include in the vcf.jar via
build.xml, and adding a unit test which creates a limited classloader
only aware of vcf.jar and tribble.jar and tries to use it to load the core
classes in the vcf jar.
Hopefully third time's the charm.
-- the underlying data structure is still present, but until I decide what to do for the extensible system I've completely disabled the subsystem
-- Added code to merge Samples, so that a mostly full record can be merged with a consistent empty record. If the two records are inconsistent, an error is thrown
-- addSample() in Sample.class now invokes mergeSample() when appropriate
-- Validation types are now only STRICT or SILENT
-- Validation code implemented in SampleDBBuilder
-- Extensive unit tests for SampleDBBuilder
To run the same set of tests as the bamboo "All Tests" plan (unit tests,
integration tests, and pipeline tests):
ant alltests
To do the same as above on only the public portion of the codebase:
ant alltests.public
Public-only tests will allow us to check for runtime public -> private
dependencies when bamboo updates the github repository (currently, we
only check for *compile-time* public -> private dependencies).
To compile/run only public tests, append ".public" to the name of an existing
test target:
ant test.public
ant integrationtest.public
ant performancetest.public
ant pipelinetest.public
ant pipelinetestrun.public
To include documentation for hidden features in the generated GATKDocs,
run with -Dgatkdocs.include.hidden=true
I will enable this flag when bamboo generates GATKDocs for unstable.
Contracts remain disabled for non-test build targets. To enable for
non-test targets, run with -Duse.contracts=true. To disable for test
targets, run with -Duse.contracts=false.
Usable targets are now:
ant javadoc (public-only)
ant javadoc.private (public + private)
ant scaladoc (public-only)
ant scaladoc.private (public + private)
As documented in the comments, you need to set the ANT_OPTS environment
variable to -Xmx1G before using the scaladoc targets.
Will modify bamboo to auto-generate these and post them to the web after
successful builds.
Required a significant refactoring of the GATKDoclet, which now has a unified place where the ClassDoc, class, annotation, and handler are all stored together.
Index expanded to use summary() annotation field
UserExceptions, ReadFilters, GATK engine all use the system to generate docs
Doclet expanded to handle lots of new cases
Now uses GATKDoc class to organize documentation for arguments.
Arguments now listed by feature (required, optional, hidden, etc) and link to detailed information about the argument in the html
Lots of code moving between Class and ClassDoc objects. Should be refactored into a single static utility class.
This is very useful and easy. After running any test (for example ant integrationtest) and seeing failures, you can rerun ONLY THE TESTS THAT FAILED by using one of the following commands:
ant failed-integration
ant failed-pipeline
ant failed-test
ant failed-performance
obviously matching whatever tests you were running and got failures on.
This should run only the failed tests, and you can keep using this command until you have fixed everything.
(Thanks to David Roazen for major help with ANT)
The NullPointerException we were seeing when trying to run with contracts enabled was being caused
by an outdated version of the asm library.
To run tests without contracts and disable their compilation, pass in "-Duse.contracts=false" to ant.
Also did some minor unrelated cleanup in build.xml
describe" as an absolute version number (if the repository has at least one
tag), using the raw SHA-1 hash value as a fallback version number in the case
where there are no tags.
Temporarily disabled contracts in integrationtests until we can find the cause
of the new error that's cropping up for Ryan.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5967 348d0f76-0448-11de-a6fe-93d51630548a
Current tribble-129M.jar in SVN does not work with current version of GATK code.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5897 348d0f76-0448-11de-a6fe-93d51630548a
only download that test configuration when running unit/integration tests.
This means that the build will (hopefully) never break because it can't
fetch a file that isn't required for the GATK to run.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5775 348d0f76-0448-11de-a6fe-93d51630548a
Also added the old model of indel calling to the FCP.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5749 348d0f76-0448-11de-a6fe-93d51630548a
Using the VCFWriterATD isCompressed to check if the VCF index will be auto generated.
Tracking BAM and Tribble indexes as @Inputs and @Outputs in generated QFunctions.
Updates to the BamGatherFunction to disable the index during merge when disable_bam_indexing = true.
Made a shortcut for live-running pipelinetest, pipelinetestrun.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5606 348d0f76-0448-11de-a6fe-93d51630548a
extracted from source files that have been modified since the help resource
file was last generated. This significantly speeds up builds where only a few
source files have been modified, at the expense of making clean builds take
slightly longer. Here's some performance data gathered by testing the old and
new versions of extracthelp in isolation and averaging across 10 runs:
old extracthelp, 1 modified source file: 20.1 seconds
new extracthelp, 1 modified source file: 7.2 seconds <-- woohoo! :)
old extracthelp, clean build: 17.8 seconds
new extracthelp, clean build: 20.5 seconds
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5590 348d0f76-0448-11de-a6fe-93d51630548a
Mark DePristo
Guillermo del Angel
David Roazen
Aaron McKenna
Khalid Shakir
Matt Hanna
Kiran V Garimella
Mauricio Carneiro
ebanks
depristo
hanna
kshakir
droazen