fab6f57e971. Enabled FTZ in LoadTimeInitializer.cc 2. Added Sandbox.java for testing 3. Moved compute to utils.cc (inside library) 4. Added flag for disabling FTZ in Makefile
Karthik Gururaj
2014-02-06 11:01:33 -0800
78642944c01. Moved break statement in utils.cc to correct position 2. Tested sandbox with regions 3. Lots of profiling code from previous commit exists
Karthik Gururaj
2014-02-06 09:32:56 -0800
b21c35482ePackages link private/testdata, so that mvn test -Dsting.serialunittests.skipped=false works.
Khalid Shakir
2014-02-06 14:31:30 +0800
3848159086Added a set of serial tests to gatk/queue packages, which runs all tests under their package in one TestNG execution. New properties to disable regenerating example resources artifact when each parallel test runs under packagetest. Moved collection of packagetest parameters from shell scripts into maven profiles. Fixed necessity of test-utils jar by removing incorrect dependenciesToScan element during packagetests. When building picard libraries, run clean first. Fixed tools jar dependency in picard pom. Integration tests properly use the ant-bridge.sh test.debug.port variable, like unit tests.
Khalid Shakir
2014-02-06 01:17:26 +0800
98ffcf6833Get gVCF to work without --dontTrimActiveRegions
Valentin Ruano-Rubio
2014-02-03 18:26:22 -0500
acda6ca27b1. Whew, finally debugged the source of performance issues with PairHMM JNI. See copied text from email below. 2. This commit contains all the code used in profiling, detecting FP exceptions, dumping intermediate results. All flagged off using ifdefs, but it's there. --------------Text from email As we discussed before, it's the denormal numbers that are causing the slowdown - the core executes some microcode uops (called FP assists) when denormal numbers are detected for FP operations (even un-vectorized code). The C++ compiler by default enables flush to zero (FTZ) - when set, the hardware simply converts denormal numbers to 0. The Java binary (executable provided by Oracle, not the native library) seems to be compiled without FTZ (sensible choice, they want to be conservative). Hence, the JNI invocation sees a large slowdown. Disabling FTZ in C++ slows down the C++ sandbox performance to the JNI version (fortunately, the reverse also holds :)). Not sure how to show the overhead for these FP assists easily - measured a couple of counters. FP_ASSISTS:ANY - shows number of uops executed as part of the FP assists. When FTZ is enabled, this is 0 (both C++ and JNI), when FTZ is disabled this value is around 203540557 (both C++ and JNI) IDQ:MS_UOPS_CYCLES - shows the number of cycles the decoder was issuing uops when the microcode sequencing engine was busy. When FTZ is enabled, this is around 1.77M cycles (both C++ and JNI), when FTZ is disabled this value is around 4.31B cycles (both C++ and JNI). This number is still small with respect to total cycles (~40B), but it only reflects the cycles in the decode stage. The total overhead of the microcode assist ops could be larger. As suggested by Mustafa, I compared intermediate values (matrices M,X,Y) and final output of compute_full_prob. The values produced by C++ and Java are identical to the last bit (as long as both use FTZ or no-FTZ). Comparing the outputs of compute_full_prob for the cases no-FTZ and FTZ, there are differences for very small values (denormal numbers). Examples: Diff values 1.952970E-33 1.952967E-33 Diff values 1.135071E-32 1.135070E-32 Diff values 1.135071E-32 1.135070E-32 Diff values 1.135071E-32 1.135070E-32 For this test case (low coverage NA12878), all these values would be recomputed using the double precision version. Enabling FTZ should be fine. -------------------End text from email
Karthik Gururaj
2014-02-05 17:09:57 -0800
6a7a197362Merge pull request #486 from broadinstitute/rp_fix_missing_annotations_CombineReferenceCalculationVariants
Ryan Poplin
2014-02-05 14:22:59 -0500
693bfac341Bug fix for missing annotations in CombineReferenceCalculationVariants. They were being dropped in the handoff between engines in a couple of places.
Ryan Poplin
2014-02-05 10:43:05 -0500
8aa8acf81dMerge pull request #485 from broadinstitute/eb_more_combine_rc_variants_iterations
Eric Banks
2014-02-05 11:32:30 -0500
740b33acbbWe were never validating the sequence dictionary of tabix indexed VCFs for some reason. Fixed.
Eric Banks
2014-02-04 01:21:17 -0500
9cac24d1e6Moving logging status of VCF indexing to DEBUG instead of INFO, otherwise it's painful when reading in lots of files
Eric Banks
2014-02-04 00:24:44 -0500
91bdf069d3Some updates to CRCV.
Eric Banks
2014-01-13 22:13:53 -0500
22bcd10372Merge pull request #484 from broadinstitute/jt_select_variants_nt_maven
droazen
2014-02-05 08:15:02 -0500
7923e786e9Rev Picard (public) to 1.107.1676 - Rename snappy to snappy-java - Add maven-metadata-local.xml to .gitignore
Joel Thibault
2014-02-04 12:19:55 -0500
0025fe190dExclude sam's older TestNG
Joel Thibault
2014-02-04 13:27:19 -0500
9eaee8c73cIntegration test for the -nt race condition corrupting AD and PL fields
Joel Thibault
2014-02-04 11:45:18 -0500
1de7a27471Disable an additional test that is runtime dependent on one of the temporarily-disabled tests
David Roazen
2014-02-04 16:07:58 -0500
76086f30b7Temporarily disable tests that started failing post-maven
David Roazen
2014-02-04 15:31:24 -0500
3b2f07990dRe-break the MWUnitTest for Joel to debug
David Roazen
2014-02-04 15:19:09 -0500
c9032f0b5cFix failing unit tests
David Roazen
2014-02-04 02:55:54 -0500
4eaa724be6Merge pull request #483 from broadinstitute/ks_new_maven_build_system
droazen
2014-02-03 10:53:18 -0800
60567c8d7eMinor ant-bridge.sh changes
David Roazen
2014-02-03 12:54:20 -0500
a4289711e2Distinct failsafe summary reports, just like invoker report directories.
Khalid Shakir
2014-02-01 19:58:49 +0800
5ab0b117d2Re-split the invoked integration tests and verify into separate phases.
Khalid Shakir
2014-02-01 18:34:40 +0800
b10b42c10aAdded a "dry" argument to ant-bridge.sh, that just prints the command that would run.
Khalid Shakir
2014-02-01 11:27:25 +0800
857e6e0d6fBumped version to 2.8-SNAPSHOT, using new update_pom_versions.sh script.
Khalid Shakir
2014-02-01 06:49:25 +0800
20b471ef7bant-bridge dist -> verify, test.compile -> test-compile Added a utility script for running a single test, usually in parallel.
Khalid Shakir
2014-02-01 05:02:08 +0800
9ca3004fc3Setting the test-utils' type to test-jar, such that the multi-module build uses testClasses instead of classes as a directory dependency.
Khalid Shakir
2014-02-01 04:02:46 +0800
f968b8a58bCrash when integration tests fail by running install and verify at the same time, instead of trying to do them separately.
Khalid Shakir
2014-02-01 02:25:07 +0800
de13f41fc3One step closer to a proper test-utils artifact. Using the maven-jar-plugin to create a test classifer, excluding actual tests, until we can properly separate the classes into separate artifacts/modules.
Khalid Shakir
2014-01-31 20:25:58 +0800
25aee7164eFixed missing "mvn" command execution in ant-bridge. Added pom.xml workarounds for duplicate classpath error, due to gatk-framework dependency containing required BaseTest, and jarred *UnitTest/*IntegrationTest classes that also exist as files under target/test-classes.
Khalid Shakir
2014-01-30 17:47:54 +0800
caa76cdac4Added maven pom.xmls for various artifacts.
Khalid Shakir
2014-01-29 17:00:06 -0500
d1a689af33Added new utility files used by maven build, including the ant-bridge script.
Khalid Shakir
2014-01-29 14:54:24 -0500
88150e0166Switched commited dependency repository from ivy to maven.
Khalid Shakir
2014-01-29 14:18:02 -0500
1e25a758f5Moved files to maven directories. Here are the git moved directories in case other files need to be moved during a merge: git-mv private/java/src/ private/gatk-private/src/main/java/ git-mv private/R/scripts/ private/gatk-private/src/main/resources/ git-mv private/java/test/ private/gatk-private/src/test/java/ git-mv private/testdata/ private/gatk-private/src/test/resources/ git-mv private/scala/qscript/ private/queue-private/src/main/qscripts/ git-mv private/scala/src/ private/queue-private/src/main/scala/ git-mv protected/java/src/ protected/gatk-protected/src/main/java/ git-mv protected/java/test/ protected/gatk-protected/src/test/java/ git-mv public/java/src/ public/gatk-framework/src/main/java/ git-mv public/java/test/ public/gatk-framework/src/test/java/ git-mv public/testdata/ public/gatk-framework/src/test/resources/ git-mv public/scala/qscript/ public/queue-framework/src/main/qscripts/ git-mv public/scala/src/ public/queue-framework/src/main/scala/ git-mv public/scala/test/ public/queue-framework/src/test/scala/
Khalid Shakir
2014-01-29 13:36:16 -0500
faaef236eaMoved gsalib, R and other resources, Queue GATK extensions generator, Queue version java files.
Khalid Shakir
2014-01-29 13:25:35 -0500
0c63d6264f1. Added synchronization block around loadLibrary in VectorLoglessPairHMM 2. Edited Makefile to use static libraries where possible
Karthik Gururaj
2014-01-27 15:34:58 -0800
018e9e2c5f1. Cleaned up code 2. Split into DebugJNILoglessPairHMM and VectorLoglessPairHMM with base class JNILoglessPairHMM. DebugJNILoglessPairHMM can, in principle, invoke any other child class of JNILoglessPairHMM. 3. Added more profiling code for Java parts of LoglessPairHMM
Karthik Gururaj
2014-01-26 19:18:12 -0800
9e7bf75e89Fix for the PairHMM transition probability miscalculation.
Valentin Ruano-Rubio
2014-01-24 13:27:51 -0500
e7598dde8bClean up
mghodrat
2014-01-26 11:36:06 -0800
81bdfbd00dTemporary commit before moving to new native library
Karthik Gururaj
2014-01-24 16:29:35 -0800
f7fa79e561Merge branch 'intel_pairhmm' of /home/karthikg/broad/gsa-unstable into intel_pairhmm Committing into central_repo
Intel Repocontact
2014-01-22 23:08:32 -0800
936e9e175e1. Converted q,i,d,c in C++ from int* to char* 2. Use clock_gettime to measure performance 3. Disabled OpenMP 4. Moved LoadTimeInitializer to different file
Karthik Gururaj
2014-01-22 22:57:32 -0800
733a84e4f9Added support to transfer haplotypes once per region to the JNI Re-use transferred haplotypes (stored in GlobalRef) across calls to computeLikelihoods
Karthik Gururaj
2014-01-22 10:52:41 -0800
868a8394f7Deleted libJNILoglessPairHMM.so from git tracking
Karthik Gururaj
2014-01-21 15:01:09 -0800
217f6948f1Merge branch 'master' of /home/mozdal/git/hmm into intel_pairhmm
Karthik Gururaj
2014-01-21 12:43:16 -0800
1b1c0c8e76Split the inner loop to avoid the overhead incurred when -fPIC flag is enabled.
mozdal
2014-01-21 11:47:30 -0800
88c08e78e71. Inserted #define in sandbox pairhmm-template-main.cc 2. Wrapped _mm_empty() with ifdef SIMD_TYPE_SSE 3. OpenMP disabled 4. Added code for initializing PairHMM's data inside initializePairHMM - not used yet
Karthik Gururaj
2014-01-21 09:57:14 -0800
0170d4f3d5Got rid of the MMX instructions in the SSE version of the code. Handling the mask operations in a class, which is defined for each version of SSE and AVX implementations separately.
mozdal
2014-01-21 09:30:15 -0800
bdd06ebfc2Merge pull request #478 from broadinstitute/eb_generalize_hc_values_as_args
Ryan Poplin
2014-01-21 09:01:54 -0800
8812278c2cMerge pull request #479 from broadinstitute/eb_move_test_up_one_level
Eric Banks
2014-01-21 06:45:55 -0800
7180c392af1. Integrated Mohammad's SSE4.2 code, Mustafa's bug fix and code to fix the SSE compilation warning. 2. Added code to dynamically select between AVX, SSE4.2 and normal C++ (in that order) 3. Created multiple files to compile with different compilation flags: avx_function_prototypes.cc is compiled with -xAVX while sse_function_instantiations.cc is compiled with -xSSE4.2 flag. 4. Added jniClose() and support in Java (HaplotypeCaller, PairHMMLikelihoodCalculationEngine) to call this function at the end of the program. 5. Removed debug code, kept assertions and profiling in C++ 6. Disabled OpenMP for now.
Karthik Gururaj
2014-01-20 08:03:42 -0800
9e858270d7Moving this test up one level to where it actually belongs.
Eric Banks
2014-01-19 02:33:11 -0500
64d5bf650ePulled out some hard-coded values from the read-threading and isActive code of the HC, and made them into a single argument.
Eric Banks
2013-12-11 10:20:07 -0500
25aecb96e0Added support for dynamic selection between AVX and un-vectorized C++, still to include SSE code from Mohammad. Debug flags turned on in this commit.
Karthik Gururaj
2014-01-18 11:07:23 -0800
abd4f552baMerge pull request #476 from broadinstitute/yf_logging_all_input_SAMFiles
Eric Banks
2014-01-17 08:54:33 -0800
c79e8ca53eAdded an info log containing the SAM/BAM files that were eventually found from the commandline (useful for when there are files hiding inside bam.lists which may or may not have been constructed correctly...) Added a @hidden option controling the appearance of the full BamList in the log
Yossi Farjoun
2014-01-15 11:02:04 -0500
d53e2fbe66Uncommenting download option in build.xml
Intel Repocontact
2014-01-16 21:55:04 -0800
f1c772ceeaSame log message as before - forgot -a option 1. Moved computeLikelihoods from PairHMM to native implementation 2. Disabled debug - debug code still left (hopefully, not part of bytecode) 3. Added directory PairHMM_JNI in the root which holds the C++ library that contains the PairHMM AVX implementation. See PairHMM_JNI/JNI_README first
Karthik Gururaj
2014-01-16 21:40:04 -0800
d7ba1f1c281. Moved computeLikelihoods from PairHMM to native implementation 2. Disabled debug - debug code still left (hopefully, not part of bytecode) 3. Added directory PairHMM_JNI in the root which holds the C++ library that contains the PairHMM AVX implementation. See PairHMM_JNI/JNI_README first
Karthik Gururaj
2014-01-16 21:36:15 -0800
b57de8eec1Merge branch 'master' of /home/karthikg/broad/archive/hmm_intra into intel_pairhmm
Karthik Gururaj
2014-01-16 20:29:51 -0800
90938b8610Minor typo in comments fixed
Karthik Gururaj
2014-01-16 19:58:04 -0800
e90405cd1f1. Nested loops over reads and haplotypes moved to C++ through JNI 2. OpenMP support added 3. Using direct access to Java primitive arrays 4. Debug messages disabled
Karthik Gururaj
2014-01-16 19:53:50 -0800
3b6b7626aaMerge pull request #472 from broadinstitute/eb_extend_private_simulate_reads_tool
Eric Banks
2014-01-15 17:51:07 -0800
e8a50227771. Added support for JNI integration for LoglessCaching PairHMM AVX implementation. 2. Contains lots of debug code 3. Only invokes JNI for subComputeReadLikelihoodGivenHaplotypeLog10
Karthik Gururaj
2014-01-15 11:07:09 -0800
de56134579Fixed up and refactored what seems to be a useful private tool to create simulated reads around a VCF.
Eric Banks
2014-01-09 12:29:54 -0500
8240ea826eChanges: 1. Added TRISTATE_CORRECTION in pairhmm-template-kernel.cc (function stripINITIALIZATION) 2. Added VEC_DIV macros to define-double.h and define-float.h 3. Edited initializeVectors to match Java C++ original: *(ptr_p_MY+r-1) = (r == ROWS - 1) ? ctx._(1.0) : ctx.ph2pr[_d]; *(ptr_p_YY+r-1) = (r == ROWS - 1) ? ctx._(1.0) : ctx.ph2pr[_c]; Modified: *(ptr_p_MY+r-1) = ctx.ph2pr[_d]; *(ptr_p_YY+r-1) = ctx.ph2pr[_c];
Karthik Gururaj
2014-01-15 10:48:58 -0800
e2c2aa7b05Merge pull request #475 from broadinstitute/eb_fix_null_alleles_bug_PT63551060
Eric Banks
2014-01-15 08:05:21 -0800
9f1ab0087aAdded in a check for what would be an empty allele after trimming.
Eric Banks
2014-01-14 12:56:54 -0500
5fab96b7eeFirst import of AVX-JNI to git
Karthik Gururaj
2014-01-14 17:26:55 -0800
201ad398acMerge pull request #473 from broadinstitute/eb_fix_qd_indel_normalization
Ryan Poplin
2014-01-14 08:56:19 -0800
e4fdc5ac44Merge pull request #474 from broadinstitute/eb_fix_haplotype_resolver_PT63333488
Eric Banks
2014-01-14 07:36:53 -0800
f67c33919bMerge pull request #468 from broadinstitute/gg_fixSAMPileup
Geraldine Van der Auwera
2014-01-14 06:30:04 -0800
edf5880022Updated SAMPileup codec and pileup-related docs
Geraldine Van der Auwera
2014-01-10 18:21:56 -0500
16ecc53749Merge pull request #469 from broadinstitute/gg_gatkdoc_fixes
Eric Banks
2014-01-14 05:56:07 -0800
fd511d12a2Fixing the Haplotype Resolver so that it doesn't complain about missing header lines.
Eric Banks
2014-01-13 22:47:43 -0500
347fab4717Merge pull request #471 from broadinstitute/eb_output_log_info_for_tim
droazen
2014-01-13 17:48:40 -0800
bdb3954eb3removed maxRuntime minValue
Geraldine Van der Auwera
2014-01-13 20:45:43 -0500
8fcad6680bAssorted fixes and improvements to gatkdocs
Geraldine Van der Auwera
2013-12-10 13:19:53 -0500
c7e08965d0The QD normalization for indels was busted and is now fixed.
Eric Banks
2014-01-10 01:36:39 -0500
851ec67bdcAdding more meta information about the user to the GATK logging output, per Tim F's request.
Eric Banks
2014-01-13 14:36:02 -0500
7cd304fb41Merge pull request #470 from broadinstitute/mf_new_RBP
droazen
2014-01-13 08:46:27 -0800
Karthik Gururaj
Khalid Shakir
Valentin Ruano Rubio
Ryan Poplin
Eric Banks
droazen
Joel Thibault
mghodrat
mozdal
Yossi Farjoun
Geraldine Van der Auwera