depristo
|
b5d1aab8dc
|
Scripts to create the GATK IAM user and give him/her rights to PutObject (and only PutObject) into the S3 storage instance. Updated the GATKRunReport to now upload using the GATK user, not mark@depristo.com. Running with -et AWS_S3 sends run reports up to the Amazon S3 cloud now. Going to request a few external users try this option so we can see it running at scale. I'm sure S3 can handle a few hundred thousand 1Kb uploads per days, though
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5132 348d0f76-0448-11de-a6fe-93d51630548a
|
2011-01-31 03:48:33 +00:00 |
|
depristo
|
f29bb0639b
|
Documentation and cleanup of the distributed GATK implementation. Detailed documentation -- given that Matt will be extending the system in the near future -- about how the locking and processing trackers work. Added error trapping to note that distributed, shared-memory parallelism isn't yet implemented, instead of just not working silently. General utility function for the analysis of distributedGATK operation in the analysis directory
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5106 348d0f76-0448-11de-a6fe-93d51630548a
|
2011-01-28 03:40:09 +00:00 |
|
depristo
|
be697d96f9
|
An apparently robust implementation of the file locking for distributed computation, using Lucene's file creation locking approach. It is worth trying out for those with large-scale, high-cost data sets. Details and discussion at group meeting on Wednesday. Some cleanup still needed.
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5079 348d0f76-0448-11de-a6fe-93d51630548a
|
2011-01-26 13:45:40 +00:00 |
|
depristo
|
c50f39a147
|
V3 of the distributed GATK. High-efficiency implementation. Support for status tracking for debugging and display. Still not safe for production use due to NFS filelock problem. V4 will use alternative file locking mechanism
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5063 348d0f76-0448-11de-a6fe-93d51630548a
|
2011-01-24 16:45:07 +00:00 |
|
depristo
|
ff61aeb762
|
continuing to push to get right answers for long-running jobs
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5054 348d0f76-0448-11de-a6fe-93d51630548a
|
2011-01-22 15:02:02 +00:00 |
|
depristo
|
8ece2b9230
|
Distributed GATK analysis scripts
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5049 348d0f76-0448-11de-a6fe-93d51630548a
|
2011-01-21 22:09:07 +00:00 |
|
depristo
|
94b64ec54a
|
Moving scala script into analysis directory
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5047 348d0f76-0448-11de-a6fe-93d51630548a
|
2011-01-21 18:42:18 +00:00 |
|
depristo
|
63e8103c4e
|
A new top-level directory to hold analysis scripts associated with specific analyses
git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@5046 348d0f76-0448-11de-a6fe-93d51630548a
|
2011-01-21 18:40:02 +00:00 |