gatk-3.8

Author	SHA1	Message	Date
chartl	ac9c335cd2	This is a python job dispatcher I've been using, which builds on Mark's FarmJob utility, and an example script of how I'm using it. Basically I wrote it to smartly break up analysis over an interval list, givin a maximum number of bases per job, a list of available queues, and a limit on each queue. It handles going over these limits in three ways: 1) [default]: Fail - No jobs are actually spawned 2) Space - User provides a string of the form A:B:C where A is the number of days to wait before scheduling jobs over the queue limits; B the number of hours, C the number of minutes. Exceeding the queue limits again will increment the space by another A:B:C 3) Stop-Resume - Spawns the maximum number of jobs, and writes a file describing the next job, and a hash code of the remaining jobs. The next time the script is run, it spawns the next set of jobs starting with what's written in the file. If the hash code (and thus the command string) changes between runs, the dispatcher fails-fast. The base job dispatcher class is also capable of dealing with dependencies if it is used correctly. git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3102 348d0f76-0448-11de-a6fe-93d51630548a	2010-03-31 19:53:13 +00:00

Author

SHA1

Message

Date

chartl

This is a python job dispatcher I've been using, which builds on Mark's FarmJob utility, and an example script of how I'm using it. Basically I wrote it to smartly break up analysis over an interval list, givin a maximum number of bases per job, a list of available queues, and a limit on each queue. It handles going over these limits in three ways:

1) [default]: Fail
  - No jobs are actually spawned

2) Space
  - User provides a string of the form A:B:C where A is the number of days to wait before scheduling jobs over
    the queue limits; B the number of hours, C the number of minutes. Exceeding the queue limits again will
    increment the space by another A:B:C

3) Stop-Resume
 - Spawns the maximum number of jobs, and writes a file describing the next job, and a hash code of the remaining jobs.
   The next time the script is run, it spawns the next set of jobs starting with what's written in the file. If the 
   hash code (and thus the command string) changes between runs, the dispatcher fails-fast.

The base job dispatcher class is also capable of dealing with dependencies if it is used correctly.



git-svn-id: file:///humgen/gsa-scr1/gsa-engineering/svn_contents/trunk@3102 348d0f76-0448-11de-a6fe-93d51630548a

2010-03-31 19:53:13 +00:00

1 Commits (91824f478e962ffe66ecea663aebfe325b966db2)