The next step is to understand basic dependencies
The method used below is not the best method for dependencies but is a short introduction to the theory behind dependencies
Feel free to download an run the below script as it setup a job that will :
- Create a Parent "Sleep job" with a range of 60
- Create a Blocked Child "Sleep job" that waits for the Parent to complete before starting
In this example, we use the job's dependency field to set up a job-based dependency. Job based dependencies are those that will wait for the entire dependent job to finish before the current job starts. There are also subjob (instance) and agenda (frame) based dependencies.
Below is the code with commented explanations of its contents contents
Code Block | ||||
---|---|---|---|---|
| ||||
#!/usr/bin/python #""" Below areLicensed requiredMaterials imports- forProperty theof script to run import os, sys # The below few lines of code are to determine the OS of the machine that your running # this script from and then define the location of the Qube! API if 'QBDIR' in os.environ: sys.path.append('%s/api/python' % os.environ['QBDIR']); elif os.uname()[0] == 'Darwin': sys.path.append('/Applications/pfx/qube/api/python'); elif os.uname()[0] == 'Linux': sys.path.append('/usr/local/pfx/qube/api/python'); else: sys.path.append('c:/program files/pfx/qube/api/python'); # The below line of code is to import the above defined Qube! API import qb # Below is the main function to run in this script def main(): # ----------------Start creation of Parent Job---------------------------------------- # Below creates an empty dictionary to be filled by the following lines of code job = {} # Below defines the name of the Qube! job job['name'] = 'python parent job'Pipelinefx L.L.C. (C) COPYRIGHT Pipelinefx Limited Liability Corporation. All Rights Reserved. US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with Pipelinefx L.L.C. $DateTime: 2013/04/11 15:06:06 $ $Revision: #1 $ $Change: 11098 $ $File: //depot/main/qube/examples/python/jobSubmit_dependency-example.py $ Description: This script submits a pair of dependent jobs, the second using the "dependency" parameter to wait for the first. """ import os, sys # The next few lines attempt to import the Qube API. If the path to the qb module # is not in $PATH or $PYTHONPATH, we will attempt to find it by looking in known # locations try: import qb except ImportError: if os.environ.get("QBDIR"): qbdir_api = os.path.join(os.environ.get("QBDIR"),"api","python") for api_path in (qbdir_api, # Below defines how many Instances/subjobs the job is to spawn "/Applications/pfx/qube/api/python/", job['cpus'] = 1 # Below defines the internal Qube! jobtype to be used to execute the job "/usr/local/pfx/qube/api/python/", job['prototype'] = 'cmdrange' "C:\\Program Files\\pfx\\qube\\api\\python", # The below parameters are explained further in the "Job submission with job package explained" page package = {}"C:\\Program Files (x86)\\pfx\\qube\\api\\python"): job['package'] = package if api_path job['package']['cmdline'] = 'sleep QB_FRAME_NUMBER' # Below defines the Agenda/Range of the job this will fill the Frames/Work section of the Qube! GUI # "0-60x10" is range 0-60 in chunks of 10 framesnot in sys.path and os.path.exists(api_path): sys.path.insert(0,api_path) try: agendaRange = '0-60x10' import qb # Below defines the internal command required to generate the agenda except: agenda = qb.genframes(agendaRange) #continue Below defines the job details for the agenda break job['agenda'] = agenda # Belowthis createsshould throw an emptyexception listif filledwe've byexhuasted theall followingother linespossibilities of code import qb listOfJobsToSubmit = []def main(): # BelowWe're evaluates the Parent job going to besubmit submittedtwo andjobs addsin the toexample - the abovesecond listwill wait listOfJobsToSubmit.append(job) # for the first # BelowSet evaluatesup the Parentbasic job toproperties beas submittedwe've anddone addsbefore. the toWe're thegoing aboveto listdo listOfSubmittedJobs# = qb.submit(listOfJobsToSubmit) # Below calls the Parent job to be submitted and then prints the job ID parentJobID = listOfSubmittedJobs[0]['id'] print 'parent: %d' % parentJobID # ----------------Start creation of Child Job---------------------------------------- # Below creates an empty dictionary to be filled by the following lines of code job a "sleep" job, so we'll limit this test to osx or linux job = {} job['name'] = 'python parent job' job['cpus'] = 2 job['prototype'] = 'cmdrange' job['requirements'] = 'host.os=linux or host.os=osx' # For the package, we'll just do a simple sleep package = {} package['cmdline'] = 'sleep 5' # Below defines the name of the Qube! job job['package'] = package # job['name'] = 'python child job' Create a typical agenda, as we've done in previous examples. agendaRange # Below defines how many Instances/subjobs the job is to spawn job['cpus= '0-60x10' agenda = qb.genframes(agendaRange) job['agenda'] = 1agenda # Now we submit #a Belowsingle definesjob how(as manya Instances/subjobssingle theelement joblist). is to spawn # This job['prototype'] = 'cmdrange'will be the parent job listOfJobsToSubmit = [] # Below defines the jobs dependancy which links back to the first job in this scriptlistOfJobsToSubmit.append(job) listOfSubmittedJobs = qb.submit(listOfJobsToSubmit) parentJobID = listOfSubmittedJobs[0]['id'] print job['waitfor'] ='parent: %d' % parentJobID #=================================================================# # TheNow belowwe'll parameterscreate area explainedsecond furtherjob in that will be dependent on the "Jobfirst submission with job package explained"# pagethe basic setup is the package same as before job = {} job['packagename'] = package'python child job' job['packagecpus'] = 1 job['cmdlineprototype'] = 'sleep 20' cmdrange' job['requirements'] = 'host.os=linux or host.os=osx' # BelowTo createscreate anthe emptydependency liston filledthe byparent, thewe followingsimply linescreate ofa code # listOfJobsToSubmit = [] 'dependency' attribute. See link in this doc's text for more information job['dependency'] = 'link-done-job-%d' % parentJobID # BelowContinuing evaluateson, thewe Childcreate jobsa totypical be"sleep" submittedjob andfor addstesting the to the above listpackage = {} listOfJobsToSubmit.append(job) package['cmdline'] = 'sleep 20' job['package'] = package # BelowAs calls before, we submit the Childsingle jobsjob toas bea submittedsingle andelement thenlist prints the job IDs listOfJobsToSubmit = [] listOfJobsToSubmit.append(job) listOfSubmittedJobs = qb.submit(listOfJobsToSubmit) for job in listOfSubmittedJobs: print 'child: %d' % job['id'] # Below runs the "main" function if __name__ == "__main__": main() sys.exit(0) |
This job uses
job['waitfor'] = parentJobID
To assign the dependancy to the child job so that it waits for the Parent job to complete
Note: The child job will run no matter the outcome of the Parent job whether it fails or completes
A non edited version of this script can be found along with others :
- Windows - C:\Program Files\pfx\qube\examples\jobSubmit03.py
- OSX - /Application/pfx/qube/examples/jobSubmit03.py
- Linux - /usr/local/pfx/qube/examples/jobSubmit03.py
Online - Python
Running this script will create 2 jobs. The first will run immediately, the second will be in a blocked state until the first job completes, then it will automatically start running.
For more information on the dependency attribute syntax, see Specifying Job Dependencies
Continue to Advanced Dependencies