Icon

This is the documentation for an older version of Qube. The latest version of the documentation can be found here: Qube

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The next step is to understand basic dependencies 

The method used below is not the best method for dependencies but is a short introduction to the theory behind dependencies 

 

Feel free to download an run the below script as it setup a job that will :

  • Create a Parent "Sleep job" with a range of 60 
  • Create a Blocked Child "Sleep job" that waits for the Parent to complete before starting

 

Basic_Dependancy.py

 

In this example, we use the job's dependency field to set up a job-based dependency.  Job based dependencies are those that will wait for the entire dependent job to finish before the current job starts.  There are also subjob (instance) and agenda (frame) based dependencies.

Below is the code with commented explanations of its contents contents

Code Block
titlejobSubmit_dependency-example.py
languagepython
#!/usr/bin/python
#"""
Below areLicensed requiredMaterials imports- forProperty theof script to run
import os, sys

# The below few lines of code are to determine the OS of the machine that your running 
# this script from and then define the location of the Qube! API
if 'QBDIR' in os.environ:
	sys.path.append('%s/api/python' % os.environ['QBDIR']);
elif os.uname()[0] == 'Darwin':
	sys.path.append('/Applications/pfx/qube/api/python');
elif os.uname()[0] == 'Linux':
	sys.path.append('/usr/local/pfx/qube/api/python');
else:
	sys.path.append('c:/program files/pfx/qube/api/python');


# The below line of code is to import the above defined Qube! API
import qb


# Below is the main function to run in this script 
def main():
    
    # ----------------Start creation of Parent Job----------------------------------------
    
    
    # Below creates an empty dictionary to be filled by the following lines of code 
    job = {}
    
    # Below defines the name of the Qube! job 
    job['name'] = 'python parent job'Pipelinefx L.L.C.
 (C) COPYRIGHT Pipelinefx Limited Liability Corporation.
  All Rights Reserved.
                               
  US Government Users Restricted Rights - Use, duplication or
  disclosure restricted by GSA ADP Schedule Contract with 
  Pipelinefx L.L.C.
 $DateTime: 2013/04/11 15:06:06 $
 $Revision: #1 $
 $Change: 11098 $
 $File: //depot/main/qube/examples/python/jobSubmit_dependency-example.py $
 Description:
  This script submits a pair of dependent jobs, the second using
  the "dependency" parameter to wait for the first.
"""

import os, sys

# The next few lines attempt to import the Qube API. If the path to the qb module
# is not in $PATH or $PYTHONPATH, we will attempt to find it by looking in known
# locations
try:
    import qb
except ImportError:
    if os.environ.get("QBDIR"):
        qbdir_api = os.path.join(os.environ.get("QBDIR"),"api","python")
    for api_path in (qbdir_api,
           # Below defines how many Instances/subjobs the job is to spawn "/Applications/pfx/qube/api/python/",
    job['cpus'] = 1          # Below defines the internal Qube! jobtype to be used to execute the job "/usr/local/pfx/qube/api/python/",
            job['prototype'] = 'cmdrange'       "C:\\Program Files\\pfx\\qube\\api\\python",
 # The below parameters are explained further in the "Job submission with job package explained" page     package = {}"C:\\Program Files (x86)\\pfx\\qube\\api\\python"):
    job['package'] = package  if api_path  job['package']['cmdline'] = 'sleep QB_FRAME_NUMBER'
	
	# Below defines the Agenda/Range of the job this will fill the Frames/Work section of the Qube! GUI
	# "0-60x10" is range 0-60 in chunks of 10 framesnot in sys.path and os.path.exists(api_path):
            sys.path.insert(0,api_path)
            try:
       agendaRange = '0-60x10'       import qb
 # Below defines the internal command required to generate the agenda except:
    agenda = qb.genframes(agendaRange)          #continue
Below defines the job details for the agenda     break
job['agenda'] = agenda 	 	# Belowthis createsshould throw an emptyexception listif filledwe've byexhuasted theall followingother linespossibilities
of code   import qb
listOfJobsToSubmit
= []def main():
     	# BelowWe're evaluates the Parent job going to besubmit submittedtwo andjobs addsin the toexample - the abovesecond listwill wait
   listOfJobsToSubmit.append(job) # for the first
 
    # BelowSet evaluatesup the Parentbasic job toproperties beas submittedwe've anddone addsbefore. the toWe're thegoing aboveto listdo
    listOfSubmittedJobs# = qb.submit(listOfJobsToSubmit)
    
    # Below calls the Parent job to be submitted and then prints the job ID
    parentJobID = listOfSubmittedJobs[0]['id']
    print 'parent: %d' % parentJobID
	
	# ----------------Start creation of Child Job----------------------------------------
	
	
	# Below creates an empty dictionary to be filled by the following lines of code
    job a "sleep" job, so we'll limit this test to osx or linux
    job = {}
    job['name'] = 'python parent job'
    job['cpus'] = 2
    job['prototype'] = 'cmdrange'
    job['requirements'] = 'host.os=linux or host.os=osx'
    
    # For the package, we'll just do a simple sleep
    package = {}
    package['cmdline'] = 'sleep 5'
 # Below defines the name of the Qube! job job['package'] = package

    #  job['name'] = 'python child job'
   Create a typical agenda, as we've done in previous examples.
    agendaRange # Below defines how many Instances/subjobs the job is to spawn
    job['cpus= '0-60x10'
    agenda = qb.genframes(agendaRange)
    job['agenda'] = 1agenda

    # Now we submit #a Belowsingle definesjob how(as manya Instances/subjobssingle theelement joblist). is 
to spawn   # This job['prototype'] = 'cmdrange'will be the parent job
    listOfJobsToSubmit = []
    # Below defines the jobs dependancy which links back to the first job in this scriptlistOfJobsToSubmit.append(job)
    listOfSubmittedJobs = qb.submit(listOfJobsToSubmit)
    parentJobID = listOfSubmittedJobs[0]['id']
    print job['waitfor'] ='parent: %d' % parentJobID

    #=================================================================#
 
    # TheNow belowwe'll parameterscreate area explainedsecond furtherjob in that will be dependent on the "Jobfirst
submission with job package explained"# pagethe basic setup is the package same as before
    job = {}
    job['packagename'] = package'python child job'
    job['packagecpus'] = 1
    job['cmdlineprototype'] = 'sleep 20'
    cmdrange'
    job['requirements'] = 'host.os=linux or host.os=osx'
 
    # BelowTo createscreate anthe emptydependency liston filledthe byparent, thewe followingsimply linescreate ofa
code    # listOfJobsToSubmit = []
   'dependency' attribute.  See link in this doc's text for more information
    job['dependency'] = 'link-done-job-%d' % parentJobID
    
    # BelowContinuing evaluateson, thewe Childcreate jobsa totypical be"sleep" submittedjob andfor addstesting
the to the above listpackage = {}
  listOfJobsToSubmit.append(job)
   package['cmdline'] = 'sleep 20'
    job['package'] = package
    
    # BelowAs calls before, we submit the Childsingle jobsjob toas bea submittedsingle andelement thenlist
prints the job IDs   listOfJobsToSubmit = []
    listOfJobsToSubmit.append(job)
     listOfSubmittedJobs = qb.submit(listOfJobsToSubmit)
    
    for job in listOfSubmittedJobs:
        print 'child: %d' % job['id']

# Below runs the "main" function 
if __name__ == "__main__":
    main()
    sys.exit(0)

This job uses 

job['waitfor'] = parentJobID

To assign the dependancy to the child job so that it waits for the Parent job to complete 

Note: The child job will run no matter the outcome of the Parent job whether it fails or completes

You can create as many Child jobs as you wish
Try copying the child job and pasting it into the script again ... This will create 2 child jobs dependant on the completion of the parent  

 

A non edited version of this script can be found along with others :

  • Windows - C:\Program Files\pfx\qube\examples\jobSubmit03.py
  • OSX - /Application/pfx/qube/examples/jobSubmit03.py
  • Linux - /usr/local/pfx/qube/examples/jobSubmit03.py

Online - Python

Running this script will create 2 jobs.  The first will run immediately, the second will be in a blocked state until the first job completes, then it will automatically start running.

For more information on the dependency attribute syntax, see Specifying Job Dependencies

Continue to Advanced Dependencies