1. Qube
2. Python job submission with Python language callback that runs an external script on the supervisor

Python job submission with Python language callback that runs an external script on the supervisor

Please note the use of Popen.subprocess() to run the external script in the callback. This returns immediately and allows the callback to continue running, rather than blocking and waiting for the external script to complete; this ties up the supervisor process for the duration of the external script's execution.

Do not use os.system() to run the external script, as this call will block until the external script exits. When a large number of callbacks tie up supervisor processes at the same time, your supervisor performance will suffer.

Do not call sys.exit() at the end of the callback code, this kills the calling supervisor process.

#!/bin/env python

import sys,os

sys.path.append('%s/api/python' % os.environ['QBDIR'])
import qb

scriptPathOnSupervisor = '/Users/jburk/test/testCbScript.py'

########################################
# build the job
########################################
job = {
    'prototype': 'cmdrange',
    'name': 'supervisor-side script execution callback test',
    'package': {'cmdline': 'hostname'},
    'agenda': [],
    'callbacks': [ ]
}
########################################
# iterate over the frame range to build the job's agenda and callbacks
########################################
for i in range(5):

    # an agenda item can simply be a frame number
    work = {'name': i}
    job['agenda'].append(work)

    # build a callback for each item in the agenda (each frame)
    cb = {
        'language': 'python',
        'triggers': 'done-job-self', 
        'code': '''
import os,subprocess

jobId = qb.jobid()
workName = %s

script = '%s'
if os.path.exists(script):
    pid =  subprocess.Popen(["python", script, str(jobId), str(workName)]).pid
''' % (i, scriptPathOnSupervisor)
    }

    # append the frame's callback to the job's callback list
    job['callbacks'].append(cb)

########################################
# submit the job
########################################
submitted = qb.submit(job)
for job in submitted:
    print 'submitted %(id)s: %(name)s' % job