Versions Compared

    Key

    • This line was added.
    • This line was removed.
    • Formatting was changed.
    Comment: Published by Scroll Versions from this space and version 6.10-0

    Jobs are

    Table of Contents
    maxLevel3

    Overview

    Jobs can be submitted from WranglerView using the items under the Submit menu. Selecting one will launch a modal submission dialog with the standard framework and job-specific parameters. Specify the fields and then press the "Submit" button to submit the job to the Qube! Supervisor.

    Submissions for specific applications, such as Maya and Nuke, are explained in detail in their own sections. Please refer to that documentation for submissions to individual applications. This page documents a large set of parameters; individual submission targets do not necessarily expose all the parameters defined here.

    Buttons

    There is a common set of buttons located at the bottom of the Submission Dialog.

    • Presets Set Defaults: Store as defaults in the User Preferences for that Jobtype, the values listed in the current submission dialog. The interface will by default use those values by default the next time the dialog is opened. This allows one you to specify common fields like Priority or Executable that should always be the same value.
    • Presets Clear Defaults: Remove any stored defaults for that Jobtype submission dialog
    • Expert Mode: Toggle button to display or hide export and non-default values from the submission dialog to reduce clutter when there are a lot of parameters. The current state of this toggle is stored in the Preferences Dialog.
    • Save [Disk Icon]: Store the current properties of the job from the dialog as a file (by default an xml XML file). This can be used to submit through the WranglerView at a later time with "Submit->File…" or from the qbsub commandline or command line with the qbsub command, or via the API.
    • Cancel: Cancel the job submission and close the dialog.
    • Submit: Submit the job with the specified parameters to the Qube! Supervisor.

    Common Parameters

    Basic Parameters for all Jobs

    ...

    • Name: name used for the job, usually artist specified.
    • Priority: priority of the job. The lower the number means the higher the A number between 1 and 9999. Lower numbers mean higher priority.
    • CPUsInstances: The number of processes (or subjobs/instances) to run the job with at the same time. When rendering, this equates to the number of frames being rendered at the same time.
    • Max Instances: (Expert Mode only) The maximum number of additional instances to run using smartShare. The default of -1 means "unlimited"; a setting of 0 (zero) disables smartShare expansion for this job.

    Render Thread and Job Reservation Controls

    This section is located near the top of the submission UI, and is collapsed when 'expert modeExpert Mode' is not selected.  For applications that do not support setting the number of threads, this section is not visible at all.

    Image RemovedImage Added

    Enabling 'Expert modeMode' at the bottom of the UI will open this field upImage Removed

    Image Added

    Tip

    Use Slots=Threads only on farms where the workers are configured to have as many slots as they have cores; they will show 0/8 or 0/16 in the Slots when they're empty, where the 0/N number is the number of free slots. This is the default configuration in Qube.!

    The other common configuration is all the worker set to only ever accept 1 job, where they show 0/1 in the Slots column in the UI. Do not use Slots=Threads in this configuration, since none of your workers will have enough free slots to accept the job.

    Thread Control Behavior

    ...

    Checking Use All Cores for applications or renderers that support auto-detection of the number of cores installed on a worker host will set the renderer's appropriate control to enable this feature; in . In the case of the a Maya job, it sets renderThreads=0, ; for the mentalray Mentalray renderer, it set sets autoRenderThreads=True.  It will try to "do the right thing" for each application where this control is visible.Image Removed

    Image Added

    ...

    Image Added

    ...

    Image Added

    Checking Render on all cores will also sets the job's reservation string to reserve all cores: you will set a "+" at the end of the host.processors=N+ . This means "only start on a worker with N free slots, but reserve them all, so that no other jobs will start on this worker while I'm running on it".  The Min Free Slots value will affect the value of N, so setting it to 4 will say "start only on a worker with 4 free slots".

    ...

    Setting Specific Thread Count, on the other hand, does not necessarily reserve all the cores on the worker, it only sets the renderer's particular number of threads control, in Maya's case it's Render Threads:

    ...

    Image Added

    But you will notice that the job's reservations reservations have not changed, and is probably still at the default value of host.processors=1.  This is to support workers which are configured for less fewer slots than they have cores.

    Image RemovedImage Added

    The Slots=Threads control will link the job reservations to the thread count, so a 4-threaded job will reserve 4 worker slots.  It also disables the Render on all cores and Min Free Slots controls.

    Image RemovedImage AddedImage Removed

    Image Added

    As you change the Specific Thread Count control the reservations value will automatically update.

     

    Parameters for Cmdline Jobs

    ...

    • Command: The command to run via the commandline on the Worker. Paths and syntax should be what the Worker's OS expects, not the submitting machine.
    • Shell (Linux/OS X & Linux): Specify the shell to use when executing the commandline command line on the Worker. Only visible in Expert Mode.

    Parameters for Cmdrange Jobs

    ...

    • Command: Same as the Cmdline Job, though it will substitute the following strings based on the frame being executed for the given task.
      • QB_FRAME_NUMBER
      • QB_FRAME_START
      • QB_FRAME_END
      • QB_FRAME_STEP
      • QB_FRAME_RANGE
    • Range: The frame range to execute.
    • Execution
      • Individual Frames
      • Chunks with n Frames
      • Split into n Partitions
    • Ordering: Specify whether tasks should be executed in ascending, descending, or binary sort (first, last, middle, split the middle values, …) order.

    ...

    Parameters for SimpleCmd Jobs

    ...

    • All the fields for either Cmdline or Cmdrange jobs (see above), plus
    • Cmd Template: String used to construct the commandline command line along with the rest of the job parameters. Python string representations are used, like e.g. %(val)s to represent the string value from "val". The If listed, %(argv)s if listed means to put places all optional arguments at that location instead of at the end. This constructs a commandline command line that is then used by the Cmdline or Cmdrange jobtypes on the Workers. See above for the string replace values for CmdRange.
    • Executable – path to the renderer or executable to run. Unless it is in the path on the Worker's environment, this needs to be set to an absolute path (for to where the executable is located on the Worker)

    Detailed

    ...

    Parameters

    Preview Frames Submission

    Include+
    scrollPageId405BE23F014B092BECDB9A2A7F86FA3C
    scrollEditorDisplayTitle_SimpleCMD_PreviewFramesSubmission
    scrollEditorUrlhttp://docs.pipelinefx.com/display/QUBE/._SimpleCMD_PreviewFramesSubmission+v6.6-3


    Qube Worker Selection

    • Hosts: list of Workers that the job can run on.
    • Omit Hosts: list of Workers that the job is prevented from running on.
    • Groups: collection of Workers that the job can run on.
    • Cluster: Job will have priority to run on the Workers that are in this (Priority) Cluster. It does not restrict the Job from only running on the Workers in the Cluster, but just gives it a higher priority to run on them
    • Host Order: select how the Qube Supervisor will first pick which Workers to run the Job. For example, choose the fastest Workers first.
    • Requirements: Worker requirements needed to be met for job to run on that Worker (comma-separated, expression-based). This is often something used to limit the job to run on a certain OS.
    • Reservations: Worker resources to reserve when running job (comma-separated, expression-based)". This is commonly used to track license usage and only run on machines with certain amounts of free memory or use multiple process slots.
    • Restrictions: Restrict job to run only on specified clusters (logically OR'd together with "||") [+ means all below, * means at that level]"
    Qube Advanced Job Control
    • Flags: List of submission flag strings (comma separated). See reference for full list of available flags and their meanings.
    • Dependency: Have job run after the completion or status change of another job or jobs. Can be set for per-job or per-frame for the dependencies. See Job Dependency Graph Panel for displaying dependencies under Job Detail Panels (Section 4.1.5).

    Valid Formats:

      • <jobid>
      • complete-job-<jobid>
      • link-<state-<type>-<jobid>
      • info-<state>-{'',[]}
        • special chars:
        • '*' mean each frame
        • '' (blank) means entire job

    Examples:

      • link-complete-job-529,  link-complete-work-430, info-complete-{'1':['540:3'], '':['600:'], '':['700']} , link-complete-job-300, 900, complete-job-123
      • info-<state>-{'':['123','456'], '':['345:'], '10':['346:2', 346:4']}
    • Email: Email the specified user on job completion.
    • Blocked: Submit the job, but do not put it in the Queue to execute.
    • Stderr->Stdout [New for Qube 6.0]: Redirect and consolidate the job stderr stream to the stdout stream.
    • Job Label: Representative label used to help identify jobs within a specified Job Process Group
    • Process Group: Explicitly set the Job Process Group for the job so it can be logically grouped with other jobs. By default the Process Group will be the same as the JobID.
    • Retry Subjob Process: Number of times to retry a failed subjob process. Value of -1 means use the default for the studio.
    • Retry Frame/Work: Number of times to retry a failed frame/work. Value of -1 means use the default for the studio.
    • Subjob Timeout: Kill the subjob process if running for the specified time (in seconds). Value of -1 means disabled.
    • Frame Timeout [New for Qube 6.0]: Kill the agenda/frame if running for the specified time (in seconds). Value of -1 means disabled.
    Qube Job Enviroment
    • Cwd: (Current Working Directory) Explicitly specify the current working directory where the Worker should start the process
    • Env vars: (Environment Variables) Explicitly set specific environment variables that should be used when running a job process on the Worker.
    • Impersonate User: Submit job as specified user. Default is current user. Format: <optionaldomain>\<username> (advanced – requires Impersonate user permission)
    Qube Output Parsing and Validation [New for Qube 6.0]
    • regex_highlights: Regular expression for highlighting information messages from stdout/stderr
    • regex_errors: Regular expression for identifying fatal errors from stdout/stderr
    • regex_outputPaths: Regular expression for identifying outputPaths of images from stdout/stderr
    • regex_maxLines: Maximum number of lines to store for regex matched patterns for stdout/stderr
    • validate_fileMinSize: Minimum size for identified outputPaths (in bytes). [0 disables test]
    Qube Actions [New for Qube 6.0]
    • generateMovie: add linked job to generate movie from output images. It will launch a second submission dialog to convert the images to a movie. This addresses the common action chain of rendering images and then converting those images into a Quicktime or similar movie. Note: This function will only work on jobtypes that retrieve the output image filenames into Qube (Maya, 3dsMax, Softimage, AfterEffects, etc).
    Shotgun Submission [New for Qube 6.0]
    • Login: login username for Shotgun
    • Task: Shotgun task
    • Project: Shotgun project
    • Shot/Asset: Shotgun shot/asset
    • Version Name: Shotgun version template. Performs variable substitution using python format %(key)s to get the value. The dict keys are prefixed with "job.", "package.", and "shotgun." respectively.
    • Description: Shotgun description. Performs variable substitution using python format %(key)s to get the value. The dict keys are prefixed with "job.", "package.", and "shotgun." respectively.
    Qube Notes

    Job Flags

    Include Page
    _jobFlags_jobFlags
    Include+
    scrollPageId405BE23F014B092BED0EE4246B9B43C9
    scrollEditorDisplayTitle_SimpleCMD_QubeWorkerSelection
    scrollEditorUrlhttp://docs.pipelinefx.com/display/QUBE/._SimpleCMD_QubeWorkerSelection+v6.6-3


    Qube Advanced Job Control

    Include+
    scrollPageId405BE23F014B092BED21EDF55CD47477
    scrollEditorDisplayTitle_SimpleCmd_AdvancedJobControl
    scrollEditorUrlhttp://docs.pipelinefx.com/display/QUBE/._SimpleCmd_AdvancedJobControl+v6.6-3


    Qube Job Environment

    Include+
    scrollPageId405BE23F014B092BED2900AF0E13DE30
    scrollEditorDisplayTitle_SimpleCMD_QubeJobEnvironment
    scrollEditorUrlhttp://docs.pipelinefx.com/display/QUBE/._SimpleCMD_QubeJobEnvironment+v6.6-3


    Qube Output Parsing and Validation

    An optional basic file size check for the cmdline/cmdrange simplecmd framework jobtypes provides basic image validation if the path to the image can be determined. If an image is identified from parsing the stdout or stderr from a cmdline/cmdrange job (or any job using the simplecmd framework), it can have rudimentary validation automatically performed on it to make sure that it is valid. 

    For this to work, the regex_outputPaths submission parameter needs to be specified for it to parse the stdout/stderr of the job for images, or manually specifying the job frame's path (in Python the paths are represented by jobFrame['resultPackage']['outputPaths']).

    Include+
    scrollPageId405BE23F014B092BED5DAC8C0718B03F
    scrollEditorDisplayTitle_SimpleCmd_jobValidation
    scrollEditorUrlhttp://docs.pipelinefx.com/display/QUBE/._SimpleCmd_jobValidation+v6.6-3


    Qube Actions

    Include+
    scrollPageId405BE23F014B092BED37493F7556F3F0
    scrollEditorDisplayTitle_SimpleCMD_QubeActions
    scrollEditorUrlhttp://docs.pipelinefx.com/display/QUBE/._SimpleCMD_QubeActions+v6.6-3


    Shotgun Submission

    Include+
    scrollPageId405BE23F015D25359C7BBCA7143E4A21
    scrollEditorDisplayTitle_SimpleCMD_ShotgunSubmission
    scrollEditorUrl/display/QUBE/._SimpleCMD_ShotgunSubmission+v6.10-0


    Qube Notes

    Include+
    scrollPageId405BE23F014B092BED3FFA6C6ABA7BCD
    scrollEditorDisplayTitle_SimpleCMD_QubeNotes
    scrollEditorUrlhttp://docs.pipelinefx.com/display/QUBE/_SimpleCMD_QubeNotes