1. Qube
2. Qube! Documentation
3. Start Here
4. Monitoring Jobs in the UI

Monitoring Jobs in the UI

Now that you have submitted some jobs, you will want to monitor them in the Qube UI. In fact, there are now two Qube UIs - the original, now called Wrangler View (because it contains features that an administrator or render wrangler would use) and qubeArtistView, intended for artists to monitor and interact with their jobs.! UI.

If you have been working through the Qube 101 documentation in order, you will probably be running the Wrangler View, which has a white background and an "Administration" menu item across the top. In this section we are going to skim across the surface of the qubeArtistView UI, but most of the things covered here will also be true in the Wrangler View, though you may have to look in slightly different spots to locate themQube! UI. Visit the Qube! UI User's Guide for more information.

Launch the Qube

...

! UI

To launch the Artist ViewQube! UI, do this:

• Windows: Click on the black and white Qube ArtistView desktop icon, or locate it in the PipelineFX->Qube! UI start menu item, or double click it at Program Files\pfx\qube\bin folder (Note: not the "Program Files (x86)" folder)QubeUI\QubeUI.bat.
• OS X: Click on the black and white Qube ArtistView desktop icon, or locate it in the Run /Applications/pfx folderLinux: Type 'qubeArtistView' at the shell prompt (no quotes)/qube/QubeUI.app
• Linux: Run /usr/local/pfx/qube/qubeui/qubeui.

You should see this:

Image RemovedImage Added

You should see the test jobs or other jobs that you have submitted. Notice that they are color coded - if you are lucky, they will all be blue, which is the color of a successfully completed job. Jobs which failed are colored red. The jobs also have a status, such as "complete", "failed" or "running."  Running jobs are shown in green. There are other colors for other statuses, which you will become familiar with as you submit and monitor jobs.

...

The row of buttons just under the Jobs tab is a set of filters which will show or hide jobs in the Jobs Panel. An important one to know about is the User Filter - when set, you will only see your own jobs. When unset, you will see all jobs. Click and unclick this button to see the difference. Note that if you are on Linux, you may have submitted jobs as the root or qubeproxy user which will be hidden if you choose this filter.<image of filter buttons>

Image Added

Let's have a look at some simple tasks you can do from this UI. First, The Jobs pane shows your job(s). By default, they are sorted by Jobid, but you can sort them by any other column by clicking on the column header. For example, you can sort them by Name by clicking on the "Name" label at the top of that column. Clicking once sorts them in ascending order, clicking again sorts them in the other direction. Now, click on one of your test jobs in the Jobs pane.

<image of jobs panel>Image Added

Notice that in the Job Properties panel below the Jobs panel, you can look at the Job Properties on one tab, and the Output on the next two tabs (Output Log & Error Log). This is the Job Details panel - we'll cover that in a second.

<image of Properties panel>

Interact with a Job

Let's retry one of your test jobs. You might want to do this if the job is working, but you have changed some of the input files and want to rerun it without changing any of the job's parameters. Just right-mouse on the job line in the Jobs panel, and choose Retry:

<image of right mouse menu, with retry highlighted>

If you want to rerun the job, but change some of the submission parameters - maybe the frame range was wrong, or you want to send the output to a different location - then you want to resubmit. Again right-mouse on the job line in the Jobs panel, and choose Resubmit. In this case you will get a Qube submission dialog, and you can modify any of the parameters before hitting "Submit". Note that when you resubmit, you are creating a new job with a new job ID. This is different from retrying an existing job.

Finally, you may want to kill a job. If you have a long running job and you know the results are going to be wrong, or you want to stop it for some other reason, you can kill the job. This is also a right-mouse button menu entry.

<image of right mouse menu, with kill highlighted>

There are other job-related tasks you can perform - for more details, see the User's Guide.

Also output in the Stdout Log and StderrLog panels.

Image Added

Interact with a Job

Right-clicking on a job (or a frame, worker, or instance, for that matter), will bring up a context menu:

Image Added

The contents of that menu depend on the entity you clicked.  For example, jobs will have a different list of menu items than will frames or workers.  The menu items are dictated by plugins which can be created or modified.  A more detailed description of the creation or modification of plugins can be found in Adding Custom Plugins.

As of version 7.5-0, the list of available actions are as follows:

• Job
  • Copy IDs to clipboard: Place the job ids of the selected jobs in your copy/paste clipboard.
  • Filter by PGRP: Display only jobs that belong to the selected job's PGRP.
  • Interrupt: Stop the job immediately, and sets the status back to pending.  This is typically used when a high priority job is starved for resources - manually interrupting a lower priority job will free up those resources to allow the high priority job to start.  The lower priority job to start up again as soon as workers become available.
  • Modify Job: Change modifiable job properties.
  • Open log directory: Open the directory containing the job logs for this job. This is only relevant if you're on the worker running the job or supervisor.
  • Prioritize: Change the priority of the selected job.
  • Remove: Stops the job immediately (if running), and permanently removes the job from the qube database.  This cannot be undone.
  • Shove: Force the supervisor to reevaluate this job for dispatch. This is a good way to start a job that is stuck in the "pending" state, even though all of its requirements can be met.
  • Create a zip file of the log directory: Zip the log directory and open the containing folder.
  • Block (Finish Current): Stops this job after the currently rendering frame(s) complete.  The job will remain in a "blocked" state until it is manually unblocked.
  • Block (Purge): Stops this job immediately, but sets the job state to "blocked".  The job will not run again until it is manually unblocked.
  • Unblock: Remove the "blocked" state from the job.  All frames that have not completed will go back to a "pending" state.  Those that have completed or failed will remain completed or failed.
  • Kill Job: Stops this job immediately and sets the state to "killed".  A job that is killed cannot be continued - it can only be retried, resubmitted, or removed.
  • Preempt Job: Preempt the selected job.
  • Resubmit Job: Submit a new job with the selected job's parameters. A submission dialog is opened so you can modify the job before submission.
  • Retry Failed Frames: Re-render all frames marked as failed
  • Retry Job: Start this job over, using the same job id.  All frames will be re-rendered, regardless of whether or not they had previously completed.
  • Browse Job Output Dir(s): Open a file browser for each output dir in this job.
• Worker
  • Ban host: Ban the selected worker on the supervisor.
  • Partially Lock: Lock a specific number of slots on the selected worker.
  • Ping: Ping the selected workers and retrieve basic information about them.
  • Reboot host: Reboot the selected workers.
  • Shut down host: Shut the selected workers down.
  • Lock Host (Finish Current): Same as above, but the currently rendering frames are allowed to finish.
  • Lock Host (Purge): Stop all jobs running on this worker immediately, then leave the worker in a "locked" state which will prevent it from accepting any more work until it is manually unlocked.
  • Unlock Host: Remove a worker lock.
  • Remove: Remove the selected workers from the Qube! farm.
• Frame
  • Select worker: Display the Workers Panel and select the worker than rendered this Frame.
  • Complete: Change the state of this frame to "complete," even if it has not yet run.
  • Retry: Re-render the selected frame(s)
  • Block: Set the frame's state to "blocked".  This will prevent the frame from being rendering until it (or the job) is manually unblocked.
  • Unblock: Remove the block from the frame, setting it back to a pending state.
  • Kill: Stop this from immediately and set the state to "killed".
• Instance
  • Select worker: Display the Workers Panel and select the worker than ran this instance.
  • Block: Stop this instance from processing work and set its state to "blocked".  This instance will not process any more work until it is manually unblocked or the job is retried.
  • Unblock: Remove a block from an instance.
  • Complete: Mark this instance as complete.
  • Interrupt: Stop this instance from processing any more work, but set its state back to "pending".  As with a job's "interrupt" this is typically done to manually free up resources that could be used by higher priority jobs.
  • Migrate: Stop this instance and set its state back to "pending".  Default supervisor settings dictate that the next time this instance starts, it will start on a different machine.
  • Retry: Restart this instance, regardless of its current state.
    
    

Watching Progress

Once a job is running, if it takes more than a few seconds (and most do) then you will want to keep an eye on progress. By default, the UI does not refresh itself, meaning that as the job progresses, nothing will change in the interface. You can force a change by clicking on the refresh button

<image of refresh button>

or you can configure Qube to automatically update the UI every so often. To do this, go to Edit in the top menu and choose Preferences. Click on Auto Refresh / Interval. You can leave the refresh interval at 60 seconds, or make it shorter or longer.

<image of properties dialog>

Note that asking the Supervisor for an update every few seconds puts more load on it, and if there are many artists all doing the same thing, you could be slowing down the overall farm just for the sake of convenience. Think twice before asking for lots of updates you aren't going to be noticing anyway.

Looking at Job Details

The Job Details panel has a number of tabs. The first one, Job Properties, lists a number of attributes of the job, including the frame range and the command and/or scene file used.

The next two are useful for finding out what went wrong with jobs that fail (or getting logging information from jobs that succeed). Most output will go to the first tab, Output Log. Some programs will send error messages and other information to a separate stream, which goes to the Error Log. Scroll to the bottom of either of these - error messages are coded in red, warnings in yellow and info (as determined by the program being run) in green.  All of the tabs and their uses are explained in more detail in the User's Guide.

Submitting New Jobs

The Artist View The UI will display changes to Jobs and Workers in real-time but frames and instances are only loaded when a job is selected. To update the frame and instance lists double click on the job. You can force an update by simply clicking on the job. Due to network interruption it is possible for some jobs to become out of sync, if you suspect this has happened click the sync button shown in 1 below.

Image Added

Use the sync button sparingly as it can place undue load on the supervisor on large farms.

Looking at Job Properties/Internals and Output

The Job Properties Panel lists basic attributes of the job. The Job Internals Panel shows more in-depth information such as the job package, callbacks in use and the job history.

The output and error of the job's rendering process can be viewed in the Stdout Log and Stderr Log Panels. You can review the image output in the Thumbnails Panel and Preview Panels. Check the User's Guide for the Qube! UI for more information on how to use these panels.

Submitting New Jobs

The Qube! UI can be used to submit jobs to Qube, ! through the Submit menu item at the top. However, in most cases you will be setting Qube up to allow submission directly from your application, such as Maya or 3dsMax. Submitting directly from applications was covered in the previous section. However, you can still use the UI to submit test jobs, to make sure that things are still working as expected, and to submit jobs for applications where you haven't installed the inApp interface.

The Frames Panel

The right side of the Artist View UI contains the Frames panel, which lists information about all the frames in the selected job. Notice that it has a separate refresh button, which allows you to query the status of just those frames. You can also right-mouse on any of the frames and kill them, retry them, suspend (block) them (and then unblock them), as well as marking them complete.