There are a number of manual tasks typically performed when overseeing jobs on the renderfarm and the renderfarm itself. The header "auto-wrangling" is meant to cover automating some of those more manual tasks and calculations.
Some of the most common render wrangling tasks can now be handled automatically by the Supervisor through a set of global parameters. Qube! has added built-in logic that detects faulty jobs and workers. These parameters can be configured by modifying the Supervisor qb.conf directly or through the WranglerView "Administration->Config (local)" dialog.
Automated Actions
- Automatically blocks faulty jobs and sends email to the submission user.
- Automatically locks faulty workers and sends email to the Qube Administrator.
To enable:
- Make sure that the supervisor_language_flags includes "auto_wrangling" (it's ON by default, if "supervisor_language_flags" isn't found in qb.conf).
- supervisor_language_flags, which lists the enabled callback "languages", must include "auto_wrangling" to enable auto-wrangling.
- To globally turn on auto-wrangling for all submitted jobs (recommended), include "auto_wrangling" in the supervisor_job_flag" configuration parameter in the supervisor's qb.conf. Restart the supervisor if qb.conf settings were changed.
- Auto-wrangling also may be enabled on a job-by-job basis, by NOT including "auto_wrangling" in "supervisor_job_flags", and explicitly setting the "auto_wrangling" flag for jobs.
- Auto-wrangling also may be enabled on a client-to-client basis, by NOT including "auto_wrangling" in "supervisor_job_flags", but including it in the "client_job_flags" on desired submission machines. This will enable auto-wrangling on all jobs submitted from those machines.
To disable:
- Remove "auto_wrangling" from the "supervisor_language_flags", in the supervisor's qb.conf file. Have the Supervisor reload its configuration with
qbadmin supe --reread. See Telling the Supervisor to Reread the qb.conf file
Auto-Wrangling-specific qb.conf parameters:
- aw_activation_work_count : This positive integer specifies the number of agenda-items that should fail on a given worker before the auto-wrangling logic is activated. (default: 5)
- aw_job_migrate_max : This positive integer specifies the maximum number of automatic migration the auto-wrangling subsystem will perform on a job, before it decides that the job is faulty and blocks it, and notifies the submitting user. In other words, this effectively determines the number of workers on which a job will be tried, before the system "gives up". (default: 3)
Other related qb.conf parameters:
- supervisor_job_flags may include "auto_wrangling" which will globally turn on auto_wrangling for all submitted jobs (i.e., by automatically turning on the "auto_wrangling" flag for every job when submitted).
- The email address specified in the mail_administrator parameter will receive email messages when auto-wrangling determines that a worker is faulty and locks it.
Job parameters:
- A job's "email address" field, if set, is used as the recepient address of email notifications sent out by the auto-wrangling subsystem.
Email
The Auto-Wrangling system will send notification emails when it takes certain actions.
- When a job is blocked because auto-wrangling determined that it's malfunctioning, an email notification is sent to the user.
- When workers are locked because auto-wrangling decided that the worker is faulty, then the qube admin is notified via email.
Auto-wrangling Logic
Currently, the built-in auto-wrangling logic is basically an implementation of the following pseudocode: