Overview
SmartShare is a feature where the system automatically expands job instances to fill up idle, or "surplus", worker job slots. The expansion is automatically balanced among jobs, so that each job gets an equal number of surplus job slots.
Jobs have a 'max_cpus' value which puts an upper limit on how wide a single job can expand. This value can be supplied via a site-wide default or specified on a job-by-job basis.
SmartShare from a user's perspective
When SmartShare is enabled, jobs will automatically start using it. Note that SmartShare is only only applicable to agenda-based jobs.
It is possible to control how much a job automatically expands, on a job-by-job basis, by setting a new job attribute called "max_cpus" at submission. This value sets an upper limit to the number of instances to which the job will expand. The max_cpus parameter is only visible in the submission UI when in Expert Mode.
If an end-user does nothing, their job will expand as wide as the system administrator has allowed (as specified in the various supervisor_*_max_cpus_* settings); the administrator can specify a default and a maximum value for max_cpus.
Jobs will expand to their max_cpus value if there are enough empty slots on the farm. As other users begin to submit jobs as well, a user's job will shrink to allow for a more "balanced" usage of the farm between jobs. No job will shrink below its "instance count" (or "cpus") value.
For example, a job submitted from the command line "qbsub" utility as in :
ends up with 10 primary instances as specified by the "cpus" option at priority 5, and may expand up to 50 instances if surplus job slots are available. In other words, as many as 40 secondary instances will be dispatched, at a lower priority.
"max_cpus" may be set to "*" (asterisk) or a negative number, such as "-1", to mean "no limit", which will allow the job to expand to the system's upper limit, as configured by the site-administrator (see supervisor_max_cpus_limit below).
Setting "max_cpus" to 0 effectively disables SmartShare for the job.
If "max_cpus" is unspecified at submission, the system will apply the default value, which is configured by the site-administrator by supervisor_default_max_cpus.
The "max_cpus" value may also be modified after submission, by the GUI, qbmodify command, or via the API.
Note that the instances that were automatically expanded by SmartShare ("secondary instances") run at low priority, and are the first subject to preemption in order to run primary instances from other jobs. Secondary instances may also be preempted in order to run secondary instances from other jobs, to balance the number of secondary instances among all jobs.
The method of preemption can either be "aggressive", meaning the instance is immediately terminated, or "passive", meaning the instance is allowed to finish processing the current agenda item before terminating, and is configured by the site-administrator (see "supervisor_smart_share_preempt_policy" below).
Configuring SmartShare
- supervisor_smart_share_mode
- Can be either “jobs” or “none”.
- default: "jobs"
- Setting it to “jobs” will make the system try to balance the number of the automatically-expanded “secondary" instances on a job-to-job basis
- Setting it to “none” disables SmartShare.
- supervisor_smart_share_preempt_policy
- method of preemption for secondary instances
- aggressive, passive, disabled
- default: aggressive
- supervisor_default_max_cpus
- used to set the site-wide default value of “max_cpus” if not given at submission
- default: -1
- Note: “*” can be used in the qb.conf file too, instead of -1
- supervisor_max_cpus_limit
- largest value of “max_cpus” that can be used on any job
- default 100
- Note: if any job specifies a value greater than this, the system will silently cap it to this value