In Obsidian, everything starts with jobs. They can be scheduled, constrained by parameters, and even restricted to running on specific hosts.
In addition to job implementation features, Obsidian supports the following features that apply to the execution and scheduling environment.
Full Execution History
Obsidian offers full transparency into when jobs run, complete and if they fail. In addition, Obsidian stores history of any time jobs are skipped, overlapped, missed due to downtime, etc.
Obsidian also stores saved job results, chaining evaluations and stack traces in the event of a job failure.
The following are the valid states a job execution may appear in.
- Ready - Ready to be executed at its scheduled time and awaiting pickup for execution.
- Running - Currently executing.
- Completed - Completed execution without error (i.e. without throwing an uncaught exception). This is a terminal state.
- Failed - Did not complete successfully. Exception stack trace is saved for review. This is a terminal state.
- Missed - Could not be executed at its scheduled time due to an outage. This is typically when Obsidian was not running but could also be caused by technical issues such as database outage, inability to verify licenses, etc. This is a terminal state.
- Conflicted - Currently in an executable window (past its scheduled time but before its pickup buffer has expired), but is conflicted with another job. Equivalent to Ready, but when a conflict exists.
- Conflict Missed - Was prevented from executing by a conflicting job until after its pickup buffer expired. This is a terminal state.
- Died - The job execution thread (or forked process) was terminated. This can happen when Obsidian is shut down before a job completes. This is a terminal state.
- Abandoned - Was in Ready state but its pickup buffer expired without an active scheduler instance available to begin execution. This can also happen when there are active scheduler instances, but due to use of Fixed Hosts, no running instances match the host designator(s) assigned to the job. This is a terminal state.
- Overlapped - Was due to be run at its scheduled time, but was overlapped by a prior executing instance of the job. This means the previous running job's completion time was after the job's pickup buffer expired. This is a terminal state.
- Chain Skipped - Configured as a chain target, but the chaining restrictions were not met, so the job is not executed. This occurs when conditional chain criteria are not met, or the job state does not match one of the trigger states. This is a terminal state.
- Pending - The state given to an Async Job that has completed in Obsidian but has not yet been updated via the API to indicate its result.
Execution Status Flow
Date Bound Scheduling
Obsidian supports setting the job's runtime schedule and state by date/time windows. This allows you to configure future changes to a schedule or job state. Obsidian also allows these changes to be scheduled in windows, so that a configuration will revert back to previous state at the completion of the window. This useful feature allows you to prepare for known outages, schedule holiday schedules in advance, and more, all without having to remember to make changes exactly when they need to go live.
Custom Calendars can also be configured with schedule changes to prevent execution on certain dates.
Jobs can also be parameterized, so that jobs can be reused in multiple configurations. Parameters allow for many varying instances of the same job class that differ by the specified parameters. See Parameterization.
Jobs can be restricted to run only on specified hosts. This allows you to use Obsidian in a multi-host environment, even when certain jobs can only run on one of the hosts in the cluster.
As of Obsidian 2.2.1, a configuration item allows you to specify whether you wish host restrictions to apply on AdHoc jobs or not. On new installations of 2.2.1, it defaults to TRUE while it defaults FALSE (to maintain behaviour from versions before 2.2.1) on upgrades from prior releases. It can by found under Scheduler Settings / Job / adHocJobsRespectFixedHostsRestrictions.
In addition to strict host selections, Obsidian 3.0 introduces the concept of host preference. For each job, you can select this option to enable prioritizing specific hosts for job execution. When a job configured with this setting is ready for execution, Obsidian will select the most preferred host to execute the job on, and if no preferred host is available, execution will fall back to any available host. This is configurable on the job configuration screen.
Obsidian has extensive recovery support to ensure reliable execution in the event of outages.
Each job is configured with a Pickup Buffer which indicates how many minutes after the scheduled time a job is considered valid for regular pickup and execution.
Every configured job also specifies a Recovery Type, which indicates how to handle a job that is not run within the Pickup Buffer. This might happen because of a configured conflict or because the scheduler was not running. Obsidian gives you fine control over how to handle these situations by exposing the following Recovery Types:
- Last indicates that only the last missed job run will be recovered.
- Conflicted indicates that only conflicted job runs will be recovered.
- All indicates that all job runs will be recovered.
- None indicates that no job runs will be recovered.
As of Obsidian 2.0, Obsidian also supports individual job failure recovery through automated retries. Each job can be configured to use Auto Retries. Specify a Count that will tell Obsidian how many times to keep trying (essentially an auto-resubmit) a failed job.
As of Obsidian 2.5.0, you can use Interval and Exponential as well. Use Interval to indicate the minimum number of minutes from the last failure to the next retry. Use Exponential if you want the auto retries interval to exponentially increase as retries are attempted. For example, if you set the interval minutes to 5 and check Exponential, the first retry will be 5 minutes after failure, the second retry after a subsequent failure will be 10 minutes later, then 20 minutes, and so on.
Resubmission & Ad Hoc Runs
Did a job fail unexpectedly? Or do you just need to run a job one time? Obsidian gives you the flexibility to resubmit failed jobs or submit them for a single execution regardless of their execution schedule. This can be done via our REST API, Embedded API or via the admin web application.
In addition, when you submit an ad hoc job run, you can supply job parameters which apply only to that execution. This can be useful for executing jobs for a particular one-time need. For example, if you run out of disk space on a host, you may have a log archival job which can be run against a specific directory for a single execution.
As of Obsidian 3.8.0, multiple Ad Hoc Runs may be submitted per minute for both immediate (next minute) execution and future execution.
Forced Termination (interruptable jobs)
Jobs that appear to be hung or need to be interrupted for any other reason can be terminated by having the job implement a simple interface. See interruptable jobs.
Expected Length Handling (job execution duration)
As of Obsidian 1.5.
Jobs can have an expected runtime length set, which causes a Warning level event to be triggered if a job falls outside of the expected range. This is used primarily for notifications. Users can be alerted in the event of a long-running (or unexpectedly short) job by configuring the appropriate Notifications.
As of Obsidian 3.4.0, Interruptable Jobs also support auto interruption when execution length exceeds maximum expected run length.
Custom calendars allow you to create named sets of dates that are used to prevent job executions. When setting a schedule for a job, you can select a custom calendar so that job execution is suppressed on any date defined in the custom calendar, regardless of the cron pattern used.
Common uses would be to define corporate holidays or those of your partners. Custom calendars are optional applications to the aforementiond date bound scheduling.