Recovery & Failover
Obsidian ensures continuity in the execution of your scheduled jobs by providing a variety of built-in recovery mechanisms.
- Multiple concurrent hosts support
- Instance outage recovery
- Configurable recovery options by job
- Resubmission of abnormally terminating jobs
Multiple Concurrent Hosts
Out-of-the-box, Obsidian can be run with as many hosts for which you are licensed. As long as at least one host is running and the jobs are not constrained to specific hosts, any server failures will not prevent jobs from being run on schedule. Any jobs running when a server fails will be recovered as defined in Instance Outage Recovery. No special configuration is required to run multiple hosts. Just start up a node and it joins the available service pool.
Instance Outage Recovery
When a given server fails, any jobs that were in the midst of running cannot be completed normally. Any other running hosts will discern that the jobs have not had any activity and they will be marked as Died. This will allow other hosts to run the job for any subsequently scheduled times. And since no special configuration is required to run multiple hosts, when the issue with the server failure is resolved, simply start it up again and it joins the pool. A given instance will reuse its licence as long as it has not been claimed by another host.
Configurable Job Recovery
Obsidian supports recovery modes by job type. These are used when no instance has been executing scheduled jobs. When an instance finally starts up, all missed scheduled runtimes will be evaluated against the configured recovery mode by job.
- All recovery means that all missed instances will be run. This feature logically requires the job to have access to the scheduled run time. This can be retrieved from the Context. See Implementing Jobs for details.
- None recovery means no recovery takes place. Missed jobs will be marked as such.
- Last recovery means only the last missed job will be run.
- Conflicted recovery means that a conflicted job will be run regardless of the elapsed time since it was originally scheduled once higher priority conflicts clear. See Conflicts for more details.
Resubmission of Abnormally Terminating Jobs
Obsidian allows you attempt re-execution of any job that did not complete normally for any failure reason whatsoever. You can correct any conditions that caused the failure and resubmit the job for re-execution. An administrative user can even configure which abnormal termination states allow resubmission.
