Recovery & Failover
Obsidian ensures continuity in the execution of your scheduled jobs by providing a variety of built-in recovery and failover mechanisms.
- Multiple concurrent hosts support
- Instance outage recovery
- Configurable recovery options by job
- Resubmission of abnormally terminating jobs
- Standby Nodes
Multiple Concurrent Hosts
Out-of-the-box, Obsidian can be run with as many hosts for which you are licensed. As long as at least one host is running and the jobs are not constrained to specific hosts, any server failures will not prevent jobs from being run on schedule. Any jobs running when a server fails will be recovered as defined in Instance Outage Recovery. No special configuration is required to run multiple hosts. Just start up a node and it joins the available service pool.
Instance Outage Recovery
When a given server fails, any jobs that were in the midst of running cannot be completed normally. Any other running hosts will discern that the jobs have not had any activity and they will be marked as Died. This will allow other hosts to run the job for any subsequently scheduled times. And since no special configuration is required to run multiple hosts, when the issue with the server failure is resolved, simply start it up again and it joins the pool. A given instance will reuse its licence as long as it has not been claimed by another host.
Configurable Job Recovery
Obsidian supports recovery modes by job type. These are used when no instance has been executing scheduled jobs. When an instance finally starts up, all missed scheduled runtimes will be evaluated against the configured recovery mode by job.
- All recovery means that all missed instances will be run. This feature logically requires the job to have access to the scheduled run time. This can be retrieved from the Context. See Implementing Jobs for details.
- None recovery means no recovery takes place. Missed jobs will be marked as such.
- Last recovery means only the last missed job will be run.
- Conflicted recovery means that a conflicted job will be run regardless of the elapsed time since it was originally scheduled once higher priority conflicts clear. See Conflicts for more details.
Resubmission of Abnormally Terminating Jobs
Obsidian allows you attempt re-execution of any job that did not complete normally for any failure reason whatsoever. You can correct any conditions that caused the failure and resubmit the job for re-execution. An administrative user can even configure which abnormal termination states allow resubmission.
Standby Nodes
Obsidian also supports standby nodes. If you choose to limit your licensing to a single node, you still can achieve limited failover by having a standby node running. The standby node will not be able to successfully obtain a license and sill therefore nit run any jobs. If the primary node fails, the standby node can claim the licence once the lease on the licence has expired. Carfey Software encourages use of multiple licensed hosts in your production environment if at all possible as you can suffer an outage while the standby node waits for the lease to expire.
Standby node(s) are a decent strategy to ensure decent load balancing and distribution even after instance failures. As above, once the licence lease expires, your standby nodes will join the serving pool and share in the load of executing your jobs.
