Event Notifications

Obsidian includes powerful notification capabilities to help your organization stay on top of unexpected events that can interfere with your business.

Obsidian's built-in notifications are provided via email. In addition, Obsidian 4.5 introduced custom Event Hooks which allow you to implement custom actions such as IM or Slack messaging when notification events are triggered. SMS messages can be sent with most providers through SMS gateway email addresses. Here is a complete listing of SMS gateways.

Email content can be customized for various events types, and by specific jobs.

If you are looking to configure SMTP in Obsidian to get notifications working, see Advanced Configuration.

Obsidian's web application allows you to configure email notifications and view notification history in the Admin Notifications screen.

How notifications are sent out depends on these factors:

• Subscriber settings
  • Active state
  • Email address
• Event category subscriptions
  • Minimum Notification Severity (Info, Warning or Error)
  • Active state
  • Applicable Target Entities (in the case of job categories)
• Job execution subscriptions
  • Trigger states(Completed, Conditional, Failed, Died, Recovery)
    • Job Results Conditions when using Conditional trigger
  • Active state
  • Applicable Jobs

Customizable Notifications

Obsidian allows you to customize your notification email content. You can define separate templates for different event types, and also define a template that applies to specific jobs. This can be useful for emailing the results of certain jobs to interested parties, while sending less detailed emails for other jobs. Full details are documented in Email Templates.

Category-Based Subscriptions

When setting up a notification subscriber in Obsidian, you are able to choose the event categories that need to have notifications sent. This allows you to target specific individuals or groups with notifications instead of creating a lot of email noise, which could result in users ignoring notifications.

The following categories are supported. Some categories are tiered, so that subscribing to a parent category automatically subscribes you to the children. You can also directly subscribe to the child categories.

• Job - Parent category. Notifies you of all child events. The following are direct Job events: conflict state changes, expected run length events, interrupts and resubmissions. In addition, notifies you at the Info level when jobs complete successfully, or at the Error level when they fail.
  • Job Chain - Notifies you when jobs are chained, or fail to chain.
  • Job Config - Notifies you when job configuration is updated including chain and conflict configuration. Useful for tracking changes.
• Job Recovery - Notifies you when job recovery occurs (e.g. when there are outages) and other runtime issues (terminal states Conflict Missed, Abandoned and Died) .
• Licence - Notifies you of license verification events.
• Queue - Notifies you of queueing issues, which may indicate database or other platform problems. Also used to track startup/shutdown/pause/resume events for scheduler nodes.
• System Parameter - Notifies you of system parameter changes. Useful for tracking changes.

Minimum Notification Severity

The following notification severities are supported for the above categories. Note that some categories do not support all severities.

• Info - Include all notifications
• Warning - Don't include informational notifications (e.g. job completion, configuration updates, etc.)
• Error - Only include error conditions (e.g. job failures, queueing issues, etc.)

Job-Specific Subscriptions

For the Job category and its children, you can choose to subscribe to an individual job, rather than all jobs. In addition, the notification severity can be set separately for each job, so you may choose to be notified of job failures for all jobs, but only be notified of successful completion of a certain set of critical jobs.

This can be used to set up fine-grained notification subscriptions for various stakeholders within your organization.

Job Execution Subscriptions

When setting up a notification subscriber in Obsidian, you have precise control over which job executions result in notifications. Not only can you target specific jobs or execution states, but you can also trigger notifications based on specific job output conditions. You can also be alerted when a job succeeds after finishing with a failure state on a previous run.

Trigger States & Conditions

Trigger states control when job execution results in a notification - you can be notified only when a job fails, receive notifications when it recovers from failure, or be alerted upon successful execution. This flexibility allows you to fine-tune your notifications based on what you actually need to know. The Conditional trigger state unlocks even greater power, allowing you to only receive notifications when a job saves output that matches specified conditions. For example, you could have a backup job which stores the remaining disk space as job output, and receive notifications when that value reaches a critical threshold.

The following trigger states are supported:

• Completed - corresponds to the same job history status (i.e. when a job completes without an error).
• Died - corresponds to the same job history status (i.e. when a job does not complete normally, potentially triggered by a forceful shutdown).
• Failed - corresponds to the same job history status (i.e. when a job throws an exception).
• Recovery - triggered when a job does not complete normally.
• Conditional - allows you to specify one or more output conditions that must be met to trigger a notification. These conditions match the source job's Job Results. All conditions must be met to trigger the notification, and the job must complete successfully.

Job-Specific Subscriptions

For each job execution subscription, you can select specific jobs it applies to, or apply it to all jobs. In addition, you can set up as many subscriptions as you require, each with their own trigger states and job result conditions.

This provides fine-grained control over which specific job execution conditions result in notifications, preventing unwanted email noise.

Custom Event Hooks

Obsidian 4.5 introduced custom Event Hooks which allow you to implement custom actions such as IM or Slack messaging when notification events are triggered. These provide low-level access to respond to events such as job failures.