Getting Started

From Obsidian Scheduler
Revision as of 19:23, 23 May 2023 by Craig (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigationJump to search

This Getting Started is for Obsidian 4.0 and newer versions. See the Previous Getting Started for prior versions.


This guide will help you choose a deployment setup that works for you, and then get it running. Before reading this page, you may wish to review Obsidian's Deployment Models to understand the different parts of Obsidian or to find out what deployment model works for you.

If you've are looking to start writing your own Obsidian jobs, see Implementing Jobs.


Supported Platforms

Please quickly review our supported platforms before continuing.

  • Obsidian is OS-independent, and runs on the Java Virtual Machine. It works in a variety of environments, including Linux, OSX and Windows, and can be run inside virtual machines.
  • Obsidian 5.x.x runs on Java 11 or above (Obsidian 4.x.x runs on Java 1.7 or above)
  • Obsidian's administration web application is a servlet application tested on Tomcat 9.x and Jetty 10.x (Obsidian 4.x.x has been tested against Tomcat 7, 8 & 8.5 and Jetty 9.x), but is likely to work in any compliant servlet container version 2.4 or higher, such as WebSphere. It works on all modern browsers and is tested on recent versions of Chrome and Firefox.
  • Obsidian requires one of the following fully-supported database platforms:
    • MySQL 5.5 through 5.7 (excluding Galera)
    • MariaDB 5.5
    • Oracle 18c, 19c, 21c on Obsidian 5.x and higher
    • Oracle 10g, 11.x, 12c on versions prior to Obsidian 5.0.0
    • PostgreSQL 9, 10
    • MS SQL Server 2008-2017
    • H2 1.4.
    • Larger major versions are likely to work without issues but are not officially supported.


Note: With some exceptions, clustered scheduler instances require access to contact license servers over the Internet, or access to an internal proxy license server. See Licenses & Nodes for more information.

Hardware Requirements

We recommend the following minimum hardware for typical installations:

  • 1 GHz processor
  • 2 GB RAM
  • 1 GB hard disk space (excluding accumulated logs)

Note that actual requirements may vary depending on job volume, clustering and other factors.

Choose Your Installation Type

Obsidian consists of two main processes:

  • Scheduler - schedules and executes jobs, sends notifications, etc. Also exposes the Embedded API.
  • Admin Web Application - provides management and monitoring UI, plus the REST API and Embedded API.


These can be run together or separately in the following configurations. You will pick one or more of these options together to provide both the scheduler and admin web application functionality.

  1. Standalone Scheduler - A Java process that performs job execution. A standalone scheduler installation is generated when you run the Obsidian installer, and may be customized by adding JAR files containing your job code. It is started and stopped by scripts included with the installation. Requires a separate admin web application to be deployed.
  2. Embedded Scheduler - Provides job execution like a standalone scheduler, but instead runs as a process embedded within and started from your application. This eliminates the need to customize and deploy a separate artifact which will perform job execution. Requires a separate admin web application to be deployed.
  3. Standalone Admin Web Application - A Java servlet application providing Obsidian's administration capabilities but no scheduling services. A standalone admin application WAR file is generated when you run the Obsidian installer, and may be customized by adding JAR files containing your job code. Requires a separate standalone, embedded or combined scheduler and web application to be deployed for job execution to be performed.
  4. Combined Scheduler and Admin Web Application - A version of the admin web application which also runs a scheduler service for job execution. A combined scheduler and admin application WAR file is generated when you run the Obsidian installer, and may be customized by adding JAR files containing your job code.


Regardless of the deployment configuration, any time you run more than one scheduler process operating against the same Obsidian database, they will automatically form a cluster and share in job execution.


Choosing which installation type works best for you depends on your specific needs. The setups which are suitable for most cases are, in order:

  • Using an Embedded Scheduler (option 2), along with a Standalone Admin Web Application (option 3). This approach lets you embed Obsidian into your existing application without having to alter the Obsidian artifacts to include your job code. Instead, Obsidian's required libraries are included in your application and they are deployed together.
  • Using a Combined Scheduler and Admin Web Application (option 4) alone or in a cluster. This is suitable if you don't have another application containing job code for Obsidian to execute, or if you simply want to use Obsidian's scripting support, which doesn't require deploying JAR files.

Downloading Obsidian

Just go to our download page, and grab the latest Obsidian's installation zip file. There is no need to register or obtain a license before downloading.

Once you've downloaded the zip file, extract it to a directory of your choice.

If you are trying to upgrade to a newer version of Obsidian, see upgrade instructions.

Quick Start for Evaluation Purposes

If you wish to quickly try Obsidian without minimal setup required, follow these steps. Otherwise, skip ahead to Initial Setup.

Really Quick Method Using Embedded Database

We do not recommend this setup for production use.

  1. Ensure you have a JDK (11 or up for Obsidian 5.x.x, 1.7 or up for Obsidian 4.x.x) installed and that the JAVA_HOME environment variable is set to your JDK installation directory.
  2. Unzip the Obsidian download zip file to a directory of your choice.
  3. At the command line, in the zip extraction directory, run the installer using this command, substituting the appropriate Obsidian version number: java -jar Obsidian-Install-n-n-n.jar h2-jetty-quick-start.xml
  4. Start Obsidian using the command: ./webObsidian.sh start scheduler for Linux or webObsidian.bat start scheduler for Windows.
  5. Go to http://localhost:8080 in your browser and check out your fully functional Obsidian web application and scheduler! You can log in to the admin web application with the default user admin and password changeme.
  6. When you're done, stop Obsidian using the command: ./webObsidian.sh stop or webObsidian.bat stop for Windows.


Note: If you restart the Quick Start installation and see "lock wait timeout" or similar errors in the log screen, you may have to delete your embedded H2 database lock file. By default, the file name is obsidian.lock.db and is located in the user home directory.

Note: If you see a java.net.BindException: Permission denied error on startup, you will need to use an alternate port. See Troubleshooting for details on how to change this.

Quick Method Using Existing Database

If you wish to get Obsidian running quickly using a database platform you already have running, follow these steps:

  1. Ensure you have a JDK (11 or up for Obsidian 5.x.x, 1.7 or up for Obsidian 4.x.x) installed and that the JAVA_HOME environment variable is set to your JDK installation directory.
  2. Unzip the Obsidian download zip file to a directory of your choice.
  3. At the command line, in the zip extraction directory, run the installer using this command, substituting the appropriate Obsidian version number: java -jar Obsidian-Install-n-n-n.jar
  4. Start Obsidian using the command: ./webObsidian.sh start scheduler for Linux or webObsidian.bat start scheduler for Windows.
  5. Go to http://localhost:8080 in your browser and check out your fully functional Obsidian web application and scheduler! You can log in to the admin web application with the default user admin and password changeme.
  6. When you're done, stop Obsidian using the command: ./webObsidian.sh stop or webObsidian.bat stop for Windows.

Initial Setup

This section covers the setup required after you've selected your installation type.

Database

Obsidian requires a database which must be created before running Obsidian. Multiple database platforms are supported. Obsidian can share a database/schema with another application.

Note that the database must exist before deployment, but by default Obsidian will automatically create all required tables.

Later, you will specify database connection parameters within Obsidian's installer to tell it how to connect. If you need to configure advanced settings or change settings directly after you've run run the installer, see Advanced Configuration.

Note: Obsidian needs to create the tables in the target database. If the schema is shared with your application’s tables, please ensure there are no name conflicts. If there are conflicts, separate schemas/databases can be used, or a table prefix can be specified as shown in Advanced Configuration. Obsidian Tables lists the tables Obsidian will created upon first deployment.


Manual Schema Creation

When you start Obsidian, it will automatically detect and create missing tables in your target database. However, if you wish to run Obsidian with a database user that does not have create privileges, or simply wish to prepare your database structure ahead of time, you can use the database scripts provided with your download zip file under the "db_scripts" directory.

First create your database, and locate the "complete.sql" file under the directory for the database you are using under the "db_scripts" directory. Statements within this file contain the token "$px$" to enable table prefixes. Before you run the SQL file, replace all instances of the token "$px$" (without quotes) with a table prefix 6 characters or shorter (letters and underscores supported), or with blank.

Then, in your SQL client of choice, execute the script and all the required Obsidian tables will be created.


Alternate Oracle Schemas

After allowing Obsidian to create all the required tables and initial data by deploying it a single time, Obsidian can be run with an alternate Oracle user by specifying the target schema on Oracle databases. See Advanced Configuration for the property to set.

If you use a separate user from the schema owner, you must grant the user certain privileges for Obsidian to function correctly.


MS SQL Server Snapshot Isolation

For maximum compatibility and to avoid deadlocks, MS SQL Server should be configured to use read committed snapshot isolation.

This can be enabled on your database by running the following commands: 

ALTER DATABASE MyDatabase SET ALLOW_SNAPSHOT_ISOLATION ON
ALTER DATABASE MyDatabase SET READ_COMMITTED_SNAPSHOT ON

Authentication

By default, Obsidian manages its own set users and logins to restrict access to the admin web application and REST API. A default “admin” user is created when the scheduler is first deployed and no additional setup is required.

Alternatively, to use LDAP authentication, select the LDAP option in the installer and enter your LDAP details.

Finally, you may implement your own custom authenticator Java class, which you also enter when running the installer.

Running the Installer

Now that you've chosen your installation type and performed initial setup of your database and authentication, you can run Obsidian's installer to configure and build a set of artifacts that you can either use to embed Obsidian, or deploy directly.

To run the installer, please follow our detailed installation guide.

Installer-Created Artifacts

Once you've run the installer, your installation directory will look something like the following, depending on which artifacts you chose to generate. Key files and directories are shown below. 

├───db_scripts
└───Obsidian-5.n.n
    │   com.carfey.properties.copy 
    │   obsidian.war
    │   obsidian-builtin-job-src.jar
    │   obsidianForkedJob.bat
    │   obsidianForkedJob.sh
    │   obsidian-javadoc.zip
    │   standaloneObsidian.bat
    │   standaloneObsidian.sh
    │   standaloneObsidianAdmin.war
    │   webObsidian.bat
    │   webObsidian.sh
    │
    ├───jetty
    ├───license
    ├───logs
    └───standalone
        (various JAR files...)
        obsidian-props.jar
        obsidian.jar

The installer creates a top-level directory named Obsidian-n.n.n containing generated artifacts. The key files and directories are:

  • com.carfey.properties.copy - Copy of properties file generated by the installer.
  • obsidian.war - Combined scheduler and admin web application WAR. This can be deployed directly to a servlet container. This artifact can be customized by adding your job code and dependencies as JARS to the /WEB-INF/lib directory within the archive.
  • standaloneObsidian.bat and standaloneObsidian.sh - Windows and Linux scripts to start the standalone scheduler process with no web application. The standalone scheduler can be customized by adding your job code and dependencies as JARS to the standalone directory.
  • webObsidian.bat and webObsidian.sh - .bat - Windows and Linux scripts to start either the standalone web application or combined scheduler and admin web application within the bundled Jetty server.
  • standaloneObsidianAdmin.war - Standalone admin web application WAR (no scheduler). This can be deployed directly to a servlet container.
  • standalone - Directory containing JARs that Obsidian requires to run the standalone scheduler process. The standalone scheduler can be customized by adding your job code and dependencies as JARS to this directory.
    • obsidian-props.jar - JAR file containing the Obsidian properties file com.carfey.properties. If you wish to make configuration changes for the standalone scheduler, the properties file within this JAR will need to be updated.

Embedding Obsidian & Customizing Artifacts

The Obsidian scheduler process needs Java classpath access to your jobs so it can detect valid jobs and execute them. This requires a bit of customization before you are ready to run custom Java jobs. If you still haven't created your custom jobs, consult Implementing Jobs before proceeding. If you only intend to use Scripting Jobs, customization will not be required and you can skip to Deployment.

If you want to embed Obsidian into your application or need to deploy jobs and their dependent libraries to the generated Obsidian artifacts, follow the steps in the appropriate sections below.

Note that only Obsidian scheduler processes need to be customized to include your job code. The admin web application itself does not require classpath access to your jobs.

Embedding Obsidian - Importing Libraries and Properties File

To embed Obsidian, you will need to update your application build to bring in the JAR files it requires, along with the Obsidian properties file. Simply follow these steps:

  1. Import all required Obsidian JAR files into your project and reference them in your build files.
    • These can be obtained from the standalone directory of your installation. Make sure you exclude obsidian-props.jar
    • Ensure your Maven, Gradle or Ant files are updated to include all required libraries in your build, along with your IDE.
  2. Make a copy of com.carfey.properties.copy from your installation with the .copy suffix removed, and add it to your project's classpath.
    • This is typically just a matter of adding it to your project's resources directory.
    • From now on, you can update this file to change the various configuration options such as database connection details.
    • You may wish to leave this file out of source control and let developers configure their own installations. However, any running Obsidian instance will need access to a valid properties file.

Maven users: Note that we do not publish Maven artifacts for Obsidian, so you will not be able to include them by referencing a public repository.

Adding Custom Jobs to Standalone Scheduler

In order for the standalone scheduler to detect and execute custom jobs written in Java, you will need to ensure Obsidian's classpath contains the compiled job code, along with its dependencies.

To do so, you need to perform just one step before starting the standalone scheduler:

  1. Copy JAR files containing custom Obsidian jobs and all runtime dependencies to the standalone directory in your installation.
    • These JARs should be alongside obsidian.jar.
    • All JAR files in this directory will be automatically added to the classpath when run via the standaloneObsidian.sh or standaloneObsidian.bat scripts.

Adding Custom Jobs to Combined Scheduler and Admin Web Application

In order for the combined scheduler and admin web application to detect and execute custom jobs written in Java, you will need to ensure Obsidian's classpath contains the compiled job code, along with its dependencies.

To do so, you need to perform just one step before deploying the Obsidian WAR file:

  1. Copy JAR files containing custom Obsidian jobs and all runtime dependencies to the WEB-INF/lib directory within the WAR archive.
    • These JARs should be alongside obsidian.jar.
    • WAR files use the ZIP format and can be extracted and recompressed with normal ZIP tools or the JDK jar utility.

Updating the Properties File in Admin Web Applications

Note that updating the WAR artifact is not necessary to perform properties file changes when using an external properties file as described here.


If you need to change configuration properties for a standalone admin web application or combined scheduler and admin web application, follow these steps:

  1. Obtain the WAR artifact from your installation that requires changes. This is either obsidian.war or standaloneObsidianAdmin.war.
  2. Extract the WAR file to a clean directory.
  3. Edit the com.carfey.properties file within the subdirectory WEB-INF/classes and save your changes.
  4. Rebundled the WAR artifact using a zip utility or the JDK jar utility.
    • Some zip utilities such as 7Zip will allow you to edit files within an archive directly without the previous steps. It is generally safe to do so.

Deployment

You are now ready to deploy your scheduler and/or admin web application.

Embedded Scheduler

When Obsidian is embedded, there is no separate deployment process. However, you will need to start and stop Obsidian with your application. If you are using Spring, we recommend you use our Spring Integration instead, which will automatically start and stop the scheduler for you.


During your application startup, once it is fully initialized (or as close as possible), start Obsidian, and save a reference to the returned result so you can shut it down later: 

// This will start the scheduler on the first call to get().

com.carfey.ops.job.SchedulerStarter starter = com.carfey.ops.job.SchedulerStarter.get(com.carfey.ops.job.SchedulerStarter.SchedulerMode.EMBEDDED);

// ...

// Later, we need to gracefully shut down the scheduler 
starter.shutDown();

This will run Obsidian's scheduler process, but will not include any of the web application or the REST API. This functionality is provided by the admin web application.

Standalone Admin Web Application

To deploy the standalone admin web application, simply deploy the standaloneObsidianAdmin.war file your servlet container of choice (e.g. Tomcat) after configuring it appropriately. You may rename the WAR file to have it deployed under a different context path, if desired (e.g. "ROOT.war" or "obsidian.war"). Consult your servlet documentation to find out how to deploy the application and start the servlet container.

Note: If you want to use the included Jetty server as your servlet container, you can use the scripts included in your installation instead of deploying the WAR. This will make the application accessible at http://localhost:8080.

  • To start
    • Linux: ./webObsidian.sh start adminOnly
    • Windows: webObsidian.bat start adminOnly
  • To stop
    • Linux: ./webObsidian.sh stop
    • Windows: webObsidian.bat stop


Security Note: By default, the admin web application allows non-secure connections. If you wish to force secure connections through HTTPS, you can edit the web.xml in your war and uncomment the <security-constraint> element in the file. This will force all requests to redirect to an encrypted connection. For details on setting up SSL on your servlet container, refer to its documentation.

Standalone Scheduler

To deploy and run the standalone scheduler, you may use the provided scripts in your installation directory:

  • Linux: ./standaloneObsidian.sh start
  • Windows: standaloneObsidian.bat start


You should always stop Obsidian gracefully when possible by using:

  • Linux: ./standaloneObsidian.sh stop
  • Windows: standaloneObsidian.bat stop


Both the start and stop commands may be supplied an additional argument to override the listenerPort which defaults to 10451.

Rather than use one of the provided scripts, you can invoke the equivalent Java command: 

java com.carfey.ops.job.SchedulerStarter start <listenerPort>
java com.carfey.ops.job.SchedulerStarter stop <listenerPort>

Combined Scheduler and Admin Web Application

To deploy the combined scheduler and admin web application, simply deploy the obsidian.war file your servlet container of choice (e.g. Tomcat) after configuring it appropriately. You may rename the WAR file to have it deployed under a different context path, if desired (e.g. "ROOT.war"). Consult your servlet documentation to find out how to deploy the application and start the servlet container.

Note: If you want to use the included Jetty server as your servlet container, you can use the scripts included in your installation instead of deploying the WAR. This will make the web application accessible at the URL http://localhost:8080.

  • To start
    • Linux: ./webObsidian.sh start scheduler
    • Windows: webObsidian.bat start scheduler
  • To stop
    • Linux: ./webObsidian.sh stop
    • Windows: webObsidian.bat stop


Security Note: By default, the admin web application allows non-secure connections. If you wish to force secure connections through HTTPS, you can edit the web.xml in your war and uncomment the <security-constraint> element in the file. This will force all requests to redirect to an encrypted connection. For details on setting up SSL on your servlet container, refer to its documentation.

Embedding the Obsidian Web Application in Another Web Application

We do not generally recommend attempting to merge the Obsidian admin web application with another servlet web application, but it can be done by following these steps:

To embed the full Obsidian web application in an existing web application, you will need to extract the necessary sections from the WEB-INF/web.xml file in the obsidian.war file and merge them into your application's web.xml file. This includes all "listener", "servlet", "servlet-mapping", "jsp-config", "filter", "filter-mapping" elements, and optionally "welcome-file-list" and "error-page" elements. Ensure that all servlet paths are maintained. If you are using a different servlet specification version, you may need to update the mappings to the appropriate format.

Note the use of our com.carfey.ops.servlet.StartupShutdownListener which takes care of starting up the scheduler instance and using the web container's default shutdown mechanism to ensure graceful shutdown. This listener will start Obsidian's scheduler process and initialize the Obsidian web application.

If you wish to embed Obsidian's web application without the scheduler running, include the following parameter element in your web.xml: 

<context-param>
   <param-name>schedulerEnabled</param-name>
   <param-value>false</param-value>
</context-param>

Additional Deployment Details

Classpath Notes

To run a scheduler with your custom code and jobs, you need to ensure the scheduler process classpath includes your code packaged into JAR files, as described above.

If a compiled job is updated, you will have to restart your application or the servlet container after deploying updated jars, unless you are using Job Forking.

Setting Host Names

Obsidian instances will automatically assign themselves host names if no host name is explicitly set, but you may wish to give them explicit names to make scheduling and monitoring simpler.

If you wish to assign explicit names, simply set the Java system property schedulerDesignation to the host name of your choice. For example, if starting an instance using the standalone scheduler using java directly, simply add the value -DschedulerDesignation=myHostName to the end of the command.

You may also use a properties file setting for the host name as detailed in Advanced Configuration.

Disabling Automatic Database Updates

Obsidian automatically applies schema updates and data upgrades to its database on startup. In some cases, it may be desirable to disable this once the database has been fully initialized and upgrades to new versions are not going to be deployed. For example, you may wish to run Obsidian with a user who does not have privileges to modify the database schema.

To do so, you can either set a Java system property, or if you are using the admin web application WAR, you can add a setting to your WEB-INF/web.xml within the archive.

As a system property: 

-DstartupRunnerClass=com.carfey.ops.run.NullRunner

In WEB-INF/web.xml, after any listener elements: 


<context-param>
    <param-name>startupRunnerClass</param-name>
    <param-value>com.carfey.ops.run.NullRunner</param-value>
</context-param>

Disabling DDL Updates

It may also be helpful to apply schema updates (DDL) outside Obsidian through a privileged account, but allow Obsidian upgrades to make the necessary data initialization/modifications. This can be done using a skipDDL configuration property documented under Advanced Configuration.

Disabling Job Scheduling in the Web Application

If you have combined scheduler and admin web application WAR already built, you can easily tweak it to disable the scheduler process.

Simply add the following to your WEB-INF/web.xml within the WAR archive after any listener elements: 


<context-param>
    <param-name>schedulerEnabled</param-name>
    <param-value>false</param-value>
</context-param>

You’re Good to Go!

You're now ready to log into the web application and start scheduling jobs! The URL you use to access Obsidian will depend on how it's been deployed, but will typically be something like http://localhost/obsidian, or http://localhost/standadminObsidianAdmin.

Default User: Each Obsidian installation using native authentication starts with a single default user named admin with password changeme. You should change this after your first log in. See User Management for how to change a user's password.


At this point we suggest you play around with the admin web application. Many screens offer inline help. Otherwise, you can refer to our Admin Web Application Guide.


To get started writing jobs that you can run in Obsidian, see Implementing Jobs.


See the User Guide if you have questions or wish to explore what Obsidian offers.

Retrieved from "https://wiki.obsidianscheduler.com/doc/index.php?title=Getting_Started&oldid=3726"