Job Forking

As of Obsidian 3.0.0, Obsidian can run each job in its own JVM. This allows for a number of possibilities, including support for dynamic changes to your deployed jobs (i.e. hot-swapping of JARs). In theory, you could even specialize classpaths on a per job basis by customizing the provided execution scripts.

JVM job forking is disabled by default. Standalone deployments support forking out-of-the-box, but it is not active until you enable it as defined in Configuring Job Forking.

Note: In order to support hot-swapping of JARs that contain compiled jobs, Obsidian expects you to ensure JAR consistency between nodes. If you plan on deploying new jobs or changing the configurability (e.g. supported parameter changes), you will want to deploy these to all Obsidian nodes that have job forking enabled and either restrict these from executing on nodes with forking disabled (see Fixed Hosts) or deploy the changes to these nodes with appropriate restarts of Obsidian.

How Does Job Forking Work?

When job forking is enabled, Obsidian will run a shell script (or batch file on Windows) for each job execution to be forked. This script in turn starts a new JVM which invokes an Obsidian class with the appropriate arguments to start the single-job execution. Forking can be customized easily by using alternate scripts from the provided defaults. To support new jobs or configurability changes to existing jobs, Classpath Scanning also runs as a forked process using the same scripts.

Forked Jobs and the Obsidian Database

Each occurrence of a Forked Job will require a few connections to the database. Make sure you plan accordingly in your connection allocations on the database and sizing for the JNDI datasource, if applicable.

Interrupt Support

As of Obsidian 3.6.0, forked jobs can be interrupted. See Configuring Job Forking below.

Configuring Job Forking

Enabling Job Forking - Cluster

To use forking, first you must enable a setting (jvmJobForkingEnabled) that will apply across the cluster. This is a scheduler setting found in the Job Spawner category. You may enable it through the admin web application, the Embedded API, REST API, or Obsidian's initialization & restore support.

If you wish to enable interruption of forked jobs that implement the necessary Interruptable interface, you must also enable the setting jvmForkedJobsCanBeInterrupted also found in the Job Spawner category, introduced as of Obsidian 3.6.0.

Enabling Job Forking - Node

Once you've done that, you'll need to enable it on each desired node. Standalone scheduler instances are configured to support forking by default. For other nodes types, you will need to set the appropriate property value.

com.carfey.obsidian.jvmJobForkingEnabledOnThisNode=true

In a standalone scheduler, job forking will be then be functional on next restart. In other deployments, or if you wish to make additional customizations, additional changes will be required.

Script Location

Job forking uses a script for each job execution to start up a fresh JVM.

You may wish to specify the forking script location for other deployments, such as embedded schedulers and combined scheduler and admin web application instances, or for other reasons.

To override the script location, set the appropriate property value in your properties file:

com.carfey.obsidian.forkedJobScriptLocation=/Obsidian-3.0.0

Obsidian is bundled with obsidianForkedJob.bat and obsidianForkedJob.sh. These are the required script names. If you wish to customize these scripts or use your own, you are free to do so, but the script names cannot be changed. Review the notes on the forked job runner to assist your customization efforts.

Classpath Override

You may wish to override the default classpath that is used by the forked execution instance. Or for deployment types other than standalone schedulers, you can use this property to specify the classpath.

com.carfey.obsidian.forkedJobscriptClasspathOverride=/home/user/workspace/obsidian/bin:/home/user/workspace/obsidian/lib/activation-1.1.jar\
:/home/user/workspace/obsidian/lib/mail-1.4.jar:/home/user/workspace/obsidian/lib/dom4j-1.6.1.jar:/home/user/workspace/obsidian/lib/obsidian.jar\
:/home/user/workspace/obsidian/lib/log4j-1.2.9.jar:/home/user/workspace/obsidian/lib/gson-2.2.2.jar:/home/user/workspace/obsidian/lib/bsh-2.0b4.jar\
:/home/user/workspace/obsidian/lib/groovy-all-2.1.8.jar:/home/user/workspace/obsidian/lib/jython-standalone-2.5.3.jar\
:/home/user/workspace/obsidian/lib/mariadb-java-client-1.1.5.jar

The sample demonstrates usage in an embedded Obsidian instance running inside an Eclipse project. The default classpath used in the obsidianForkedJob.(sh|bat) script is built automatically assuming a standalone scheduler deployment. As such, it uses the jars in the standalone/ directory for building the classpath. If you require a custom classpath on a per job basis, modification of the forking script(s) will be required.

Database Configuration Parameters

If your environment does not have database configuration entries in the properties/yaml file accessible to the forked job (e.g. config is done via System properties and/or programmatically), you will want to enable the following option for forked jobs. When enabled, database configuration will be passed into the scripts instead of relying on locating a properties file with the same information.

com.carfey.obsidian.jvmJobForkingIncludeDbSysParms=true

Script Customization

If you aren't using a standalone scheduler, or you simply need to customize Job Forking, review the following section for detailed documentation on the scripts used and how they can be customized.

Script Arguments

Obsidian calls the script with the following arguments.

• stack_file Always the first argument. Used by the Scheduler Node to be aware of execution exceptions,
• -argumentsFile Required if you need database parameters or additional execution data such as job nickname, job class or running host.
• -classpathScan Flag indicating the run is a classpath scan request.
• -jobHistoryId Required. Job execution instance identifier.
• -classpathOverride Optionally provided if the com.carfey.obsidian.forkedJobscriptClasspathOverride property is present.

The following arguments can be found in the arguments file that is created for each invocation. Job parameters are not included in classpath scan requests. Database parameters are only included if the com.carfey.obsidian.jvmJobForkingIncludeDbSysParms property is true. All these parameters are parsed into the provided scripts as arguments. See the notes inside each script for more details.

• jobNickname
• jobClass
• runningHost
• dbUrl
• dbJndi
• dbUser
• dbPass
• dbMaxConn
• dbConnTimeout
• dbUnusedConnTimeout
• dbTablePrefix
• dbSchema

Forked Obsidian Job Runner

Should you wish to write your own obsidianForkedJob.sh/obsidianForkedJob.bat or wish to modify the one(s) provided, you'll need to know how to invoke the forked job runtime. The runtime is accessed by invoking the Obsidian class com.carfey.ops.job.ForkedJob. Its main method must be called with 2 arguments.

1. job_history_id or "scanClasspath"
2. stack_file

A number of the other arguments are available to be passed as JVM system properties for database connectivity as noted below:

-Dcom.carfey.obsidian.db.url=$dbUrl -Dcom.carfey.obsidian.db.jndiType=$dbJndi -Dcom.carfey.obsidian.db.userId=$dbUser \
-Dcom.carfey.obsidian.db.password=$dbPass -Dcom.carfey.obsidian.db.maxConnections=$dbMaxConn -Dcom.carfey.obsidian.db.connectionTimeout=$dbConnTimeout \
-Dcom.carfey.obsidian.db.unusedPoolConnectionTimeoutSeconds=$dbUnusedConnTimeout -Dcom.carfey.obsidian.db.tablePrefix=$dbTablePrefix -Dcom.carfey.obsidian.db.schema=$dbSchema

Per Job Customizations - Classpaths and More

Three other parameters are provided as context for additional customization efforts, such as having alternate classpaths for particular jobs, classpath library ordering, controlling which Java executable is used and so on. Please contact the Obsidian support team to discuss your needs or with any questions you have.

The scripts parse jobNickname, jobClass and runningHost from the arguments file. These can be utilized within the script to determine what specialization to apply. These could be used in conjunction with the default classpath, the classpath_override or on its own.

For example, if you wanted to use a custom classpath if the job_class is of type "com.example.OldLibrariesJob" and another one when the job nickname is "New Open Document Format Job", you might modify the shell script like this:

if [ "New Open Document Format Job" == "$jobNickname"] ; then
	cp=/Obsidian-3.0.0/newlibs/odf-4.0-beta.jar:/Obsidian-3.0.0/libs/... rest of classpath here
elif [ "com.example.OldLibrariesJob" == "$jobClass"] ; then
	cp=/Obsidian-3.0.0/oldlibs/my-old-lib-1.0a.jar:/Obsidian-3.0.0/libs/... rest of classpath here
elif [ "" != "$classpathOverride" ]  ; then
	cp=$classpathOverride
elif [ "$(expr substr $(uname -s) 1 6)" == "CYGWIN" ] ; then
	cp=$(find standalone -name "*.jar" -exec printf "{};" ';')
else
	cp=$(find standalone -name "*.jar" -exec printf :{} ';')
fi

Notes about Spring and JNDI

Since Job Forking is starting a specialized short-lived Obsidian instance designed to either run a classpath scan or execute a job via a Java class' main method, database/email configuration via JNDI and Spring container support are not available out-of-the-box as these are made possible through your deployment and configuration options that Obsidian simply expects to be available. It's still possible to make use of these, it will just require an intermediary step between the forking scripts and the com.carfey.ops.job.ForkedJob class.

How to Support Spring and JNDI

You need some points of customization to make container resources available.

Possible customizations needed may include:

• A new entry point class that initializes a Spring Container, calls the Obsidian com.carfey.ops.job.ForkedJob class and then shuts down the container.
• A new or existing entry point mechanism that is able to call the Obsidian com.carfey.ops.job.ForkedJob class in your server container.
• Forking script modifications to utilize the above.

For example, let's say you have integrated Spring and Obsidian and want Spring support while using Job Forking.

1. Write a new class that initializes your Spring container following the same Obsidian integration as you have done for your primary application. This class must take in the Script Arguments.
2. In code, after initialization is complete, call com.carfey.ops.job.ForkedJob with the script arguments passed in.
3. When the main method returns, shutdown the Spring container in code.
4. Modify the forking script(s) to call the new class.