Troubleshooting

The following are steps to troubleshooting common problems users have had in Obsidian. If you don't see your problem listed, contact us or use our Free Live Chat Support link on the left.

My job isn't running when expected. Why not?

This may be a case where your schedule isn't correct for the execution times you want, or the job may be failing.

1. First, you should confirm that your job isn't actually running. You can check for the Job Activity screen to see if there are any scheduled or completed jobs. You may see that a job failed; to see details on it, simply expand the row to see an exception stack trace and the option to resubmit the job. If your job is failing, you may wish to set up notifications so that you know of future problems quickly.
2. If you don't see any jobs listed there, you can confirm what execution times your schedule will result in. To see what times your job will execute, check the Run Time Preview screen.
3. Next, you can check for any general server health issues. To do so, first you can search the Logs screen for any errors or warnings that may indicate a server issue.
4. Finally, check your application logs (e.g. Tomcat stdout logs, etc.). If there are database issues, these logs may be the only place where server issues are logged.
5. If all else fails, contact us or use our Free Live Chat Support link on the left.

When I try to enter my job's class name in the UI, I get the message "Could not locate the class [MyJob]". What's the problem?

This problem is generally caused by one of these issues:

1. If you are running a combined scheduler and web application, the JAR file containing the job is not on the classpath of the servlet container (Tomcat/Jetty/etc). JARs are placed under WEB-INF/lib in the web application.
2. If you are running a standalone web application with no scheduler running, your other Obsidian instance which includes the scheduler has not had its Classpath Scanning configuration set, or has not been started with the appropriate classpath required for it to save the job metadata the web application uses to validate jobs. See Classpath for Building.
3. Obsidian has not been restarted after updating the classpath with a new JAR.
4. Job does not implement com.carfey.ops.job.SchedulableJob.
5. Job dependencies (i.e. other JARs or classes) are not available on the classpath. These must be deployed with the job.
6. User has entered the job class with leading or trailing whitespace, or the job class has been misspelled.
7. User did not enter the fully qualified class name (e.g. MyJob instead of com.company.MyJob).

Just a note, you can use the Classpath Scanning feature to automatically find your jobs so you won't need to type them in as they'll be added to the list of available jobs.

Why isn't my job showing up?

Out of the box, Obsidian will display any previously scheduled jobs and Obsidian's bundled jobs. You can type in your class name as described in Admin Jobs.

If you are using Obsidian's classpath scanning and don't see your job pre-populated, make sure you set appropriate the "packageScannerPrefix" value in the scheduler settings screen, and ensure the user account running Obsidian has read privileges to the WEB-INF directory of Obsidian's deployment directory.

In addition, some users have had issues with classpath scanning not locating jobs in JARs created by tools which don't include directory entries, such as Eclipse. Scanning relies on these directory entries, so please ensure they are included. For example, in Eclipse, if you choose to export a JAR, make sure the Add Directory Entries option is selected.

I'm trying to start Obsidian using the Quick Start method, but I'm seeing "java.net.BindException".

java.net.BindException: Permission denied
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383)
at java.net.ServerSocket.bind(ServerSocket.java:328)
at java.net.ServerSocket.<init>(ServerSocket.java:194)

This is a common error on *nix platforms, including OS X. On these platforms, ports 0-1023 are privileged. By default, only the root user will be able to bind to them. Obsidian's quick start uses port 8080 by default; the error also occurs if that port is already in use.

Changing the port as described below will fix the issue.

To resolve this, you have two options:

1. Use the "sudo" command to elevate your privileges to bind on a privileged port.
2. Change the port you use.

To change the port, edit the -Dembedded.tomcat.http.port value in webObsidian.sh or webObsidian.bat, or pass it as a JVM argument when starting Obsidian. The companion -Dembedded.tomcat.stop.port (default 9597) must also be free. If you run the Linux user systemd service, edit the script rather than the unit — the unit invokes the script. See Starting Obsidian with embedded Tomcat.

On Obsidian 5.x and 6.x, which bundled Jetty for the quick start instead of embedded Tomcat, change the port in the Jetty configuration or in the same webObsidian scripts.

I'm trying to start Obsidian on Windows using the Quick Start method, but a window appears and then disappears quickly. What's going on?

The quick start method runs Obsidian in a separate process to ensure you can gracefully stop it. Unfortunately, the separate process is failing before you can see what the problem is. To temporarily see the error and have it try to run in the same process, open the script in a text editor and find the line that launches Java: in webObsidian.bat it begins with start "Obsidian" java, and in standaloneObsidian.bat it begins with start java. Remove the start command and its window title if present (so the line begins with java), save the file and try again. You should get an informative message. Correct the underlying cause, restore the original line and run it again.

My Obsidian systemd service stops when I log out of Linux

The unit installed by linux/install-embedded-linux-integration.sh is a user service, so systemd stops it when the user's last session ends and does not start it at boot. Enable lingering for the account:

loginctl enable-linger <username>

Then confirm the service is enabled:

systemctl --user is-enabled [email protected]

If the service must run independently of any user account, create your own unit under /etc/systemd/system/ based on linux/[email protected] in your install directory. See Running Obsidian as a Linux user service.

I tried running Obsidian in Jetty, but encountered the error "There is an error in invoking javac. A full JDK (not just JRE) is required"

This applies to Obsidian 4.x through 6.x deployed to your own Jetty installation. It is caused by Jetty wanting a JDK for JSP compilation when it isn't configured.

The simplest way to resolve this is to uncomment the following line from start.ini in your Jetty installation by removing the hash sign:

# -Dorg.apache.jasper.compiler.disablejsr199=true

Obsidian 7.0.0 is tested on Tomcat 10.x and its WARs contain no JSPs, so no JSP compilation — and therefore no JDK — is required.

My jobs aren't running and I'm seeing this error: "Licence was refused: Licence is valid but no keys are available for lease. Are you running too many nodes?"

This can be caused by a few issues, but if you are unsure, contact us:

1. You are running more Obsidian instances that you have available nodes on your license key. If you are running two nodes using the year-long free clustering node, your year may have run out.
2. Obsidian is unable to refresh your license lease properly, resulting in a new lease being acquired each time it contacts our license servers. This means to our servers it looks like multiple instances are trying to run concurrently. Ensure that Obsidian has access to read and write the Java temp directory (java.io.tmpdir), since it requires this to track the license it has leased. Below is a sample of granting access to the temp directory on WebSphere Application Server 8.5:

   // Update policy file at: C:/WebSphere/AppServer/profiles/AppSrv01/properties/server.policy
   
   grant codeBase "file:${user.install.root}/installedApps/Cell01/obsidian_war.ear/-" {
   
   permission java.io.FilePermission "${java.io.tmpdir}${/}-", "read,write";
   
   };
   

3. Obsidian was not shut down properly, and was started in another location before the old license lease expired. Obsidian works by leasing a license for a specified amount of time, and refreshing it before it expires. When shut down gracefully, Obsidian will release this lease and make it available to any other node using the same key. If it is shut down non-gracefully, the lease will stay active, and only the same server will be able to refresh that same license before it expires. By default, a lease lasts an hour or two to gracefully handle sporadic connectivity.
4. You are using the wrong license key. Check the license key set in the scheduler settings screen of the web application. This is the value Obsidian is using to verify its licensing status. Note that the value in your properties file is only used to initialize the Obsidian database, so once it is set, you must update the value in the scheduler settings screen.
5. You may have runaway background Obsidian processes which are using a license that you are unaware of.

I'm getting a "Lock wait timeout exception". What's going on?

java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1055)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3491)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3423)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1936)

If you encounter an error like the one above and are using MySQL (particularly version 5.0), there is a chance you are hitting a MySQL bug. This problem seems to occur mainly on new Obsidian instances, so once corrected, there is a good chance you will never see the issue again.

Two approaches have been used with success:

1. In MySQL's my.cnf file, temporarily change (or add) the following line, and then restart mysqld:

   innodb_lock_wait_timeout = 600

   
   This appears to only be an issue when Obsidian first gets running, and it can typically be reverted after a while.
   
   
2. At a MySQL prompt, run the following command:

   optimize table semaphore;

   
   If this resolves your issue, we suggest optimizing all your tables using mysqlcheck.

I am running Obsidian on Oracle but I'm seeing the error "ORA-00942: table or view does not exist"

This is usually caused by not setting the proper schema value for the Obsidian database user.

This is easily resolved by setting the property com.carfey.obsidian.db.schema to the name of your user (or other target schema). See Advanced Configuration for more details on this property.

Example: com.carfey.obsidian.db.schema=obsidian

I'm seeing an error in the logs like "Maximum X connections already created for pool". What's going on?

Obsidian uses an internal database connection pool to avoid opening connections more than necessary, which is an expensive operation. The default maximum size for this pool is 40, which is more than sufficient for almost all installations.

If you have set a smaller value (e.g. 10), and have a very large amount of API or job activity, you can try increasing this value.

Most frequently though, this issue arises because Obsidian is configured to use a JNDI data source with its own connection pool, and there are incompatibilities between the pools' settings. As a rule, your JNDI database pool maximum size must be equal to or greater than the the maximum size of the Obsidian pool via property com.carfey.obsidian.db.maxConnections as outlined in Advanced Configuration. In addition, if there is an eviction process for idle connections, ensure than the value for the maximum idle age is 65 seconds or above to avoid warnings about unreleased connections. Alternatively, you can disable the Obsidian connection pool entirely by specifying com.carfey.obsidian.db.disablePooling=true in your com.carfey.properties file. This will cause Obsidian to never cache an opened idle connection.

Here is a good base JNDI pool configuration for Tomcat 7.

<Resource name="jdbc/obsidian" auth="Container"
type="javax.sql.DataSource"
driverClassName="oracle.jdbc.OracleDriver"
url="jdbc:oracle:thin:@localhost:1521/OBSIDIAN"
username="OBSIDIAN"
password="<password>"
maxActive="50" maxIdle="10"
maxWait="2000" timeBetweenEvictionRunsMillis="65000"
/>

I'm seeing deadlocks in the logs while running on MS SQL Server

MS SQL Server has different behaviour with the read committed isolation compared to other database platforms, which can result in periodic deadlocks in some installations.

To change this and eliminate deadlocks, enable the read committed snapshot isolation on your database.

I'm seeing an error in the logs about a time zone conflict. What's going on?

If you start multiple schedulers with different JVM time zones, Obsidian will abort startup to prevent inconsistent scheduling behaviour. The error will look something like "Cannot start scheduler in timezone [PST] since it conflicts with currently running node in timezone [EST]".

Normally, if a scheduler is shut down, it can be started again with a new time zone setting. However, if a scheduler is not gracefully shut down, the stored time zone may persist. In this case, the following sample SQL can be used while all schedulers are stopped to clear out all time zone state:

-- Add table prefix as required.
DELETE FROM OPERATIONS_PARAMETER WHERE NAME LIKE '%time%one%';