Troubleshooting

The following are steps to troubleshooting common problems users have had in Obsidian. If you don't see your problem listed, contact us or use our Free Live Chat Support link on the left.

My job isn't running when expected. Why not?

This may be a case where your schedule isn't correct for the execution times you want, or the job may be failing.

1. First, you should confirm that your job isn't actually running. You can check for the Job History screen to see if there are any scheduled or completed jobs. You may see that a job failed; to see details on it, simply expand the row to see an exception stack trace and the option to resubmit the job. If your job is failing, you may wish to set up notifications so that you know of future problems quickly.
2. If you don't see any jobs listed there, you can confirm what execution times your schedule will result in. To see what times your job will execute, check the Run Time Preview screen.
3. Next, you can check for any general server health issues. To do so, first you can search the Logs screen for any errors or warnings that may indicate a server issue.
4. Finally, check your application logs (e.g. Tomcat stdout logs, etc.). If there are database issues, these logs may be the only place where server issues are logged.
5. If all else fails, contact us or use our Free Live Chat Support link on the left.

When I try to enter my job's class name in the UI, I get the message "Could not locate the class [MyJob]". What's the problem?

This problem is generally caused by one of these issues:

1. If you are running a combined scheduler and web application, the JAR file containing the job is not on the classpath of the servlet container (Tomcat/Jetty/etc). JARs are placed under WEB-INF/lib in the web application.
2. If you are running a standalone web application with no scheduler running, your other Obsidian instance which includes the scheduler has not had its Classpath Scanning configuration set, or has not been started with the appropriate classpath required for it to save the job metadata the web application uses to validate jobs.
3. Obsidian has not been restarted after updating the classpath with a new JAR.
4. Job does not implement com.carfey.ops.job.SchedulableJob.
5. User has entered the job class with leading or trailing whitespace, or the job class has been misspelled.
6. User did not enter the fully qualified class name (e.g. MyJob instead of com.company.MyJob).

Just a note, you can use the Classpath Scanning feature to automatically find your jobs so you won't need to type them in as they'll be added to the list of available jobs.

Why isn't my job showing up?

Out of the box, Obsidian will display any previously scheduled jobs and Obsidian's bundled jobs. You can type in your class name as described in Admin Jobs.

If you are using Obsidian's classpath scanning and don't see your job pre-populated, make sure you set appropriate the "packageScannerPrefix" value in the System tab, and ensure the user account running Obsidian has read privileges to the WEB-INF directory of Obsidian's deployment directory.

I'm trying to start Obsidian using the Quick Start method, but I'm seeing "java.net.BindException".

java.net.BindException: Permission denied
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383)
at java.net.ServerSocket.bind(ServerSocket.java:328)
at java.net.ServerSocket.<init>(ServerSocket.java:194)

This is a common error on *nix platforms, including OS X. On these platforms, ports 0-1023 are privileged. By default, only the root user will be able to bind to them. Obsidian's quick start default configuration uses port 8080, but perhaps the port is already in use.

Changing the port as described below will fix the issue.

To resolve this, you have two options:

1. Use the "sudo" command to elevate your privileges to bind on a privileged port.
2. Change the port you use.

To change the port you use:

• As of Obsidian 2.4.0
  • Edit the files winstone.properties, winstoneAdminOnly.properties, webObsidian.sh, webObsidian.bat to change the port used.
• Before Obsidian 2.4.0:
  • If using the Ant script, change the deployPort setting: ant run.war -DdeployPort=8080 -Dspawn=true
  • If using the command line to start Winstone, edit winstone.properties by setting the httpPort property to the desired value (e.g. 8080).

I tried running Obsidian in Jetty 9, but encountered the error "There is an error in invoking javac. A full JDK (not just JRE) is required"

This is caused by Jetty wanting a JDK for JSP compilation when it isn't configured.

The simplest way to resolve this is to uncomment the following line from start.ini in your Jetty installation by removing the hash sign:

# -Dorg.apache.jasper.compiler.disablejsr199=true

My jobs aren't running and I'm seeing this error: "Licence was refused: Licence is valid but no keys are available for lease. Are you running too many nodes?"

This can be caused by a few issues, but if you are unsure, contact us:

1. You are running more Obsidian instances that you have available nodes on your licence key. If you are running two nodes using the year-long free clustering node, your year may have run out.
2. Obsidian is unable to refresh your licence lease properly, resulting in a new lease being acquired each time it contacts our licence servers. This means to our servers it looks like multiple instances are trying to run concurrently. Ensure that Obsidian has access to read and write the Java temp directory (java.io.tmpdir), since it requires this to track the licence it has leased. Below is a sample of granting access to the temp directory on WebSphere Application Server 8.5:

   // Update policy file at: C:/WebSphere/AppServer/profiles/AppSrv01/properties/server.policy
   
   grant codeBase "file:${user.install.root}/installedApps/Cell01/obsidian_war.ear/-" {
   
   permission java.io.FilePermission "${java.io.tmpdir}${/}-", "read,write";
   
   };
   

3. Obsidian was not shut down properly, and was started in another location before the old licence lease expired. Obsidian works by leasing a licence for a specified amount of time, and refreshing it before it expires. When shut down gracefully, Obsidian will release this lease and make it available to any other node using the same key. If it is shut down non-gracefully, the lease will stay active, and only the same server will be able to refresh that same licence before it expires. By default, a lease lasts an hour or two to gracefully handle sporadic connectivity.
4. You are using the wrong licence key. Check the licence key set in the System tab of the web application. This is the value Obsidian is using to verify its licensing status. Note that the value in your properties file is only used to initialize the Obsidian database, so once it is set, you must update the value in the System tab.
5. You may have runaway background Obsidian processes which are using a licence that you are unaware of.

I'm getting a "Lock wait timeout exception". What's going on?

java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1055)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3491)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3423)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1936)

If you encounter an error like the one above and are using MySQL (particularly version 5.0), there is a chance you are hitting a MySQL bug. This problem seems to occur mainly on new Obsidian instances, so once corrected, there is a good chance you will never see the issue again.

Two approaches have been used with success:

1. In MySQL's my.cnf file, temporarily change (or add) the following line, and then restart mysqld:

   innodb_lock_wait_timeout = 600

   
   This appears to only be an issue when Obsidian first gets running, and it can typically be reverted after a while.
   
   
2. At a MySQL prompt, run the following command:

   optimize table semaphore;

   
   If this resolves your issue, we suggest optimizing all your tables using mysqlcheck.