Storage

A StorageProvider is where JobRunr keeps everything related to background job processing - the job type, method, arguments, state, and metadata are serialized to JSON and stored in your database. Nothing is kept in process memory: this is what makes your jobs durable, resilient to crashes. Because every job lives in the database, JobRunr can distribute work across multiple servers and pick jobs back up after a restart or crash, so nothing is lost.

JobRunr supports all major SQL and document-based NoSQL databases out of the box. Pick yours below to get the dependency and configuration you need.

Important

You need to add the correct driver dependency (JDBC driver or database client) for the database you choose. Each page below lists the one it needs.

Caution

Whatever database you pick, JobRunr must read from the same node it writes to. Replication is asynchronous and JobRunr is usually faster than it, so a read replica, a MongoDB secondary, or an Aurora/DocumentDB reader endpoint hands JobRunr stale data - which surfaces as ConcurrentModificationExceptions. Always point JobRunr at the primary (writer) node.

SQL databases

NoSQL databases

Your database is not listed? See Custom StorageProvider for how to implement support for any database.

Setting up an SQL database

Setting up an SQL database is easy because you probably don’t need to do anything: by default JobRunr automatically creates the necessary tables for your database. Just like Liquibase and Flyway, it ships with a database migration manager that manages the schema for you.

If you don’t want to give JobRunr’s DataSource DDL rights, you can create the tables yourself using one of the methods below.

Run the DatabaseCreator

The DatabaseCreator class creates the necessary tables from a terminal. Provide a user that has DDL rights:

java -cp jobrunr-8.8.0.jar org.jobrunr.storage.sql.common.DatabaseCreator {jdbcUrl} {userName} {password}

If the command succeeds, a confirmation message is shown.

Pro tip

With JobRunr Pro you don’t have to do this by hand. You can hand JobRunr a separate user that is only used during database setup and migrations. JobRunr keeps creating and updating the tables for you, while the user your application runs with never needs DDL rights. See Database migrations.

Generate the SQL scripts yourself

To generate the SQL scripts for your database so you can apply them manually (the files are generated in the current directory):

java -cp jobrunr-8.8.0.jar org.jobrunr.storage.sql.common.DatabaseSqlMigrationFileProvider {databaseType} ({tablePrefix})

Where databaseType is the migration type listed on each database’s page (e.g. postgres, mysql, oracle).

Then configure JobRunr to skip table creation, so it uses the tables you created instead of trying to create them again:

JobRunr.configure()
    .useStorageProvider(SqlStorageProviderFactory.using(dataSource, null, DatabaseOptions.SKIP_CREATE))
    // ...
    .initialize();

jobrunr:
  database:
    skip-create: true

quarkus.jobrunr.database.skip-create=true

jobrunr.database.skip-create=true

Pro warning

In JobRunr Pro, DatabaseOptions is a class that exposes additional configuration options (e.g., the database migration credentials).

Pro tip

JobRunr Pro can export the same scripts in Flyway or Liquibase format, so you can add them to the migrations you already manage.

Table prefix and schema

JobRunr supports a table prefix that is prepended to all table names. This is handy if you want to place the tables in a specific schema. Note that the delimiter between the schema and table must be added manually:

jobrunr.database.tablePrefix=MY_SCHEMA.

Setting up a NoSQL database

For the supported NoSQL databases, JobRunr creates a database called jobrunr and all the necessary collections automatically. See the MongoDB and Amazon DocumentDB pages for the client dependency and configuration.

To keep JobRunr’s collections in the database your application already uses, set database.database-name. The table prefix works here too and is prepended to every collection name:

jobrunr:
  database:
    database-name: my-application
    table-prefix: jobrunr_

quarkus.jobrunr.database.database-name=my-application
quarkus.jobrunr.database.table-prefix=jobrunr_

jobrunr.database.database-name=my-application
jobrunr.database.table-prefix=jobrunr_

What JobRunr creates in your database

JobRunr manages these objects itself through its migration manager - you never write to them directly, and their columns and fields are an implementation detail that may change between releases.

In an SQL database:

In a NoSQL database, the same objects exist as collections in the jobrunr database: jobs, recurring_jobs, background_job_servers, metadata and migrations.

If you configured a table prefix, it is prepended to all of these names. To see the exact DDL for your database, generate the SQL scripts - you can then adapt them to your needs. If you apply them yourself and keep tracking migrations in your own tool, you can drop the script that creates jobrunr_migrations: JobRunr only requires jobrunr_jobs, jobrunr_recurring_jobs, jobrunr_backgroundjobservers and jobrunr_metadata to be present when it starts with table creation skipped.

Multiple databases in one application

The framework integrations detect your database automatically - an SQL DataSource bean, or a NoSQL client bean such as MongoClient. When your application has more than one candidate, you have to tell JobRunr which one to use.

Choosing the database type

JobRunr picks its storage from the beans it finds - a DataSource selects the SQL provider, a MongoClient the MongoDB one - so most applications never set this. Use database.type when that choice isn’t the one you want:

jobrunr:
  database:
    type: sql

quarkus.jobrunr.database.type=sql

jobrunr.database.type=sql

So there are two reasons to set it: your application has more than one candidate bean, or you want DocumentDB or the in-memory provider, which are never picked implicitly.

With the Fluent API there is nothing to disambiguate: you pass the StorageProvider you want yourself.

Using a dedicated datasource

Use database.datasource when you have several DataSources of the same kind - for example one for your domain data and a separate one for JobRunr. The value is the name of the datasource bean, not a JDBC URL, and the setting only exists in the framework integrations.

jobrunr:
  database:
    datasource: jobRunrDataSource

quarkus.jobrunr.database.datasource=jobRunrDataSource

# the name of the DataSource bean JobRunr should use
jobrunr.database.datasource=jobRunrDataSource

Without this setting, JobRunr uses the default (unnamed) DataSource bean.

Configure the StorageProvider yourself

You can always build the StorageProvider yourself instead of letting JobRunr configure it - useful when you want a specific table prefix, explicit DatabaseOptions, or a provider that is not picked up automatically.

There are two ways to create one for an SQL database. SqlStorageProviderFactory inspects the JDBC connection and returns the matching implementation:

StorageProvider storageProvider = SqlStorageProviderFactory.using(dataSource);

Or instantiate the provider directly - each database page names its class (PostgresStorageProvider, MySqlStorageProvider, MongoDBStorageProvider, …). All SQL providers accept a DataSource and optionally a table prefix and DatabaseOptions. Other constructors are available too, please check the Javadoc of the StorageProvider implementation you’re interested in.

The framework integrations step aside as soon as you define a StorageProvider bean yourself:

JobRunr.configure()
    .useStorageProvider(new PostgresStorageProvider(dataSource))
    .useBackgroundJobServer()
    .initialize();

@Factory
public class JobRunrStorageProviderFactory {

    @Singleton
    @Replaces(bean = StorageProvider.class, factory = JobRunrSqlStorageProviderFactory.class)
    public StorageProvider storageProvider(DataSource dataSource, JobMapper jobMapper) {
        // JobRunr needs the pooled DataSource, not Micronaut Data's transaction-bound proxy
        DataSource target = DelegatingDataSource.unwrapDataSource(dataSource);
        PostgresStorageProvider storageProvider = new PostgresStorageProvider(target);
        storageProvider.setJobMapper(jobMapper);
        return storageProvider;
    }
}

@ApplicationScoped
public class JobRunrStorageProviderProducer {

    @Produces
    @Singleton
    public StorageProvider storageProvider(DataSource dataSource, JobMapper jobMapper) {
        PostgresStorageProvider storageProvider = new PostgresStorageProvider(dataSource);
        storageProvider.setJobMapper(jobMapper);
        return storageProvider;
    }
}

@Bean(destroyMethod = "close")
@DependsOnDatabaseInitialization // so JobRunr creates its tables after Flyway or Liquibase have run
public StorageProvider storageProvider(DataSource dataSource, JobMapper jobMapper) {
    PostgresStorageProvider storageProvider = new PostgresStorageProvider(dataSource);
    storageProvider.setJobMapper(jobMapper);
    return storageProvider;
}

Warning

Always call setJobMapper(jobMapper) when you create the StorageProvider in a framework - the auto-configuration normally does this for you, and without it JobRunr throws a NullPointerException as soon as it serializes a job. The JobMapper bean is provided by the starter, so you can simply inject it.

In Micronaut, JobRunr’s own factory is @Primary, so @Replaces is needed to override it whenever a DataSource or MongoClient bean is present. In Spring Boot and Quarkus, your bean wins automatically.

The Micronaut example unwraps the DataSource for a reason. When Micronaut Data is on the classpath it wraps every DataSource bean in a proxy whose getConnection() hands back the connection bound to the current transaction - and throws NoConnectionException when there is none. A BackgroundJobServer polls for work, claims jobs and writes state transitions on its own threads, outside any transaction, so it needs the real pooled DataSource. If you want job scheduling to join your @Transactional methods, that is what the JobRunr Pro transaction plugin is for.