Upgrade to v8

This guide documents how to migrate to JobRunr v8 and JobRunr Pro v8.

JobRunr and JobRunr Pro v8 are still in beta. For feedback on any feature in this release don’t hesitate to reach to start a discussion over on GitHub: https://github.com/jobrunr/jobrunr/discussions. If you are a JobRunr Pro customer, you can directly email us your feedback at support@jobrunr.io, please make sure to specify your version.

Important: This release has a few breaking changes and is not backward compatible due additional fields (e.g., the createdBy field in RecurringJob). Please carefully review the breaking changes section. This is particularly important if you’re configuring JobRunr via Spring properties.

Table of contents

Installation

For the OSS version, the v8 release is available in the Maven Central repository.

For Pro users, the v8 release is available on our private Maven repository.

Simply replace the version you’re on with 8.0.0-beta.2.

Carbon Aware Jobs

From version 8.0.0-beta.2 and on, the v8 release of JobRunr allows to add a margin to the schedule of (recurring) jobs in order to reduce the carbon footprint of your servers. This is a brand new feature and therefore requires no special migration.

♥ Thanks to the folks from MindWave for helping us realize this feature!

Usage

To learn more about the feature, follow the How To Reduce Your Carbon Impact With Carbon Aware Jobs guide.

Ahead of time RecurringJob scheduling

In v7 and lower, JobRunr schedules recurring jobs very close to the moment they need to run (typically a minute earlier). In v8, JobRunr schedules the recurring job as soon as the previous run is finished. This new behaviour is required for carbon aware recurring jobs to work.

For example, take a recurring job that runs daily at midnight.

Behaviour in v7:

• JobRunr attempts to schedule the job everyday at ~11:59pm.

Behaviour in v8 (assuming the recurring job is recently created):

• JobRunr schedules an initial run at 12:00 am and attempts to schedule the next for the following day at 12:00 am when the previous run is done.

Why attempt? Because JobRunr prevents concurrent runs of the same recurring job by default.

Usage

Nothing to do, this is the default behaviour of recurring jobs.

Limitations

• JobRunr OSS only: users will need to take manually delete the job scheduled ahead of time when they delete or make a change to the schedule of a RecurringJob. A dialog has been added to the dashboard to remind of this when deleting a recurring job.
• JobRunr Pro only: recurring jobs with a scheduling interval lower than a minute still behave as in v7, this because frequently running recurring jobs do not benefit from carbon aware optimizations.
• JobRunr Pro only: a triggered recurring job on the dashboard, won’t prevent JobRunr from scheduling new ones.

Reduced database load

In JobRunr v8, we’ve reviewed the datatypes, queries and indexes to improve performance and reduce the load on the database. In our tests we’ve seen at least 2x improvement across all databases when all JobRunr features (such as dynamic queues, rate limiters, batch jobs, etc.) are enabled. In practice, you can expect less load on your database as queries run faster and data takes less space.

WARNING: Databases with column type changes: Oracle, MariaDB, MySQL, and SQL Server

Usage

The changes are automatically applied when JobRunr is managing the SQL migrations. For user controlled migration, see the Database Migrations documentation to learn how to get the SQL scripts from JobRunr.

Multi-Cluster Dashboard

The JobRunr Pro Multi-Cluster Dashboard is a separate web server that offers a unified view over multiple JobRunr Pro clusters. Monitoring multiple instances can get tiresome when running a lot of different clusters, all running their own jobs. With the Multi-Cluster Dashboard, you can monitor the health of all clusters in one place.

Usage

For more information on how this works and how to configure it, please see the Multi-Cluster Dashboard Documentation.

The multi cluster dashboard is designed to be deployed as a standalone application and requires JDK 21 or higher.

K8s autoscaling

JobRunr Pro v8 provides different metrics (e.g., the worker’s usage, the amount of enqueued jobs, etc.), to customize Kubernetes autoscaling.

Usage

We’ve written another guide to demonstrate the power of these metrics when provided to KEDA.

Kotlin Serialisation support

JobRunr v8 introduces a new JsonMapper; the KotlinxSerializationJsonMapper to improve the Kotlin developer’s experience when using JobRunr.

Usage

Setup the dependencies kotlinx-serialization dependencies as demonstrated in the official Kotlin documentation. And add JobRunr’s Kotlin language support. JobRunr’s support for Kotlin Serialisation is only available from Kotlin versions 2.1 or higher so we need to add the artifact jobrunr-pro-kotlin-2.1-support, or jobrunr-kotlin-2.1-support for JobRunr OSS.

plugins {
    kotlin("plugin.serialization") version "2.1.20"
}

dependencies {
    implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.8.0")
    implementation("org.jobrunr:jobrunr-pro-kotlin-2.1-support:8.0.0-beta.2") // Replace the artifact name by `jobrunr-kotlin-2.1-support` when using JobRunr OSS
}

For now, you’ll need to programmatically configure the KotlinxSerializationJsonMapper either using JobRunr fluent API or by replacing the default JsonMapper bean provided by JobRunr’s autoconfiguration.

Note: we have not tested versions lower than 1.8.0 of kotlinx-serialization-json. Additionally, KotlinxSerializationJsonMapper may not be able to deserialize items serialized by another JsonMapper.

Example with the fluent API:

@OptIn(InternalSerializationApi::class, ExperimentalSerializationApi::class)
fun main() {
    val scheduler = JobRunrPro.configure() // replace JobRunrPro by JobRunr when using the OSS version
        .useJsonMapper(KotlinxSerializationJsonMapper())
        .useStorageProvider(InMemoryStorageProvider())
        .useDashboard()
        .useBackgroundJobServer()
        .initialize()
        .jobScheduler!!
}

♥ Thanks to @SIMULATAN for contributing this feature!

Label order

Since their introduction, JobRunr stored labels in no particular order (technically as a Set). From v8 onwards, the labels will be stored in the order defined by the user (technically as a List). This gives developers control over the order the labels are going to be displayed on the Dashboard.

Usage

If you’re using the @Job annotation to configure labels, there is nothing to do, other than ensuring the order is to your liking.

If you’re using the JobBuilder to configure labels:

1. You’re using withLabels(String... labels), e.g., aJob().withLabels("label1", "label2"), simply make sure the ordering is to your liking.
2. You’re using withLabels(Set<String> labels), e.g., aJob().withLabels(Set.of("label1", "label2")), move to aJob().withLabels(List.of("label1", "label2")). The expected type is now List<String>.

Label configuration

With JobRunr Pro v8, you have even greater power over ordering and colors of labels thanks to the introduction of the Dashboard configuration: LabelConfiguration.

Usage

Users can provide a list of LabelConfiguration, each defining a labelPrefix and optionally a color in rgb or hex format. If the color is omitted, a random color is consistently applied to all labels that match the configured prefix.

Additionally, once a label is configured, it’ll be displayed in the order it appears in the configuration. If the following is the configuration:

jobrunr:
    ...
    dashboard:
      enabled: true
      label-configuration:
        - label-prefix: tenant
          color: "#333333"
        - label-prefix: building
          color: "#777777"

Any label prefixed with tenant will appear first in the list of labels of a job when displayed on the dashboard. If prefixed with building it’ll appear second if a label with tenant is also present (otherwise it’ll be displayed first). Any other label not prefixed with a configured label-prefix will appear after the tenant and building prefixed labels.

Automatic deletion of batch child jobs via the Dashboard

When deleting a BatchJob from the Dashboard, the user is provided with the option of deleting all the child Jobs of the batch.

Usage

When deleting a batch job on the dashboard, you can choose to also delete all of the batch child jobs.

Limitations

This option is only available if deleting one BatchJob at a time.

Pause and resume dynamic queues

JobRunr Pro’s dynamic queues allow to effortlessly achieve fair job processing, e.g., between tenants in a multi-tenant application. Dynamic queues can now be paused and resumed from the Dashboard or programmatically.

Usage

Pausing a dynamic queue from the Dashboard is as simple as going to your configured dynamic queue page and clicking on the spinning wheel. To resume you only have to click again.

Pausing a dynamic queue programmatically requires access to a DynamicQueueManager instance (which can be created by providing a StorageProvider and a JsonMapper). To pause a queue or multiple, call: pauseDynamicQueueJobProcessing() with the queues to pause. To resume, call resumeDynamicQueueJobProcessing() with the queues to resume.

Ratelimiting configuration at runtime

Faster processing with SmartQueue

Would you like even higher processing throughput? Try our new queuing mechanism: the SmartQueue. To increase the throughput JobRunr tries to eliminate the latency to get the jobs to process. With this queuing system, a worker no longer needs to go to the database when they finish, it can pick a Job from the ones JobRunr fetched right before it finishes.

This feature is particularly useful when processing short running jobs. Using it for long running jobs may also slightly reduce the overall processing time, making it suitable for mixed queues with short and long running jobs. This mechanism can also reduce the overall number of DB calls. In the best case scenario, a single thread fetches jobs to process for all the workers in one go.

Note: do not use the SmartQueue if you’re using RateLimiters. We still need to improve the SmartQueue to work well with RateLimiters too.

Usage

You’ll need to provide the SmartQueueBackgroundJobServerWorkerPolicy. Either via the Fluent API:

JobRunrPro
.configure()
.useBackgroundJobServer(
    usingStandardBackgroundJobServerConfiguration()
    .andBackgroundJobServerWorkerPolicy(new SmartQueueBackgroundJobServerWorkerPolicy())
);

Or via a Spring/Quarkus/Micronaut Bean

@Bean
public BackgroundJobServerWorkerPolicy backgroundJobServerWorkerPolicy() {
    JobRunrProperties.BackgroundJobServer backgroundJobServerProperties = properties.getBackgroundJobServer();
    BackgroundJobServerThreadType threadType = ofNullable(backgroundJobServerProperties.getThreadType()).orElse(BackgroundJobServerThreadType.getDefaultThreadType());
    int workerCount = ofNullable(backgroundJobServerProperties.getWorkerCount()).orElse(threadType.getDefaultWorkerCount());
    return new SmartQueueBackgroundJobServerWorkerPolicy(workerCount, threadType);
}

The above is an example for Spring Boot, the user should adapt the code to their needs.

The SmartQueueBackgroundJobServerWorkerPolicy can be provided with a BackgroundJobServerThreadType and a worker count.

Limitation

Can’t be used together with dynamic queues.

Improvement to saveReplace and createOrReplace

Replacing jobs is quite useful to prevent duplicate jobs, while running with the most fresh parameters. In v8, this API has undergone a couple of significant improvements:

1. You can now insert or replace multiple jobs at once via the JobScheduler or JobRequestScheduler API using createOrReplace(Stream<JobBuilder>) or the StorageProvider API using saveReplace(List<Job>).
2. For Postgres, MariaDB, MySQL and SQLite, JobRunr now performs an upsert.

Usage

Example using BackgroundJob:

BackgroundJob.createOrReplace(jobIds.stream().map(uuid -> aJob().withId(uuid).withDetails(() -> System.out.println("this is a test"))));

You may similarly use the BackgroundJobRequest or an instance of JobScheduler or JobRequestScheduler.

@AsyncJob to reduce boilerplate

This feature reduces the boilerplate needed to enqueue a Job. A call to a method, annotated with @Job from a class annotated with @AsyncJob, will be intercepted by JobRunr. Instead of the method being executed, an enqueued Job will be created and saved for later execution. This works similarly to Spring’s @Async, which executes the method asynchronously.

Usage

@Test
public void testAsyncJob() {
    asyncJobTestService.testMethodAsAsyncJob();
    await().atMost(30, TimeUnit.SECONDS).until(() -> storageProvider.countJobs(StateName.SUCCEEDED) == 1);
}

@AsyncJob
public static class AsyncJobTestService {

    @Job(name = "my async spring job")
    public void testMethodAsAsyncJob() {
        LOGGER.info("Running AsyncJobService.testMethodAsAsyncJob in a job");
    }
}

Limitations

Currently only available for Spring applications.

@Recurring synchronization

This feature aims to reduce manual intervention by automatically deleting a RecurringJob when JobRunr cannot find the associated method or when the method is no longer annotated with @Recurring.

Usage

Nothing to do, the feature is active by default.

Load license key using properties

The license key can now be loaded via the property jobrunr.license-key. This is useful to have a unified a way of management environment variables by using the features offered by frameworks.

The license key should be treated as secret, it should not be stored under version control.

Usage

jobrunr.license-key=your-jobrunr-pro-license-key

For Quarkus, you’ll need to prefix this with quarkus..

Database row locking strategy configuration

Pre JobRunr v7, JobRunr made use of optimistic locking to guarantee that jobs are not processed concurrently. In v7, we moved to FOR UPDATE SKIP LOCKED for the databases that support it. In v8, JobRunr Pro allows to configure this behaviour. You can choose not to use FOR UPDATE SKIP LOCKED even if your database supports it.

Usage

You can disable and fallback to optimistic locking with jobrunr.database.select-for-update-skip-locked-enabled=false or using SqlStorageProviderFactory#using(javax.sql.DataSource, java.lang.String, org.jobrunr.storage.DatabaseOptions).

Automatic deletion of failed jobs

By default, failed jobs are never deleted and require user intervention to handle them. This behaviour is reasonable as you almost certainly want to be aware of job failures. JobRunr Pro v8 allows to configure the period after which to automatically delete failed jobs.

Usage

Via a property:

jobrunr.background-job-server.delete-failed-jobs-after=PT365D

For Quarkus, you’ll need to prefix this with quarkus..

Via the fluent API:

JobRunrPro.configure()
    //...
    .useBackgroundJobServer(
        usingStandardBackgroundJobServerConfiguration()
        .andDeleteFailedJobsAfter(Duration.ofDays(365))
    )
    .initialize();

Breaking changes

Spring Boot Starter

• We’ve removed the org prefix from JobRunr’s Spring properties, make sure you update your application.properties otherwise your JobRunr configuration will not work.

StorageProvider

• The Redis and Elasticsearch StorageProviders were deprecated in v7 and are removed from JobRunr in v8. If you rely on LettuceStorageProvider, JedisStorageProvider or ElasticSearchStorageProvider, you may want to look into moving to another database.
• JobRunr Pro Column type changes for Oracle, MariaDB, MySQL, and SQL Server.
• JobRunr Pro CockroachStorageProvider requires at least CockroachDB v25.1.
• JobRunr Pro Set<String> getDistinctDynamicQueues(StateName... stateNames); has been removed from the StorageProvider. Closest alternative method is DynamicQueueStats getDynamicQueueStats(Function<String, String> toLabel).
• JobRunr Pro Method List<Job> getJobsToProcess(BackgroundJobServer backgroundJobServer, Optional<String> dynamicQueue, AmountRequest amountRequest) becomes List<Job> getJobsToProcess(BackgroundJobServer backgroundJobServer, DynamicQueueRequest dynamicQueueRequest, AmountRequest amountRequest), expecting a DynamicQueueRequest as second parameter.
• JobRunr Pro Removed StorageProvider#getRecurringJobIdsOfJobsWithState(StateName... states); use StorageProvider#getRecurringJobIdsOfJobs(JobSearchRequest jobSearchRequest) instead.
• [JobRunr OSS] Method boolean recurringJobExists(String recurringJobId, StateName... states) has been removed, use Instant getRecurringJobLatestScheduledInstant(String recurringJobId, StateName... states) instead.

Job

• AbstractJob#setLabels, JobBuilder#withLabels and RecurringJobBuilder#withLabels now expect a List instead of a Set as an argument.
• JobRunr Pro JobBuilder: method runAfter(...) is replaced by runAfterSuccessOf(...)
• JobParameterNotDeserializableException move from package to org.jobrunr.jobs to org.jobrunr.jobs.exceptions.
• Attribute recurringJobId has been removed from ScheduledState. Use Job#getRecurringJobId() instead.

JobContext

• JobDashboardProgressBar: method increaseByOne renamed to incrementSucceeded.
• JobDashboardProgressBar: method getProgress renamed to getProgressAsPercentage.

RecurringJob

• RecurringJob: constructors signature have changed. We recommend using RecurringJobBuilder if you need to programmatically create a RecurringJob.
• JobRunr Pro A triggered recurring job on the dashboard, won’t prevent JobRunr from scheduling new ones.
• Method RecurringJob#durationBetweenRecurringJobInstances() has been removed, use RecurringJob#getSchedule() and call Schedule#durationBetweenSchedules().
• CronExpression#create has been removed, use the constructor instead.
• Interval#create has been removed, use the constructor instead.

Micrometer integration

• Job stats metrics have changed name. Before: jobrunr.jobs.[statename] (where statename varies). After: jobrunr.jobs.by-state.

Kotlin Support

• JobRunr Pro The Exposed Transaction Support package name change: from org.jobrunr.exposed.transaction.sql to org.jobrunr.kotlin.storage.sql.exposed
• Dropped support for Kotlin 1.9.