3.0.0b1

What's Changed

Breaking changes

A long, complex, necessary and highly appreciated work from @onlyann who has been quite active this year: rewriting most of the main job processing loop of the Procrastinate Worker (#1114) as discussed in #933. It was followed by quite a few improvements we wanted to make for a long time and took the opportunity of a breaking release to do, as described below.

Changes include:

• The migration process has changed, especially if you apply migrations while the application is running. The way it was before was that you had to apply the migrations that came with version X.Y.Z before upgrading the code to X.Y.Z with complex procedures around minor/major versions. The new way is that migrations are noted as pre or post, you apply the pre migrations before deploying the accompagnying new version of Procrastinate, and you apply the post migrations after deploying the said code. If you are several versions late, you'll have to repeat these for every intermediate release that comes with migrations. See the Migration documentation.
• Aborted asynchronous jobs are now cancelled through asyncio's cancel method, which can be handled gracefully by users using shielding, as explained in the Cancel documentation. Synchronous jobs will not be cancelled, but job_context.should_abort() will return True so that you can handle the abortion request in your code.
• A better graceful stopping mechanism with a timeout controlled by shutdown_graceful_timeout. During shutdown, jobs (both synchronous and asynchronous) get an abort request as described in the previous bullet point. See all details in the Shutdown documentation.
• The way a job's abortion request is stored used to be part of the job's status (ABORTING). It's now an independent field (Job.abort_requested). This has implications if you introspect job states, such as with the Job objects or via the Django integration.
• The timeout worker option was renamed to fetch_job_polling_interval to better reflect its purpose. It's accompanied by a new setting abort_job_polling_interval. Both regulate how often the worker contacts the database to request new jobs or cancellation requests, in case they don't receive notifications from the LISTEN/NOTIFY mechanism. This is explained in the Polling documentation. Note that the work from @onlyann led to workers making significantly less SQL queries, especially for workers with high concurrency settings.
• There have been a few changes on the JobContext object that your tasks receive if the task decorator was passed with pass_context=True:
  • job_context.job_result was removed. The only relevant information it contained was the start timestamp, which is now accessible via job_context.start_timestamp
  • The attributes app and job had static typing, mentioning that those attributes could be None while they actually were always set. They are now properly typed.
  • job_context.should_abort_async() was removed. Knowing whether the job should abort or not, is not doing I/Os anymore, so both sync and async code can now use job_context.should_abort(). job_context.abort_reason tells you whether a job is being aborted because the user requested an abortion, or because the worker is stopping.
  • The job_context.task attribute is now a read-only property. Not that it wasn't a good idea to modify it before, but in the very surprising case that one would do, that's now not possible anymore.
• In the arguments of builtin_tasks.remove_old_jobs(), remove_error was renamed to remove_failed for consistency. Similarly, in JobManager.delete_old_jobs(), include_error was renamed include_failed.
• Support for Python 3.8 was removed following its deprecation.

Bug Fixes

• Respect the priority of a periodic task by @medihack in #1217
• Fix a performance regression when fetching a job by @onlyann in #1225

Related PRs:

• Sync tasks will get an abort request on shutdown (by @onlyann in #1215).
• Abort job polling interval by @onlyann in #1192
• Refactor worker by @onlyann in #1114
• Add support for abort notification by @onlyann in #1172
• Abort async tasks using asyncio by @onlyann in #1190
• Use additional abort field on jobs table for abortion requests by @medihack in #1139
• Simplify job context by @onlyann in #1213
• Rename remove_error and include_error to remove_failed and include_failed by @medihack in #1168
• Upgrade migration process by @medihack & @ewjoachim in #1245
• Drop Python 3.8 support by @medihack in #1224

Miscellaneous

• Fill missing generic type parameters for JobContext.task by @fau-st in #1218
• Async InMemoryConnector (testing) by @onlyann in #1159
• Allow running on windows by @spapas in #1236
• Constraint to prevent jobs in todo state with abort_requested by @medihack in #1256
• Add devcontainer setup by @medihack in #1264
• Don't use dropped python version for static type checking by @medihack in #1269
• Fix to lowest Python version for static type checking in CI by @medihack in #1271
• Update v3 from main branch by @medihack in #1278
• Merge branch 'main' into v3 by @medihack in #1166
• V3 update by @ewjoachim in #1181
• Merge main into v3 by @medihack in #1203

Documentation

• Update CONTRIBUTING.md by @ewjoachim in #1204
• Add state graph (using mermaid) to the discussion page by @medihack in #1208
• Fix two small typos in Django docs by @tobami in #1238

Dependencies

• Make docs dependencies non optional again by @medihack in #1202

Kudos

MVPs

• Special thanks to @onlyann who has brought significant changes and helped us bring a lot of the good stuff you can see in this release.
• Never-ending stream of kudos to @medihack who stepped up as a maintainer and has been an excellent steward & force for good to the project.

New contributors for this release

• @tobami made their first contribution in #1238
• @spapas made their first contribution in #1236

Feedback

We'd love for you to share your thoughts about this release to us:

• Was the upgrade simple enough? Were these release notes detailed enough? Was the timing good? What do you think of the project and its updates? In the future, would you prefer new features or more stability? Please feel free to chime in in the discussion accompanying this release to make your voice heard.

Full Changelog: 2.14.0...3.0.0b1

2.15.1

What's Changed

Miscellaneous

• Fix small Windows issues by @spapas in #1236

Documentation

• Fix two small typos in Django docs by @tobami in #1238

New Contributors

• @tobami made their first contribution in #1238
• @spapas made their first contribution in #1236

Full Changelog: 2.15.0...2.15.1

2.15.0

Migrations

02.14.01_01_add_indexes_for_fetch_job.sql

Note

The new indexes will block the table while they are being built. Unless you have millions of jobs in the table, it should not be a problem, but you may want to create the index concurrently if you can. Note that you cant create an index concurrently in a transaction, which is why the migration code we suggested doesn't use the CONCURRENTLY keyword. To create the index concurrently, replace CREATE INDEX in the migration code with CREATE INDEX CONCURRENTLY.

What's Changed

Bug Fixes

• fix fetch job performance regression by @onlyann in #1225

Full Changelog: 2.14.1...2.15.0

2.14.1

Migrations

None

What's Changed

Bug Fixes

• Fill missing generic type parameters for JobContext.task by @fau-st in #1218
• Respect the priority of a periodic task by @medihack in #1217

Dependencies

• Make docs dependencies non optional again by @medihack in #1202
• Document dependencies procedures by @ewjoachim in #1204

Full Changelog: 2.14.0...2.14.1

2.14.0

Migrations

None

What's Changed

Miscellaneous

• Fix GitHub's upload-artifact breakage by @ewjoachim in #1179
• Add return type informations for decorated tasks by @fau-st in #1196

Documentation

• Improve task and defer docs by @medihack in #1174
• Add page to docs about job relationships in Django by @medihack in #1191

Dependencies

• Make recent Python versions use recent Django versions by @ewjoachim in #1197

New Contributors

• @fau-st made their first contribution in #1196

Full Changelog: 2.13.2...2.14.0

2.13.2

Migrations

None

What's Changed

Bug Fixes

• Fix Django + cron using the wrong connector by @ewjoachim in #1164

Miscellaneous

• Add Kai Schlamp as an author by @ewjoachim in #1163

Documentation

• Remove sphinx-autodoc-typehints, upgrade Myst & Sphinx by @ewjoachim in #1153

Full Changelog: 2.13.1...2.13.2

2.13.1

Migrations

None

What's Changed

Bug Fixes

• Wrap Django exceptions reported by @katlyn in #1162

Full Changelog: 2.13.0...2.13.1

2.13.0

Migrations

None

What's Changed

Bug Fixes

• Run sync task in its own thread by @medihack in #1160

Full Changelog: 2.12.0...2.13.0

2.12.0

Migrations

None

What's Changed

Bug Fixes

If you want to document tasks with Sphinx's Autodoc, check the new doc! Following a report from @JacobCoffee in #1152

Full Changelog: 2.11.0...2.12.0

2.11.0

Migrations

None

What's Changed

Miscellaneous

• Make django Admin interface support much better UX-wise by @ticosax in #1140

Full Changelog: 2.10.0...2.11.0