Fully overhaul package for modern Swift/Vapor/Fluent #1

gwynne · 2024-01-27T12:46:19Z

Pretty much what it says. Changes in public API:

JobMetadataMigration is no longer configurable
QueuesFluentDbType has been removed

Current plan is for these changes to be tagged as 3.0.0-beta.2

grahamburgsma

Just want to say thanks for giving this package some love! ❤️

Sources/QueuesFluentDriver/JobModel.swift

…. Remove soft-deletion support (Fluent will not support this in the future). Stop hardcoding supported database names altogether, rely on dialect flag instead. Normalize database column names.

0xTim

Some queries but I really think we should remove all the future APIs these days for a major release

Package.swift

Sources/QueuesFluentDriver/FluentQueue.swift

Patrick-Kladek · 2024-02-03T08:14:07Z

Hi, is there an ETA when this is released? This PR is still in draft otherwise I would have jumped right into it to try it out. I would love to get rid of my redis instance to simplify the stack & reduce costs. Anyway thanks for the amazing work you do with vapor.

…they revealed

…del. Make `attempts` non-optional in the database. Don't use variable column names in the migration. Consolidate, document, and expand upon the SQLKit/FluentKit extensions.

Patrick-Kladek · 2024-02-14T19:03:04Z

Sources/QueuesFluentDriver/JobModelMigrate.swift


-public struct JobMetadataMigrate: Migration {
-    public init() {}


I think we might still need this, otherwise I get this error:

'JobMetadataMigrate' initializer is inaccessible due to 'internal' protection level

Patrick-Kladek · 2024-02-14T19:08:05Z

I've tried this branch, however jobs fail with following error:

Printing description of error:
▿ PSQLError(code: server, serverInfo: [sqlState: 42601, file: scan.l, line: 1188, message: syntax error at or near ")", position: 72, routine: scanner_yyerror, localizedSeverity: ERROR, severity: ERROR], triggeredFromRequestInFile: PostgresKit/PostgresDatabase+SQL.swift, line: 60, query: PostgresQuery(sql: UPDATE "_jobs_meta" SET "state" = $1, "updated_at" = current_timestamp() WHERE "_jobs_meta"."id" = (SELECT "_jobs_meta"."id" FROM "_jobs_meta" WHERE "_jobs_meta"."state" = $2 AND "_jobs_meta"."queue" = $3 AND coalesce("_jobs_meta"."delay_until", current_timestamp()) <= current_timestamp() ORDER BY "_jobs_meta"."delay_until" ASC LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING "_jobs_meta"."id", binds: [(****; TEXT; format: binary), (****; TEXT; format: binary), (****; TEXT; format: binary)]))
  ▿ backing : <Backing: 0x6000010306c0>

This error is not visible in the console (its redacted):

[ ERROR ] Job run failed: PSQLError – Generic description to prevent accidental leakage of sensitive data. For debugging details, use `String(reflecting: error)`.

I added a breakpoint in AsyncAwaitSupport.swift:70 to get the error.

Patrick-Kladek · 2024-02-14T19:09:56Z

Sources/QueuesFluentDriver/FluentQueue.swift

+                return try await self.sqlDb.update(JobModel.sqlTable)
+                    .set(\JobModel.$state, to: QueuesFluentJobState.processing)
+                    .set(\JobModel.$updatedAt, to: .now())
+                    .where(\JobModel.$id, .equal, .group(select.query))
+                    .returning(JobModel.sqlColumn(\.$id))
+                    .first(decodingColumn: JobModel.key(for: \.$id), as: String.self)
+                    .map(JobIdentifier.init(string:))


My investigation shows that the error is thrown in here. However my limited knowledge with sql prevents me from investigating further or fixing it as I have no idea what's going on.

Not sure if important, the postgres table `_jobs_meta is empty

gwynne · 2024-02-14T19:14:23Z

@Patrick-Kladek This PR is not at all ready for use yet, unfortunately 😰. I had to set it aside for a bit due to time constraints, but I will come back to finish it shortly; I wouldn't recommend trying it out until I've marked it ready for review, as you've seen, it doesn't even properly compile quite yet 😅. Sorry for any confusion!

Patrick-Kladek · 2024-02-14T19:19:44Z

@gwynne thanks for the heads up. I thought I'll give it a try as it was sitting here for a while and thought I would document my experience and at least help this way.

Please don't feel pressured to finish this asap, I see you are doing a lot on vapor and wanted to say thank you. If there is anything I could help you with, please let me know.

…roperty from FluentQueuesDriver, simplify error checking in makeQueue(), rename QueuesFluentJobState -> StoredJobState, reorganize JobModel, make JobModelMigration an AsyncSQLMigration, remove unused error cases from QueuesFluentError enum, add additional SQLKit convenience methods

…ng except specifying the database at configuration time and starting transactions. No longer even uses a Fluent model.

Patrick-Kladek · 2024-03-12T12:46:45Z

@gwynne can I help you with this task somehow? Maybe we can discuss specific tasks/goals and then I can give this a try. Are we waiting for some other dependency to be finished first?

…QLKit

codecov · 2024-04-28T18:25:14Z

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

Fully overhaul package for modern Swift/Vapor/Fluent.

db5f266

gwynne requested a review from 0xTim January 27, 2024 12:46

grahamburgsma reviewed Jan 29, 2024

View reviewed changes

Sources/QueuesFluentDriver/JobModel.swift Outdated Show resolved Hide resolved

gwynne added 2 commits January 30, 2024 20:24

Remove duplicate "delay this job until this date" column in job model…

9920685

…. Remove soft-deletion support (Fluent will not support this in the future). Stop hardcoding supported database names altogether, rely on dialect flag instead. Normalize database column names.

Fix compilation condition

6115ac9

0xTim reviewed Jan 31, 2024

View reviewed changes

Package.swift Outdated Show resolved Hide resolved

Sources/QueuesFluentDriver/FluentQueue.swift Show resolved Hide resolved

Sources/QueuesFluentDriver/FluentQueue.swift Show resolved Hide resolved

Sources/QueuesFluentDriver/FluentQueue.swift Show resolved Hide resolved

0xTim approved these changes Jan 31, 2024

View reviewed changes

Some general cleanup

f1bcf9c

gwynne added 5 commits February 3, 2024 11:23

Case-insensitivity-defeating rename 1/2

e21dbcd

Case-insensitivity-defeating rename 2/2

cfdfdc6

Some more general cleanup

c4a0fd9

Add a couple of tests adapted from the Redis driver and fix the bugs …

225b98c

…they revealed

Get rid of the separate JobDataModel, integrate the fields into JobMo…

98d8c64

…del. Make `attempts` non-optional in the database. Don't use variable column names in the migration. Consolidate, document, and expand upon the SQLKit/FluentKit extensions.

Patrick-Kladek reviewed Feb 14, 2024

View reviewed changes

gwynne added 3 commits February 16, 2024 04:31

README cleanup, add DocC catalog

86ee376

Update to use the new SQLKit branch and switch to SQLKit for everythi…

05617a8

…ng except specifying the database at configuration time and starting transactions. No longer even uses a Fluent model.

gwynne added 4 commits April 28, 2024 13:13

Bump minimum Swift version to 5.8 and switch to released version of S…

78b03e3

…QLKit

Add missing Sendable annotation

8b75c79

Fix JobModel migration when not using Postgres

75ad09d

Fix tests

e62bf89

gwynne marked this pull request as ready for review April 28, 2024 18:14

gwynne added 2 commits April 28, 2024 13:16

Fix CI

788d5dc

Fix CI (again)

ec777ac

gwynne merged commit 3720aa6 into main Apr 28, 2024
11 of 12 checks passed

gwynne deleted the modernization branch April 28, 2024 18:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fully overhaul package for modern Swift/Vapor/Fluent #1

Fully overhaul package for modern Swift/Vapor/Fluent #1

gwynne commented Jan 27, 2024

grahamburgsma left a comment

0xTim left a comment

Patrick-Kladek commented Feb 3, 2024

Patrick-Kladek Feb 14, 2024

Patrick-Kladek commented Feb 14, 2024

Patrick-Kladek Feb 14, 2024

Patrick-Kladek Feb 14, 2024

gwynne commented Feb 14, 2024

Patrick-Kladek commented Feb 14, 2024

Patrick-Kladek commented Mar 12, 2024

codecov bot commented Apr 28, 2024


		public struct JobMetadataMigrate: Migration {
		public init() {}

Fully overhaul package for modern Swift/Vapor/Fluent #1

Fully overhaul package for modern Swift/Vapor/Fluent #1

Conversation

gwynne commented Jan 27, 2024

grahamburgsma left a comment

Choose a reason for hiding this comment

0xTim left a comment

Choose a reason for hiding this comment

Patrick-Kladek commented Feb 3, 2024

Patrick-Kladek Feb 14, 2024

Choose a reason for hiding this comment

Patrick-Kladek commented Feb 14, 2024

Patrick-Kladek Feb 14, 2024

Choose a reason for hiding this comment

Patrick-Kladek Feb 14, 2024

Choose a reason for hiding this comment

gwynne commented Feb 14, 2024

Patrick-Kladek commented Feb 14, 2024

Patrick-Kladek commented Mar 12, 2024

codecov bot commented Apr 28, 2024

Welcome to Codecov 🎉