Add depsolving and package search to cloudapi #4390

bcl · 2024-09-27T23:36:49Z

This pull request includes:

adequate testing for the new functionality or fixed issue
adequate documentation informing people about the change such as
- submit a PR for the READMEs listed here
- submit a PR for the osbuild.org website repository if this PR changed any behavior not covered by the automatically updated READMEs

bcl · 2024-09-27T23:38:13Z

Note, this builds on the depsolving PR so I've closed it in favor of this one. Last 2 commits are package search and the previous ones lay the groundwork and implement depsolve.

github-actions · 2024-11-15T04:11:57Z

This PR is stale because it has been open 30 days with no activity. Remove "Stale" label or comment or this will be closed in 7 days.

thozza

While this PR delivers on its promises, I don't like the approach from an architectural point of view.

The server handles the depsolving and package search functionality directly, which is wrong. We already have a DepsolveJob type that the worker handles. So, the correct approach should be for the server to enqueue the DepsolveJob and use its result.

The same applies to package search—a new worker Job type should be added, and the search should occur on the worker.

thozza · 2024-11-28T12:16:29Z

internal/cloudapi/v2/compose.go

+		// If there is an architecture in the blueprint it overrides the request's arch
+		if len(bp.Arch) > 0 {
+			reqArch = bp.Arch
+		}


Just overriding the value does not seem like the best solution. We should ensure that the values in the image request and BP match if both are set or produce an error if they don't (as it is an invalid request).

No, the value in the blueprint should override the other (or maybe the request should override the optional arch in the bp, but they are not expected to match). This is how it works in the weldr API, if arch is set in a blueprint it overrides the default host arch.

The client should ensure a valid API call. This would mean extracting the architecture from the BP and making the correct compose request. The question is whether the architecture property should be added to the BP structure. Specifically, the API is already structured so that the requests are for specific environments and architecture, while BP is agnostic of these properties.

The Cloud API has no notion of the host arch, and everything must be specified in the request.

thozza · 2024-11-28T12:20:14Z

internal/cloudapi/v2/compose_test.go

+	request := &ComposeRequest{
+		Distribution: "fedora-40",
+		ImageRequest: &ImageRequest{
+			Architecture:  "x86_64",
+			ImageType:     ImageTypesAws,
+			UploadOptions: &uo,
+			Repositories:  []Repository{},
+		},
+		Blueprint: &Blueprint{
+			Name:         "arch-test",
+			Architecture: common.ToPtr("aarch64"),
+		},
+	}


IMHO, this compose request should not be treated as valid. The architecture values don't match.

thozza · 2024-11-28T12:22:38Z

internal/cloudapi/v2/openapi.v2.yml

+  /depsolve/blueprint:
+    post:
+      operationId: postDepsolveBlueprint
+      summary: Depsolve one or more blueprints


This endpoint depsolves only a single BP.

thozza · 2024-11-28T12:33:07Z

internal/cloudapi/v2/compose.go

 // slightly different from the Cloudapi's Customizations section
 // This starts with a new empty blueprint.Customization object
 // If there are no customizations, it returns nil
-func (request *ComposeRequest) GetCustomizationsFromBlueprintRequest() (*blueprint.Customizations, error) {
-	if request.Blueprint.Customizations == nil {
+func (rbp *Blueprint) GetCustomizationsFromBlueprintRequest() (*blueprint.Customizations, error) {


The function name should be GetCustomizationsFromBlueprint, judging from the comment above it.

bcl · 2024-12-02T21:26:19Z

So, the correct approach should be for the server to enqueue the DepsolveJob and use its result.

What's the benefit of doing that? We already have the solver available in the server.

thozza · 2024-12-03T09:07:25Z

So, the correct approach should be for the server to enqueue the DepsolveJob and use its result.

What's the benefit of doing that? We already have the solver available in the server.

I can think of these:

To maintain reasonable architecture of Cloud API and not taint it with how things used to be done in Weldr API (which made a lot of assumptions of everything running on the same host).
To enable the SaaS version, eventually take advantage of any functionality you'll add (e.g., the package search).
The fact that the solver and the worker are on the same host as the API server in the on-prem scenario is an implementation detail in this case.

If included it overrides the architecture in the compose image request. Related: RHEL-60125

This will allow depsolving blueprints and returning package metadata for the dependencies. Related: RHEL-60125

This function only depends on the Blueprint (cloudapi request type, not the internal/blueprint) so move it to a function on that so that it can be reused by other users of the cloudapi Blueprint. Related: RHEL-60125

Related: RHEL-60125

In order to reuse PackageMetadata with DepsolveResponse and not include unused fields this changes the sigmd5 entry to an optional field. This doesn't effect the use of PackageMetadata in the Compose response since it is always set, and it allows it to be omitted in the response for depsolving. Also adds a basic test for stagesToPackageMetadata Related: RHEL-60125

This also adds an actual repository json file for the test-disro. Without this the repo.ListDistros() function doesn't return any actual distros. Related: RHEL-60125

This converts the request's blueprint to an internal/blueprint, optionally selects the blueprint's distro and arch to override the host's and depsolves the blueprint. Also adds mock dnfjson depsolving to the test framework. And tests for the new behavior. Resolves: RHEL-60125

This will be used to retrieve information about specific packages, or packages matching a search glob pattern. Related: RHEL-60136

This takes the same input as depsolving (a blueprint and an optional list of repositories), searches the package metadata and returns the info about the matching packages in a structure that is similar to that returned by the weldrapi projects API. If the distro and/or arch are set in the blueprint they override the host's distro and arch. Resolves: RHEL-60136

kingsleyzissou

Would it be possible to split the package search out into another PR?

Merely for the fact that package search is more pressing than depsolve for the frontend

bcl · 2025-01-18T00:59:14Z

Replaced by #4564

bcl mentioned this pull request Sep 27, 2024

cloudapi depsolve (COMPOSER-2131) #4372

Closed

4 tasks

bcl changed the title ~~Main cloudapi packages~~ Add depsolving and package search to cloudapi Sep 27, 2024

bcl force-pushed the main-cloudapi-packages branch 2 times, most recently from ef7f046 to 80d2024 Compare October 15, 2024 21:14

github-actions bot added the Stale label Nov 15, 2024

bcl removed the Stale label Nov 15, 2024

bcl force-pushed the main-cloudapi-packages branch from 80d2024 to 2a20e5c Compare November 15, 2024 18:18

bcl requested a review from thozza November 26, 2024 23:29

bcl force-pushed the main-cloudapi-packages branch from 2a20e5c to e7c1d68 Compare November 26, 2024 23:36

thozza reviewed Nov 28, 2024

View reviewed changes

thozza requested review from ondrejbudai, achilleas-k and croissanne November 28, 2024 12:45

bcl marked this pull request as draft December 17, 2024 16:46

bcl mentioned this pull request Dec 17, 2024

cloudapi: Add architecture support to the optional Blueprint #4374

Open

4 tasks

bcl added 9 commits January 10, 2025 16:21

cloudapi: Add architecture support to the optional Blueprint

f2fcf82

If included it overrides the architecture in the compose image request. Related: RHEL-60125

cloudapi: Add /depsolve/blueprint route

72acfeb

This will allow depsolving blueprints and returning package metadata for the dependencies. Related: RHEL-60125

cloudapi: Move GetCustomizationsFromBlueprintRequest

0aedfcc

This function only depends on the Blueprint (cloudapi request type, not the internal/blueprint) so move it to a function on that so that it can be reused by other users of the cloudapi Blueprint. Related: RHEL-60125

cloudapi: Add solver to Server

a30b3aa

Related: RHEL-60125

v2_test: Add a test-distro-1 repository

68ea228

This also adds an actual repository json file for the test-disro. Without this the repo.ListDistros() function doesn't return any actual distros. Related: RHEL-60125

cloudapi: Add /packages/search route to cloudapi

d10ea2d

This will be used to retrieve information about specific packages, or packages matching a search glob pattern. Related: RHEL-60136

bcl force-pushed the main-cloudapi-packages branch from e7c1d68 to ea59f30 Compare January 11, 2025 00:25

kingsleyzissou reviewed Jan 13, 2025

View reviewed changes

bcl mentioned this pull request Jan 18, 2025

Add depsolve to cloudapi #4564

Merged

4 tasks

bcl closed this Jan 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add depsolving and package search to cloudapi #4390

Add depsolving and package search to cloudapi #4390

bcl commented Sep 27, 2024 •

edited by mgold1234

Loading

bcl commented Sep 27, 2024

github-actions bot commented Nov 15, 2024

thozza left a comment

thozza Nov 28, 2024

bcl Dec 2, 2024

thozza Dec 3, 2024

thozza Nov 28, 2024

thozza Nov 28, 2024

thozza Nov 28, 2024

bcl commented Dec 2, 2024

thozza commented Dec 3, 2024

kingsleyzissou left a comment

bcl commented Jan 18, 2025

Add depsolving and package search to cloudapi #4390

Add depsolving and package search to cloudapi #4390

Conversation

bcl commented Sep 27, 2024 • edited by mgold1234 Loading

bcl commented Sep 27, 2024

github-actions bot commented Nov 15, 2024

thozza left a comment

Choose a reason for hiding this comment

thozza Nov 28, 2024

Choose a reason for hiding this comment

bcl Dec 2, 2024

Choose a reason for hiding this comment

thozza Dec 3, 2024

Choose a reason for hiding this comment

thozza Nov 28, 2024

Choose a reason for hiding this comment

thozza Nov 28, 2024

Choose a reason for hiding this comment

thozza Nov 28, 2024

Choose a reason for hiding this comment

bcl commented Dec 2, 2024

thozza commented Dec 3, 2024

kingsleyzissou left a comment

Choose a reason for hiding this comment

bcl commented Jan 18, 2025

bcl commented Sep 27, 2024 •

edited by mgold1234

Loading