[bug] Unexpected Recipe Revision Resolution with Multiple Remotes in Conan 2

[jasal82]

Describe the bug

Description
When using multiple remotes in Conan 2 with --update enabled, the resolution mechanism for recipe revisions (rrevs) appears to deviate from the behavior in Conan 1. Specifically, it does not respect the order of remotes as defined in remotes.json. Instead, recipe revisions are sorted by their timestamps across all remotes, which can lead to unexpected behavior and Conan1-incompatible package resolution.

Steps to Reproduce

1. Configure two remotes:
   • Remote 1: Contains recipe revisions with older timestamps.
   • Remote 2: Contains the same recipe revisions re-uploaded, resulting in newer timestamps.
2. Require a package with the same revision distributed across these two remotes but with different timestamps.
3. Enable the --update flag while resolving the package.

Expected Behavior
Conan should prioritize recipe revisions from the first remote in remotes.json, regardless of timestamp differences. For example:

• If [email protected]#bcd exists in both remotes with different timestamps, it should be resolved from Remote 1 if it appears first in remotes.json.

Actual Behavior
With --update enabled, Conan 2 queries recipe revisions from all remotes and sorts them by timestamp. This results in the recipe from Remote 2 being chosen, even when it has the same content as the revision in Remote 1 but a newer timestamp.

Code Reference
The issue seems related to the _find_newest_recipe_in_remotes function in proxy.py.

Example
Given the following remote distribution:

- Remote 1:
  - [email protected]#abc @ 2025/01/01
  - [email protected]#bcd @ 2025/01/02
- Remote 2:
  - [email protected]#abc @ 2025/01/05

With the current behavior:

• [email protected]#abc from Remote 2 is resolved as the latest revision due to the newer timestamp.
  Expected behavior:
• [email protected]#bcd from Remote 1 should be resolved as the latest revision since it is the most recent recipe that appears first in remotes.json.

Use case
We have a cache remote which is used to upload binary packages for recipes that were built by CI. It is different from the remote where the recipes are stored, and this is intentional (because the release repository does not enable delete permissions for immutability, but cache remote needs delete permissions for periodic cleanup tasks). When uploading a binary package the recipe entry is automatically created in the cache remote and gets the timestamp of creation instead of the original recipe timestamp. Both remotes are used when building consumers, with the recipe remote having higher priority.

Proposed solution
This is a tough one. I don't really have an idea how to solve this properly. Maybe the Conan 1 way of resolving across remotes could be restored, but that would probably break existing Conan 2 solutions. Or maybe it would be possible to store the original timestamp of downloaded recipes in the local cache so that it can be preserved when uploading the recipe again to another remote (this seems to be the only viable solution in my opinion).

Environment
Conan Version: 2.x

[memsharded]

Hi @jasal82

Thanks for your report.

In principle, this is intentional and by design in Conan 2. The --update behavior in conan 1 got some criticisms for not being that clear, so the rule in Conan 2 is much simpler: "it will fetch the latest in the provided remotes". If some server got a latest timestamp, this is the one expected to be downloaded, and this seems to fullfill very much the user expectations so far, it has been working and understood quite well.

But there might be something that I am missing, because Conan 2 does not track the remote servers origins for packages. So the fact that the same recipe revision is downloaded from one remote or the other should not have any effect, as the final result in the Conan cache will be identical. But specially, because the package binary resolution is not coupled anymore in Conan2 to the recipe previous resolution, so Conan should be able to successfully get your binaries from the other server.

We have a cache remote which is used to upload binary packages for recipes that were built by CI. It is different from the remote where the recipes are stored, and this is intentional (because the release repository does not enable delete permissions for immutability, but cache remote needs delete permissions for periodic cleanup tasks). When uploading a binary package the recipe entry is automatically created in the cache remote and gets the timestamp of creation instead of the original recipe timestamp. Both remotes are used when building consumers, with the recipe remote having higher priority.

This is what I don't fully understand. Conan 2 does a second iteration over remotes, independent of the recipe iteration one, looking for the binary, so it should be able to find it. It is not very clear what is failing to resolve or how it fails in the process, if you could please clarify that. Thanks!

[jasal82]

Hi @memsharded , the problem is not related to the binary resolution, it's the recipe resolution which breaks. And that is because the timestamp is relevant for determining the latest revision. We're not downloading the same revision from one or the other remote, we're getting a completely different revision as the latest one because the timestamps are different between the two repositories.

[jasal82]

As I wrote above, if I download a specific recipe revision from one remote and upload it to another, the uploaded one has a different timestamp. As long as there is only that single recipe revision it doesn't matter, because both have the same contents. But if there are other recipe revisions and we're requiring only based on the version, the latest revision is determined via the timestamps and these might have a different order across the revisions in different remotes.

[jasal82]

The cause of our problem is the new simple rule "it will fetch the latest in the provided remotes". This makes the assumption that recipes with a more recent timestamp belong to a more recent revision of the recipe which is generally a wrong assumption in my opinion. The assumption is only correct if any given recipe revision with identical content is only uploaded exactly once and only to a single remote OR if their timestamps are synchronized across all remotes. With a single remote it would even be safe to reupload the exact same revision again because Conan would be able to check that the content of the recipe to upload are the same as the one already in the remote and reject the upload. But if we're using multiple remotes this does not work anymore because I can download an older revision (which is not the "latest") from remote 1 and upload it to another remote which does not yet contain that revision. This will discard the original timestamp from remote 1 and assign a new timestamp. When we now add this second remote the revision in remote 2 will actually be considered the latest one because it has the latest timestamp, but it's actually only a copy of an existing older revision in remote 1.

[jasal82]

@memsharded Do you need any additional information? This is a blocker for us as it completely breaks our binary package caching strategy.

[jasal82]

Which timestamp is used for determining the "newest" recipe on the remote in the search algorithm? It seems that Conan 2 uses the timestamp in the index.json on the remote (respectively the one in the .timestamp file, which is generated by the server during the upload). Shouldn't it be using the timestamp from the conanmanifest.txt? That one is preserved across a download/upload cycle as far as I can see.

[jasal82]

Ok, I checked the REST calls. The latest revision is determined on server side and queried using the v2/conans/<pkg>/<version>/<user>/<channel>/latest API. That one uses the timestamp from the index.json which is created by the server and is always equal to the upload time. However, it should use the timestamp in the conanmanifest.txt instead. Can you confirm that this is a bug?

[jasal82]

Possibly related to #14068

[jasal82]

Created a PR which addresses the immediate problems we're facing due to this issue. However, this is not a fix for the issue itself, because I think it is a fundamental design flaw which requires more than just a workaround.