MavenBndRepository does not download remote updates to snapshot jars

[pconley-onc]

Bndtools version: 7.0.0 (Git-Descriptor: 7.0.0.RC2-3-gb82dc867a; Git-SHA: b82dc86) and 7.1.0 (Git-Descriptor: 7.0.0.DEV-1660-g47e504d78; Git-SHA: 47e504d)
Java version: OpenJDK 64-Bit Server VM Temurin-17.0.3+7 (build 17.0.3+7, mixed mode)
OS: Ubuntu 22.04

We have a non-bnd OSGi bundle being published to a private Maven repository several times daily by CI. I'm trying to use this bundle in a bnd workspace through a MavenBndRepository, but it's not being updated properly: the bundle was initially downloaded correctly, but now that new versions have been published, clicking on Refresh Repositories Tree doesn't download them.

My configuration is as follows. build.bnd has a plugin (among others)

-plugin: \
    aQute.bnd.repository.maven.provider.MavenBndRepository; \
       snapshotUrl=https://${mavenRepoUrl}/snapshots; \
       index=${.}/onc.maven; \
       name="ONC Maven"; \
       readOnly=true, \
    ...

onc.maven:

ca.uvic.onc:oceans-next:0.1.0-SNAPSHOT

When opened in Eclipse, the jar reports a Last Modified timestamp of 2025-01-10T16:56:34Z (this is wrong - it's actually 16:56:34 at UTC-8, but I assume this isn't a problem); this matches lastModified value in ~/.m2/repository/ca/uvic/onc/oceans-next/0.1.0-SNAPSHOT/maven-metadata-D1C6EDBA.xml, which contains (among other things)

<snapshotVersion>
  <extension>jar</extension>
  <value>0.1.0-20250111.010316-10</value>
  <updated>20250111010316</updated>
</snapshotVersion>

However, the latest version according to our Maven repository's maven-metadata.xml is

<snapshotVersion>
  <extension>jar</extension>
  <value>0.1.0-20250113.174326-11</value>
  <updated>20250113174326</updated>
</snapshotVersion>

Clicking Refresh Repositories Tree doesn't download the new metadata file or the new jar, and doesn't log any errors. It appears that re-launching Eclipse sometimes updates the jar, but I haven't been able to reproduce that reliably.

In case it makes a difference, I do have a ~/.m2/settings.xml containing

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd">

<profiles>
  <profile>
    <repositories>
      <repository>
        <id>onc-maven-snapshots</id>
        <name>ONC Maven</name>
        <url>${mavenRepoUrl}</url>
        <releases>
          <enabled>false</enabled>
        </releases>
        <snapshots>
          <enabled>true</enabled>
          <updatePolicy>always</updatePolicy>
          <checksumPolicy>ignore</checksumPolicy>
        </snapshots>
      </repository>
    </repositories>
  </profile>
</profiles>

</settings>

I don't know whether this is a bndtools issue or my own problem.

[chrisrueger]

Clicking Refresh Repositories Tree doesn't download the new metadata file or the new jar, and doesn't log any errors. It appears that re-launching Eclipse sometimes updates the jar, but I haven't been able to reproduce that reliably.

Could you elaborate how you determine that the latest version is not used? I mean do you look somewhere (e.g. BND'S Jar Viewer) and see outdated info? or do you notice it in the running code which still seems to use an old version?

When your CI process publishes your bundle to your m2 repo, does it always use the same version e.g. ca.uvic.onc:oceans-next:0.1.0-SNAPSHOT ?

From https://bnd.discourse.group/t/caching-mvn-source-jars-multiple-downloads-on-eclipse-start/244/3 I read:

Also, the MavenBndRepository should only try and download the source if it’s not already present in the local Maven cache.

So in your case a re-"download" is not needed, since you always use the same version and it is already there.

Some thoughts:

• update to bndtools 7.1.0 . The points below assume 7.1.0

• try using the Bndtools Explorer Refresh button. It does a bit more stuff.
  

• bnd keeps some caches, e.g. /cnf/cache. Try clearing them

• Right Click on Repo Browser on a version then Copy to Clipboard / Copy Info. Does this info look correct? It also shows folders to check.

[pconley-onc]

Could you elaborate how you determine that the latest version is not used?

The code isn't easily distinguishable between different versions. Yesterday I was mostly relying on the jar's mod time from ls -l, plus the generated version in maven-metadata.xml. Today I added a manifest header containing the build time to be more certain.

Also, the MavenBndRepository should only try and download the source if it’s not already present in the local Maven cache.

So in your case a re-"download" is not needed, since you always use the same version and it is already there.

According to my understanding of Maven - which might be flawed - a "-SNAPSHOT" version overrides the normal versioning rules: maven-metadata.xml contains the "true" version, based on the publish timestamp and the number of previous publications, and should update the jar if the metadata indicates it's out of date. As far as I can tell, bnd is trying to handle this from #4280 by forcibly updating snapshot jars on refresh.

On bndtools 7.1.0:

• Bndtools Explorer Refresh has no effect
• Clearing bnd's caches has no effect
• The repository browser's Copy Info looks correct

However, I had no idea you could right-click on artifacts in the repository browser: doing this displays an option "Refresh snapshot". I've figured out that I can update the jar with

1. In the repositories browser, right-click on the version -> select Clear from Cache
   • It sometimes works if I skip this, but not consistently. I might be doing something wrong.
2. Again in the repositories browser, right-click -> select Refresh snapshot
3. In the bndtools explorer, click Reload workspace

It's a little unfortunate that this requires so many extra steps beyond simply refreshing the repositories tree (especially since this is just a runtime dependency and needing to reload the workspace causes a lengthy rebuild), but it does seem to work.

[chrisrueger]

Thanks @pconley-onc for the additional info. I have no solution yet but did some more debugging to find out what is happening under the hood:

• Right-click Clear from Cache on e.g. assertj-core/3.24.2
  • is deleting ~/.m2/repository/org/assertj/assertj-core/3.24.2

I can confirm that after "Clear from Cache" only a bndtools explorer / Reload workspace triggers a re-download from remote maven

Regarding your sentence:

When opened in Eclipse, the jar reports a Last Modified timestamp of 2025-01-10T16:56:34Z

Do you mean this timestamp in the JarViewer? (red arrow)

also

• check the contents of your EclipseWorkspace/.metadata/.plugins/bndtools.jareditor/.BndtoolsJAREditorTempFiles/temp
• this folder stores temp files used by the JarEditor of bndtools
• in the past there was a cleanup bug fixed in 7.1.0 (see BndtoolsJAREditorTempFiles keeps growing: 1 jar file creates two folders, but only one is removed on closing #6340)

Potential separate JarViewer bug:
If I then double-click on a revision in Repo-Browser after Clear from Cache then I get

It is caused that the JarViewer somehow does not handle the just-deleted file properly which has a null:// FileStore

(may in bndtools.jareditor.internal.JARTreePage.getFolder(URI, IProgressMonitor) and JarEditor.uri seems to still point to the just removed jar in ~/.m2/repository/org/assertj/assertj-core/3.24.2)

Conclusion

In my opinion Clear from Cache is what you need since it deleted the folder in your local maven repo e.g. ~/.m2/repository/org/assertj/assertj-core/3.24.2

It is to be discussed when a re-download should be triggered (currently it seems only bndtools explorer / Reload workspace is triggering a re-download).

a) maybe Clear from Cache should automatically trigger a re-download
b) or a separate menue entry like Re-Fetch remote artifact
c) or a combination of a) and b) Clear from Cache and Re-Fetch remote artifact
d) or maybe fixing the JarViewer Bug above would be enough when it re-downloads when the file does not exist in the local maven repo

[pconley-onc]

Do you mean this timestamp in the JarViewer? (red arrow)

Yes, that's the one (with the "Last Modfied" typo)

check the contents of your EclipseWorkspace/.metadata/.plugins/bndtools.jareditor/.BndtoolsJAREditorTempFiles/temp

One folder is created when I open the jar in the editor, then deleted when I close it.

I neglected to mention that when I do "Refresh snapshot" and try to open the jar in the jar editor, I get the following dialog:

Clicking "Yes" appears to do nothing.

It is to be discussed when a re-download should be triggered (currently it seems only bndtools explorer / Reload workspace is triggering a re-download).

It would be most convenient for me & my team if "Refresh snapshot" triggered the re-download itself, but I'm obviously unaware of what other considerations exist. The existing three-step process is adequate.

[chrisrueger]

Clicking "Yes" appears to do nothing.

Thanks for the screenshot. I have not seen this dialog, but I did not test with a snapshot but a normal revision.

I try to have a look.