Jgit with reftable causes "Cannot load refs/meta/reject-commits" error #130

syntonyze · 2025-01-13T08:29:16Z

Version

7.0.0

Operating System

Linux/Unix

Bug description

1 - Set up a git repository (or use an existing one) and enable reftable (you can do this by running git refs migration, this will need git 2.47 at least):

 /opt/homebrew/Cellar/git/2.47.1/bin/git  refs migrate --ref-format=reftable

2 - Set up constant writes to the repository. This could be a simple for loop that pushes to the repository or a test framework. We reproduced this by using Gatling against Gerrit, for example, you could use the git simulation from the gatling-sbt-gerrit-test.

3 - Schedule jgit gc over a period of time X, ensuring that pruneexpire and prunepackexpire are set to X*2. We run GC every 1 hour and set pruneexpire and prunepackexpire every 2 hours. Our git config looks like this:

[core]
	repositoryformatversion = 1
	filemode = true
	bare = true
	logallrefupdates = false

[gc]
	pruneExpire = 120.minutes.ago
	prunePackExpire = 120.minutes.ago
	auto = 0
	autoPackLimit = 0

[extensions]
	refstorage = reftable

We execute the jgit process as follows:

java_args='-Xss8m -Xms8g -Xmx8g ' time jgit-7.0.0.sh --git-dir <path-to/repo.git gc --preserve-oldpacks

After a few iterations (usually 3 or 4), following a GC execution pushes start to fail with:

[2025-01-08T16:12:13.362Z] [HTTP-6331] ERROR com.google.gerrit.pgm.http.jetty.HiddenErrorHandler : Error in GET /a/ghs-gym/info/refs?service=git-receive-pack
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) [Guice/ErrorInjectingConstructor]: IOException: Cannot load refs/meta/reject-commits
  at ReceiveCommits.<init>(ReceiveCommits.java:514)

Full stack trace can be found here

Actual behavior

Even thought the GC is successful, subsequent pushes to the repository fail due to https://pastebin.com/raw/PAX4N4S1.

Expected behavior

GC is successful and the repository is not corrupted.

Relevant log output

[2025-01-08T16:12:13.362Z] [HTTP-6331] ERROR com.google.gerrit.pgm.http.jetty.HiddenErrorHandler : Error in GET /a/ghs-gym/info/refs?service=git-receive-pack
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) [Guice/ErrorInjectingConstructor]: IOException: Cannot load refs/meta/reject-commits
  at ReceiveCommits.<init>(ReceiveCommits.java:514)
  while locating ReceiveCommits annotated with @UniqueAnnotations$Internal(7)
  at AsyncReceiveCommits.<init>(AsyncReceiveCommits.java:263)
  while locating AsyncReceiveCommits annotated with @UniqueAnnotations$Internal(7)

Learn more:
  https://github.com/google/guice/wiki/ERROR_INJECTING_CONSTRUCTOR

1 error

======================
Full classname legend:
======================
AsyncReceiveCommits:        "com.google.gerrit.server.git.receive.AsyncReceiveCommits"
IOException:                "java.io.IOException"
ReceiveCommits:             "com.google.gerrit.server.git.receive.ReceiveCommits"
UniqueAnnotations$Internal: "com.google.inject.internal.UniqueAnnotations$Internal"
========================
End of classname legend:
========================

	at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:251)
	at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1151)
	at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:907)
	at jdk.proxy2/jdk.proxy2.$Proxy209.create(Unknown Source)
	at com.google.gerrit.httpd.GitOverHttpServlet$ReceiveFactory.create(GitOverHttpServlet.java:539)
	at com.google.gerrit.httpd.GitOverHttpServlet$ReceiveFactory.create(GitOverHttpServlet.java:518)
	at org.eclipse.jgit.http.server.ReceivePackServlet$InfoRefs.begin(ReceivePackServlet.java:70)
	at org.eclipse.jgit.http.server.SmartServiceInfoRefs.doFilter(SmartServiceInfoRefs.java:69)
	at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:210)
	at org.eclipse.jgit.http.server.SmartServiceInfoRefs.doFilter(SmartServiceInfoRefs.java:87)
	at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:210)
	at org.eclipse.jgit.http.server.RepositoryFilter.doFilter(RepositoryFilter.java:109)
	at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:210)
	at org.eclipse.jgit.http.server.NoCacheFilter.doFilter(NoCacheFilter.java:50)
	at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:210)
	at org.eclipse.jgit.http.server.glue.UrlPipeline.service(UrlPipeline.java:189)
	at org.eclipse.jgit.http.server.glue.SuffixPipeline.service(SuffixPipeline.java:70)
	at org.eclipse.jgit.http.server.glue.MetaFilter.doFilter(MetaFilter.java:147)
	at org.eclipse.jgit.http.server.glue.MetaServlet.service(MetaServlet.java:106)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:293)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:283)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
	at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:403)
	at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92)
	at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:120)
	at com.google.gerrit.httpd.RequireIdentifiedUserFilter.doFilter(RequireIdentifiedUserFilter.java:50)
	at com.google.gerrit.httpd.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:62)
	at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:139)
	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
	at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:66)
	at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)
	at com.google.gerrit.httpd.AllowRenderInFrameFilter.doFilter(AllowRenderInFrameFilter.java:56)
	at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)
	at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:141)
	at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:60)
	at com.google.gerrit.httpd.ProjectBasicAuthFilter.doFilter(ProjectBasicAuthFilter.java:111)
	at com.google.gerrit.httpd.ProjectBasicAuthFilter.doFilter(ProjectBasicAuthFilter.java:111)
	at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:92)
	at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:64)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at com.googlesource.gerrit.plugins.ootb.FirstTimeRedirect.doFilter(FirstTimeRedirect.java:63)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.IOException: Cannot load refs/meta/reject-commits
	at com.google.gerrit.server.git.BanCommit.loadRejectCommitsMap(BanCommit.java:77)
	at com.google.gerrit.server.git.receive.ReceiveCommits.<init>(ReceiveCommits.java:578)
	at com.google.gerrit.server.git.receive.ReceiveCommits$$FastClassByGuice$$3fdaaf17.GUICE$TRAMPOLINE(<generated>)
	at com.google.gerrit.server.git.receive.ReceiveCommits$$FastClassByGuice$$3fdaaf17.apply(<generated>)
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
	at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1148)
	at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:907)
	at com.google.gerrit.server.git.receive.$Proxy210.create(Unknown Source)
	at com.google.gerrit.server.git.receive.AsyncReceiveCommits.<init>(AsyncReceiveCommits.java:308)
	at com.google.gerrit.server.git.receive.AsyncReceiveCommits$$FastClassByGuice$$3fbcafb2.GUICE$TRAMPOLINE(<generated>)
	at com.google.gerrit.server.git.receive.AsyncReceiveCommits$$FastClassByGuice$$3fbcafb2.apply(<generated>)
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
	at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1148)
	... 76 more
Caused by: java.nio.channels.ClosedChannelException
	at java.base/sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:159)
	at java.base/sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:356)
	at org.eclipse.jgit.internal.storage.io.BlockSource$2.read(BlockSource.java:86)
	at org.eclipse.jgit.internal.storage.reftable.BlockReader.readBlockIntoBuf(BlockReader.java:261)
	at org.eclipse.jgit.internal.storage.reftable.BlockReader.readBlock(BlockReader.java:255)
	at org.eclipse.jgit.internal.storage.reftable.ReftableReader.readBlock(ReftableReader.java:431)
	at org.eclipse.jgit.internal.storage.reftable.ReftableReader.seek(ReftableReader.java:214)
	at org.eclipse.jgit.internal.storage.reftable.ReftableReader.seekRef(ReftableReader.java:152)
	at org.eclipse.jgit.internal.storage.reftable.MergedReftable.seekRef(MergedReftable.java:115)
	at org.eclipse.jgit.internal.storage.reftable.Reftable.exactRef(Reftable.java:197)
	at org.eclipse.jgit.internal.storage.reftable.ReftableDatabase.exactRef(ReftableDatabase.java:241)
	at org.eclipse.jgit.internal.storage.file.FileReftableDatabase.exactRef(FileReftableDatabase.java:161)
	at com.google.gerrit.server.git.BanCommit.loadRejectCommitsMap(BanCommit.java:69)
	... 94 more

Other information

Server: We tested this against a Gerrit 3.10.3, running jgit c824610ab.
Client: we run jgit gc using jgit 6.80, 6.8.1 and 7.0.0, the problem arises with all versions.

We are still trying to reproduce this more quickly, perhaps programmatically, but so far this is the only way we have managed.

NOTE: I wonder if this is would be another manifestation of #102 , we will try to apply that patch and see if we can still reproduce.

The text was updated successfully, but these errors were encountered:

msohn · 2025-01-13T13:58:02Z

Please try with this fix for #102 https://eclipse.gerrithub.io/c/eclipse-jgit/jgit/+/1206683.
Without this fix an instance of FileReftableDatabase will not recognize ref updates done by another instance of FileReftableDatabase used in the same or a different thread or from another process (which could also run e.g. git gc).

You can also convert an existing repository to reftable storage using JGit CLI command convert-ref-storage.

syntonyze · 2025-01-17T20:13:18Z

Thanks @msohn , unfortunately this didn't work for us:

Even with fix https://eclipse.gerrithub.io/c/eclipse-jgit/jgit/+/1206683, the run eventually failed with:

Caused by: java.io.IOException: Cannot load refs/meta/reject-commits

The Exception is thrown when loading the reject commits map in BanCommit, as shown by https://pastebin.com/raw/PAX4N4S1:

Caused by: java.nio.channels.ClosedChannelException
	at java.base/sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:159)
	at java.base/sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:356)
	at org.eclipse.jgit.internal.storage.io.BlockSource$2.read(BlockSource.java:86)
	at org.eclipse.jgit.internal.storage.reftable.BlockReader.readBlockIntoBuf(BlockReader.java:261)
	at org.eclipse.jgit.internal.storage.reftable.BlockReader.readBlock(BlockReader.java:255)
	at org.eclipse.jgit.internal.storage.reftable.ReftableReader.readBlock(ReftableReader.java:431)
	at org.eclipse.jgit.internal.storage.reftable.ReftableReader.seek(ReftableReader.java:214)
	at org.eclipse.jgit.internal.storage.reftable.ReftableReader.seekRef(ReftableReader.java:152)
	at org.eclipse.jgit.internal.storage.reftable.MergedReftable.seekRef(MergedReftable.java:115)
	at org.eclipse.jgit.internal.storage.reftable.Reftable.exactRef(Reftable.java:197)
	at org.eclipse.jgit.internal.storage.reftable.ReftableDatabase.exactRef(ReftableDatabase.java:241)
	at org.eclipse.jgit.internal.storage.file.FileReftableDatabase.exactRef(FileReftableDatabase.java:161)
	at com.google.gerrit.server.git.BanCommit.loadRejectCommitsMap(BanCommit.java:69)
	... 94 more

A ClosedChannelException, when reading a block from the ref table.
Perhaps this issue is different from the one addressed by https://eclipse.gerrithub.io/c/eclipse-jgit/jgit/+/1206683?

zotttelbart · 2025-01-20T16:05:18Z

Can you try it with https://eclipse.gerrithub.io/c/eclipse-jgit/jgit/+/1206692 ?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jgit with reftable causes "Cannot load refs/meta/reject-commits" error #130

Jgit with reftable causes "Cannot load refs/meta/reject-commits" error #130

syntonyze commented Jan 13, 2025

msohn commented Jan 13, 2025

syntonyze commented Jan 17, 2025

zotttelbart commented Jan 20, 2025

Jgit with reftable causes "Cannot load refs/meta/reject-commits" error #130

Jgit with reftable causes "Cannot load refs/meta/reject-commits" error #130

Comments

syntonyze commented Jan 13, 2025

Version

Operating System

Bug description

Actual behavior

Expected behavior

Relevant log output

Other information

msohn commented Jan 13, 2025

syntonyze commented Jan 17, 2025

zotttelbart commented Jan 20, 2025