reftable in jgit doesn't respect changes outside of jgit

[zotttelbart]

Version

7.0.0

Operating System

Windows

Bug description

Changes on reftable outside of jgit (example gitbash) are not visible for jgit

Actual behavior

` Git git = Git.init().setDirectory(new File("c:\temp\jgit")).call();

	git.remoteAdd().setUri(new URIish("https://github.com/eclipse-jgit/jgit.git")).setName("origin").call();
	git.fetch().setRemote("origin").setProgressMonitor(new TextProgressMonitor()).call();
	git.checkout().setName("master").setStartPoint("origin/master").setCreateBranch(true).call();
	((FileRepository) git.getRepository()).convertRefStorage("reftable", true, false);
	

	git.tag().setName("foo").call();
	System.out.println(git.getRepository().findRef("foo"));

	System.out.println("Now change the Tag via git bash (git tag --force foo v7.0.0.202408201547-m3)");
	System.out.println("and press enter. You will see, the tag in jgit doesn't change");
	Scanner scanner = new Scanner(System.in);
	scanner.next();
	
	System.out.println(git.getRepository().exactRef("refs/tags/foo"));
	
	git.close();

`

outputs:

`Ref[refs/tags/foo=ef0923a81c00465034705a24a30b4c8d6878729f(3)]
Now change the Tag via git bash (git tag --force foo v7.0.0.202408201547-m3)
and press enter. You will see, the tag in jgit doesn't change

Ref[refs/tags/foo=ef0923a81c00465034705a24a30b4c8d6878729f(3)]
`

For jgit, the ref didn't change

output from gitbash

`$ git tag --force foo v7.0.0.202408201547-m3
Unlink of file 'C:/temp/jgit/.git/reftable/000000000003-000000000003-1b2b152a.ref' failed. Should I try again? (y/n) y
Unlink of file 'C:/temp/jgit/.git/reftable/000000000003-000000000003-1b2b152a.ref' failed. Should I try again? (y/n) y
Unlink of file 'C:/temp/jgit/.git/reftable/000000000003-000000000003-1b2b152a.ref' failed. Should I try again? (y/n) n
Unlink of file 'C:/temp/jgit/.git/reftable/000000000003-000000000003-1b2b152a.ref' failed. Should I try again? (y/n) n
Updated tag 'foo' (was ef0923a81)

$ git show-ref | grep foo
727666694e80ed39488451503d72f65bb4ef69f8 refs/tags/foo
`

you see #101 but the ref is changed

Expected behavior

Changes on tables.list should call

ReftableDatabase.clearCache() and
FileReftableStack.reload()

in FileReftableDatabase

Relevant log output

No response

Other information

I hope, this helps to improve jgit :)

[zotttelbart]

Hello,

the bug is still present in jgit 7.1 and git 2.47.1.

[msohn]

Pushed https://eclipse.gerrithub.io/c/eclipse-jgit/jgit/+/1206683 for review

[zotttelbart]

@msohn thx you. Its work!
Do you have any idea to solve #101?

[lucamilanesio]

@msohn thx you. Its work!

See the discussion on the original JGit RefTable Change 146568 with @hanwen which brought some lights on the design behind the behaviour you are observing.

In a nutshell, when you create the repository object and the associated refDatabase (backed by a refTable), it takes a consistent snapshot of the refs by reading the stack of immutable files and keeping them open. Whatever happens on the filesystem is not going to be detected on the fly until you try to perform an update.

If the update happens behind JGit's back but not inside the JGit's RefTable code, it won't be noticed until you close the repository and open it again.

Bottom line: I believe it is the way it works. Also it is consistent with the RefTable specs of what readers should do.

Do you have any idea to solve #101?

I believe that's unrelated to this problem and is more about compaction.

[zotttelbart]

In a nutshell, when you create the repository object and the associated refDatabase (backed by a refTable), it takes a consistent snapshot of the refs by reading the stack of immutable files and keeping them open. Whatever happens on the filesystem is not going to be detected on the fly until you try to perform an update.

If the update happens behind JGit's back but not inside the JGit's RefTable code, it won't be noticed until you close the repository and open it again.

In Eclipse / egit its impossible to use repos with reftables. It leeds to corrupted filetables. With refdirectories (and packed-refs) its not a problem because, jgit refresh the cached refs.

[lucamilanesio]

In Eclipse / egit its impossible to use repos with reftables. It leeds to corrupted filetables.

I've actually performed some JGit + CGit tests with a repository using ref-table and ended up with a corrupted filetables as well. Using JGit only doesn't show any issues though. I suspect that's a different issue related to the C implementation of ref-tables.

With refdirectories (and packed-refs) its not a problem because, jgit refresh the cached refs.

The two problems are separate: one thing is to not notice the updates on the filesystem and another is to generate corrupted data.

With Gerrit, for example, the issue isn't really in JGit but a bug in Gerrit where the Repository reference counting was diverging hence keeping the ref-table open indefinitely. Once the underlying Gerrit issue is fixed, I cannot reproduce the stale ref-table anymore.

Maybe the same is possible with EGit?

[lucamilanesio]

@msohn I applied the Gerrit fix on Repository reference-counting and configured the JGit RepositoryCache to expire the repositories after 1 msec with the following:

[core]
	repositoryCacheExpireAfter = 1
	repositoryCacheCleanupDelay = 1

As a result, the JGit ref-table works as expected when used by Gerrit:

• ref-table files are released when repo isn't used
• older ref-table files are released when performing auto-compaction
• JGit can see the changes on the repository when performed outside Gerrit

Bottom line: there could be some similar problem in EGit with the reference counting that should be investigated IMHO.
The code reported in this bug report is actually working as documented in the ref-table specifications for the reader.

@msohn @hanwen @zotttelbart, what do you think?

From Gerrit's perspective, all works without needing changes on JGit's implementation of the ref-table.

[zotttelbart]

I don't know what to expect from jgit. I use jgit in an user application (like git gui) where the user can interact (checkout, merge, reset etc.) with the repository. Some operations (like fetch) are performed in background with jgit. The user can always use gitbash (cgit) parallel.
The application create JGIT at start and close it on exit (what can be hours later). With refdirectories and packed-refs, there was never a problem with this. The lock mechanism of cgit and jgit works to avoid race conditions.

I don't think, its a good idea to close the underlaying jgit instance in the application on idle and reopen the instance if needed.

How EGit does that? Is Egit really close jgit every time, the git operation is ended?

Generally I see some different in naming of reftablefiles. cgit starts with 0x, jgit doesn't.

[zotttelbart]

I tested it in egit with version 7.1.0.

I do

git checkout in cgit

eclipse didn't recognize the new branch. In commit view in egit all files which changed through the checkout are displayed as staged changes and it shows the old branch:

With refdirectories all works expected and egit recognize the changed HEAD ref.