Add an option to avoid updating specifi submodules

[libratiger]

it seems that there is no way to ignore the submodule dir, is there any advice?

[illicitonion]

What are you trying to achieve specifically?

There are a few different things "ignoring" could mean; off the top of my head:

1. Being ok with the dir being there (i.e. ignoring it for "repo cleanness" checking)
2. Wanting to pretend there were no changes there (i.e. do the whole graph analysis, but cut off any edges that go into a submodule)
3. Wanting to not return any results in there (i.e. do the whole graph analysis, but remove any nodes in that submodule from the output)
4. Just not even checking out the submodule at all (presumably as a speed optimisation, but really trying to achieve 2 or 3).

Can you explain your use-case a bit more? Why are you trying to ignore the submodule?

[libratiger]

1. because when I do not ignore the submodules, the cli report me that the commitID can not match.
2. because the step1 and I do not care much about the submodule, so i just want to ignore it.

[illicitonion]

• because when I do not ignore the submodules, the cli report me that the commitID can not match.

Interesting, I don't expect that to be the case - do you have a repro you can share?

[libratiger]

• because when I do not ignore the submodules, the cli report me that the commitID can not match.

Interesting, I don't expect that to be the case - do you have a repro you can share?

sorry it is a repo for my company, but you can create this very simple, but add a submodule in a git repo, and the add some commits in the repo

[illicitonion]

but you can create this very simple, but add a submodule in a git repo, and the add some commits in the repo

We have tests which show that submodules work in at least some cases:

target-determinator/tests/integration/java/com/github/bazel_contrib/target_determinator/integration/Tests.java

Lines 396 to 446 in d4ef4cc

	@Test
	public void succeedForUncleanSubmodule() throws Exception {
	gitCheckout(Commits.SUBMODULE_CHANGE_DIRECTORY);
	Files.createFile(testDir.resolve("demo-submodule-2").resolve("untracked-file"));
	doTest(Commits.SUBMODULE_ADD_DEPENDENT_ON_SIMPLE_JAVA_LIBRARY,
	Commits.SUBMODULE_CHANGE_DIRECTORY,
	Set.of("//demo-submodule-2:submodule_simple"));
	}
	@Test
	public void addTrivialSubmodule() throws Exception {
	doTest(Commits.SIMPLE_JAVA_LIBRARY_TARGETS, Commits.SUBMODULE_ADD_TRIVIAL_SUBMODULE, Set.of());
	assertThat(
	"The submodule should now be present with its README.md but isn't",
	Files.exists(testDir.resolve("demo-submodule").resolve("README.md")));
	}
	@Test
	public void addDependentTargetInSubmodule() throws Exception {
	doTest(
	Commits.SUBMODULE_ADD_TRIVIAL_SUBMODULE,
	Commits.SUBMODULE_ADD_DEPENDENT_ON_SIMPLE_JAVA_LIBRARY,
	Set.of("//demo-submodule:submodule_simple"));
	}
	@Test
	public void changeSubmodulePath() throws Exception {
	doTest(
	Commits.SUBMODULE_ADD_DEPENDENT_ON_SIMPLE_JAVA_LIBRARY,
	Commits.SUBMODULE_CHANGE_DIRECTORY,
	Set.of("//demo-submodule-2:submodule_simple"));
	assertThat(
	"The old submodule directory should not exist anymore",
	not(Files.exists(testDir.resolve("demo-submodule"))));
	assertThat(
	"The moved submodule should now be present with its README.md but isn't",
	Files.exists(testDir.resolve("demo-submodule-2").resolve("README.md")));
	}
	@Test
	public void deleteSubmodule() throws Exception {
	doTest(Commits.SUBMODULE_CHANGE_DIRECTORY, Commits.SUBMODULE_DELETE_SUBMODULE, Set.of());
	assertThat(
	"The old submodule directory should not exist anymore",
	not(Files.exists(testDir.resolve("demo-submodule-2"))));
	}

If you could give more detailed instructions (ideally a simple git repository, or a list of commands to run to create one), I'd for sure be interested in working out what's going wrong here.

[libratiger]

the .gitmodules file like the following

[submodule "xx"]
  path = xx
  url = http://aaa/xx.git
  branch = master
[submodule "yy"]
  path = yy
  url = http://aaa/yy.git
  branch = master

[sitaktif]

the cli report me that the commitID can not match

I don't understand what you mean by that. Do you mind pasting the full logs of an invocation (including the command line you used to invoke target-determinator), lightly redacting anything (e.g. directory or repo names) that you cannot share?

Can you reproduce the issue yourself on a minimal code repository (ideally on Github for convenience) and share that with us?

Currently, we don't have enough information to understand what is different between our test cases and your failure case.

[libratiger]

driver b0e36adca2fa0eb6ffe2b2eacdcd669b50dd7f8e
2022/07/07 10:45:46 Discovering affected targets
2022/07/07 10:45:46 Processing revision 'before' (sha: b0e36adca2fa0eb6ffe2b2eacdcd669b50dd7f8e)
2022/07/07 10:48:12 failed to process change: failed to load metadata at revision 'before' (sha: b0e36adca2fa0eb6ffe2b2eacdcd669b50dd7f8e): failed to checkout revision 'before' (sha: b0e36adca2fa0eb6ffe2b2eacdcd669b50dd7f8e) in /Users/userA/dailywork/exp/release: failed to update submodules during checkout revision 'before' (sha: b0e36adca2fa0eb6ffe2b2eacdcd669b50dd7f8e): exit status 128. Output: Submodule 'product_fl' (http://gitlab.company.local/algorithm/product_fl.git) registered for path 'algorithm/product_fl'
Submodule 'product_library' (http://gitlab.company.local/algorithm/product_library.git) registered for path 'algorithm/product_library'
Submodule 'cli' (http://gitlab.company.local/debug/product_cli.git) registered for path 'cli'
Submodule 'engineS' (http://gitlab.company.local/product_engine/product.git) registered for path 'engineS'
Submodule 'idl' (http://gitlab.company.local/re/idl.git) registered for path 'idl'
Submodule 'sql' (http://gitlab.company.local/sql/go-db.git) registered for path 'sql'
Submodule 'chaos_test' (http://gitlab.company.local/qa/chaos_test.git) registered for path 'tests/chaos_test'
Cloning into '/Users/userA/dailywork/exp/release/algorithm/product_fl'...
Cloning into '/Users/userA/dailywork/exp/release/algorithm/product_library'...
Cloning into '/Users/userA/dailywork/exp/release/cli'...
Cloning into '/Users/userA/dailywork/exp/release/engineS'...
Cloning into '/Users/userA/dailywork/exp/release/idl'...
Cloning into '/Users/userA/dailywork/exp/release/sql'...
Cloning into '/Users/userA/dailywork/exp/release/tests/chaos_test'...
Submodule path 'algorithm/product_fl': checked out 'f3bad31eb4eaf81b8458c962573d91063bde7355'
Submodule path 'algorithm/product_library': checked out 'ea578f0882b495307f1b689dea0b7dae65ab6170'
Submodule path 'cli': checked out 'aa28f8190ec91d6b049aea8cc2f170e9ea74ed13'
Submodule path 'engineS': checked out '01eca6043b8fbd76eaea60d82cb13b29ecd675c3'
fatal: remote error: upload-pack: not our ref 2d7f5d7727c04197d2ad0f173e242c4705cbe84f
fatal: Fetched in submodule path 'idl', but it did not contain 2d7f5d7727c04197d2ad0f173e242c4705cbe84f. Direct fetching of that commit failed.

[libratiger]

@sitaktif @illicitonion I have post the error in the above comments

[libratiger]

I used the laster released driver version

[sitaktif]

fatal: Fetched in submodule path 'idl', but it did not contain 2d7f5d7727c04197d2ad0f173e242c4705cbe84f. Direct fetching of that commit failed.

I see, at the "before" revision, the idl submodule points to a commit that doesn't exist. Please be aware that this essentially means that the BEFORE revision is not valid in your main repo. However, there is something to be said about the fact that if the submodule is at a path that is specified with the --ignore-file argument, we should not update the submodule at all. I'll amend the title to reflect this 👍

[sitaktif]

There is concern about overloading the meaning of --ignore-file. One possibility is to add an argument specifically to avoid checking out submodules.