Tool gets stuck when defining glob filepattern on big repo

[kostyay]

Hey
This is my config:

aliases:
  - type: kebabcase
  - type: filepattern
    paths:
      - '**/manifest.yaml'
    patterns:
      - 'featureFlag: FLAG_KEY'
      - 'feature_flag: FLAG_KEY'

Running the tool like this:

ld-find-code-refs -t KEY -d ./ -p default -r REPO --debug -b master -T github -B master --dryRun -C 0 -l 1

Get the following output and the tool gets stuck:

INFO: 2023/08/10 20:15:58 coderefs.go:27: absolute directory path: XXXX
INFO: 2023/08/10 20:15:58 git.go:48: git branch: master
DEBUG: 2023/08/10 20:15:58 git.go:169: identified head sha: XXXXX

Could it be that I got rate limited because I was running it a bunch of times?
There are 317 files matching that pattern in the repo.

Sometimes it works then I get this output:
dry run found 1113 code references across 720 flags and 844 files

However its still not finding everything I'd expect to be found.
Is there a more verbose debugging mode?

[jazanne]

@kostyay unfortunately, it is a known issue that the doublestar pattern does not perform very well for alias searching. We recommend using a more limited file pattern if possible. Given that you have 317 patterns to match in the repo, i understand that could be difficult.

There isn't more verbose debugging at this time for alias searching.

[kostyay]

Ive tried using
xxx/*/*/manifest.yaml and it didnt help either

EDIT:
I realized that patterns must contain a regex group, otherwise they won't work anyway.
This is what its supposed to look like:

    patterns:
      - 'featureFlag: (FLAG_KEY)'
      - 'feature_flag: (FLAG_KEY)'

[kostyay]

@jazanne
You can significantly speed up the initial parsing by caching the results returned by this function:

ld-find-code-refs/aliases/alias.go

Line 79 in 68b13f0

matches, err := doublestar.FilepathGlob(absGlob)

Example:

var filepathGlobCache = map[string][]string{}

func filepathGlob(absGlob string) ([]string, error) {
	cachedFilePaths, ok := filepathGlobCache[absGlob]
	if ok {
		return cachedFilePaths, nil
	}

	matches, err := doublestar.FilepathGlob(absGlob)
	filepathGlobCache[absGlob] = matches

	return matches, err
}