feat(core): map lock file data to external dependencies

[meeroslav]

This PR brings accurate external dependencies mapping to the project graph.

Implementation details

• Parses lock file to LockFileData structure when a lock file is changed
• Stores lock file hash in the nxdeps.json cache for later comparison (add an additional field to nx deps cache object)
• Maps LockFileData to partial ProjectGraph (graph with only externalNodes and their dependencies populated)
• Modifies project graph building to re-use the partial ProjectGraph while building the graph
• Changes print-affected to ignore externalNodes's dependencies
• Modifies the hasher to exclude lockfile file-based hashing and includes hashing of the task target
• Extends hasher's external nodes hashing to traverse nested dependencies
• Fixes several rigid tests

Benefits

• Project Graph contains accurate information about packages and versions used
• Hasher uses an accurate dependency tree
• The package json generator uses a fixed version instead of ranges
• Opens the possibility to fix several issues (e.g. Imports of lazy loaded libraries forbidden not triggered #11178)
• Fixes issues that were blocked by missing dependencies

Performance (calculated on Nx repo) with Yarn:

• Compare with cached lock file: 0.039ms
• Full lock file parsing: ~150ms (due to cache miss)
  • Parse Syml (original function from yarn): ~124ms (point for replacement)

    This is should run in the background daemon so practically it shouldn't be noticeable, but nevertheless should still be improved

  • Hash Lock: 0.7ms
  • Map Packages: 25ms (point for improvement)
  • Detect root packages after mapping: 2-3ms (Yarn and Pnpm only, Npm has clean differentiation built in)

Fixes #12033
Fixes #11874

[nx-cloud-staging]

☁️ Nx Cloud Report

CI is running/has finished running commands for commit 2bd13ad. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this branch

---

✅ Successfully ran 12 targets

• nx affected --target=e2e --base=d91c7ca4ec6c3eb6ffd7e4eb0fda4a61a5b5bd8c --head=2bd13add1591855d33943d42c880deacd614998b --exclude=e2e-storybook,e2e-storybook-angular,e2e-react-native,e2e-detox,e2e-make-angular-cli-faster --parallel=1
• nx affected --target=test --base=d91c7ca4ec6c3eb6ffd7e4eb0fda4a61a5b5bd8c --head=2bd13add1591855d33943d42c880deacd614998b --parallel=1
• nx affected --target=build --base=d91c7ca4ec6c3eb6ffd7e4eb0fda4a61a5b5bd8c --head=2bd13add1591855d33943d42c880deacd614998b --parallel=3
• nx-cloud record -- yarn depcheck
• nx affected --target=lint --base=d91c7ca4ec6c3eb6ffd7e4eb0fda4a61a5b5bd8c --head=2bd13add1591855d33943d42c880deacd614998b --parallel=3
• nx-cloud record -- yarn nx workspace-lint
• nx-cloud record -- yarn check-lock-files
• nx-cloud record -- yarn check-commit
• nx-cloud record -- yarn nx format:check --base=d91c7ca4ec6c3eb6ffd7e4eb0fda4a61a5b5bd8c --head=2bd13add1591855d33943d42c880deacd614998b
• nx-cloud record -- yarn check-imports
• nx-cloud record -- yarn documentation
• nx build typedoc-theme

---

Sent with 💌 from NxCloud.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
nx-dev	✅ Ready (Inspect)	Visit Preview	Oct 5, 2022 at 9:45PM (UTC)

[vsavkin]

it has to be the other way around.

We don't know what your executor depends on. It's not like run-commands is an exception. It's the opposite: nrwl/webpack etc are exceptions

[meeroslav]

It searches for executors in the externalNodes and if found returns the hash of that package and its dependencies. For run-commands we know we shouldn't look for the dependencies of @nrwl/workspace.

Are you saying that everything that isn't @nrwl/* should be considered as hash-all-external-nodes?

[meeroslav]

Done!

[vsavkin]

Given that the number of external dependencies can be large and we traverse them for every task, should we cache the hash for every external dependency?

[meeroslav]

Yes, I think that would be a good idea. To hash it during the parse phase.

[meeroslav]

I added hash caching in the externalNodes

[github-actions]

This pull request has already been merged/closed. If you experience issues related to these changes, please open a new issue referencing this pull request.