npm packages Docker integration

[bazaglia]

I'm wondering what are the current limitations to build a Docker image out of targets created using rules_js. I'm able to create a tar from the ts_project rule or swc from rules_swc, but I'm struggling with bundling the npm packages.

I've tried to use the container_layer rule to create a npm_deps layer that is copied to the /node_modules of my image:

container_layer(
    name = "npm_deps",
    data_path = "node_modules",
    files = [
        "@npm//axios",
        "@npm//@types/node",
    ],
)

The problem is, indirect dependencies end up exclusively inside .aspect_rules_js (follow-redirects is an indirect dependency of axios):

/node_modules # tree -L 2 -a
.
├── .aspect_rules_js
│   ├── @[email protected]
│   ├── [email protected]
│   └── [email protected]
├── @types
│   └── node
└── axios
    ├── CHANGELOG.md
    ├── LICENSE
    ├── README.md
    ├── SECURITY.md
    ├── UPGRADE_GUIDE.md
    ├── dist
    ├── index.d.ts
    ├── index.js
    ├── lib
    ├── package.json
    ├── tsconfig.json
    └── tslint.json

In the following example, lodash.camelcase is an indirect dependency of an external npm package. When running my Docker container, it fails with:

node:internal/modules/cjs/loader:936
  throw err;
  ^

Error: Cannot find module 'lodash.camelcase'

That makes sense since one of the npm packages tried to import it but it isn't linked (but is available inside .aspect_rules_js). Is there a way to get the indirect dependencies also linked to the node_modules?

As a side note, symlinks seem to become the whole directory copied, so there are three axios and two follow-redirects (I'm limiting the tree to 2 subdirectories, but inside the node_modules of packages in .aspect_rules_js the duplication happens).

[alexeagle]

We haven't tried this at all so far. Maybe @thesayyn has some ideas since he has worked on both sides of this.

[thesayyn]

Hmm, I suspect a few things that could go wrong here;

1- container_layer mishandles TreeArtifact symlinks. (this is more likely given the side note)
2- you are trying to import something you haven't declared a dependency on.
3- lodash.camelcase seems to be missing from the virtual store altogether hence could be improper handling of transitive depsets.

I'd like to take a look if you could provide a repro.

[bazaglia]

Hey @thesayyn! Here's the repro. My understanding is that the symlinks mishandling starts in the pkg_tar already. More info in the README: https://github.com/bazaglia/rules_js_repro

[thesayyn]

Thanks! I'll take a look

[bazaglia]

I made a comment on an existing pkg_tar issue about this limitation: bazelbuild/rules_pkg#115 (comment)

[alexeagle]

Yeah I think that's the principled answer, do we know how hard that is to contribute?

[thesayyn]

#249 fixes this issue. if you want to have more fine-grained layers eg node_modules into their own layer, one has to modify the expand_runfiles function to yield more than one tar.