chore: Use Nx [proposed] #135
Conversation
Peeja
force-pushed
the
nx
branch
3 times, most recently
from
3480ba2 to
d35844c
Compare
There was a problem hiding this comment.
My first response is:
This all looks great, with the caveat that I'm not in the weeds these days on the latest monorepo tools. But Nx seems to have good sentiment around web.
I'm not sure I'd go all in the Nx Cloud thing yet. It doesn't look like Nx Cloud is included for now -- am I reading that correctly? I'm somewhat loathe to add more services into our builds beyond Github, though I can't tell if maybe there are replacements for Github there? And then I also could be convinced if we could drop seed.run when we merge upload-service-infra.
But beyond that this all seems reasonable. Questions are just that.
Oh one other bit: once w3ui is merged, is it safe to then start having other devs bring in other repos? Or can we jump into that as soon as this is merged?
| "@docusaurus/core": "^3.0.0", | ||
| "@docusaurus/preset-classic": "^3.0.0", | ||
| "@nx/js": "^20.3.2", |
There was a problem hiding this comment.
just curious what's the difference between these two NX packages?
| @@ -184,7 +184,7 @@ export const createService = (context) => ({ | |||
| ...createAdminService(context), | |||
| // @ts-expect-error `uploadTable` items now have a `cause` field. This does | |||
| // not matter since `admin/store/inspect` handler does not use this table. | |||
| store: createLegacyAdminService(context).store | |||
| store: createLegacyAdminService(context).store, | |||
There was a problem hiding this comment.
??? I thought we were set to no trailing commas?
There was a problem hiding this comment.
No, we use "trailingComma": "es5", which is trailing commas anywhere that ES5 won't complain about them (whereas modern ES is more permissive).
| @@ -32,22 +33,5 @@ | |||
| "skipLibCheck": true, | |||
| "stripInternal": true, | |||
| "resolveJsonModule": true | |||
| }, | |||
| "typedocOptions": { | |||
There was a problem hiding this comment.
currious why this isn't neccesary?
There was a problem hiding this comment.
It's not necessary because we're not using TypeDoc anymore. But I'm not fully ripping that out right now, and I don't remember now why I felt it was helpful to get rid of this. Something about the way the TS configs inherit from each other? I probably could add it back, but we're just going to rip it out again shortly when we completely remove TypeDoc and Docusaurus as unused.
Why bring in code from a language that already has built in support for monorepo patterns from the jump? :P #unneccesarySnark |
|
A discussion for another time, Hannah. 😛 Seriously, though, if anything, it would be so we can have dependencies run across the two languages. Golang's native monorepo support is roughly equivalent to |
Yeah, that's right. I have added a key for it, but that just means that at the end of a local run you get a URL with prettier output. It's not actually even associated with the project right now, they just have a free service that does that. |
What's going on here
As I encountered friction during the monorepo import, I started exploring a few options. Nx quickly stood out as a way to untangle a lot of issues, mainly dealing with task dependencies across multiple packages. So I decided to run with it for a bit and see if it kept panning out. It did, and here's the result.
We don't have to keep Nx—a fair bit of the work here was just clarifying and standardizing the npm scripts, which is important with or without Nx—but I'm going to make the case for it in this PR.
What's in this PR
This PR adds Nx to the existing monorepo. It doesn't include the
w3uiimport, which will be in a separate PR to keep things clearer. But I've used that import to drive some of the decisions in this PR.This PR does not yet include the ability to release. That should be added before merging, but I want to get this in front of eyeballs ASAP.
What to look for
Here are some things to try:
pnpm install, and trypnpm nx run-many -t test, which will run thetesttask in every project in the repo. It should be passing.Why Nx?
Builds on pnpm: Nx builds on top of existing monorepo tools. It doesn't replace pnpm—it uses pnpm's notion of the workspace with packages and scripts.
Task dependencies: A lot of tasks depend on each other. Before
attwlooks over the (built) types for problems, a package has tobuild. Before one package canbuild, all the packages it depends on have tobuild, too. Nx manages the dependency graph across the entire monorepo and runs everything in order, as needed.A great task runner: Instead of just dumping all the output into a terminal together like pnpm mostly does, Nx has output that handles lots of projects and tasks better. It also provides HTML output that can make it even easier to look through the results.
Caching: Each task can be cached (if appropriate) so it doesn't have to re-run if its input files haven't changed, which makes running large DAGs of tasks over and over again during development a lot faster.
Only test affected projects: The
affectedcommand can run a task (such astest) for all of and only the projects that changes in the current branch have touched.Flaky test detection and handling: I haven't tried this myself yet, but it sounds pretty great.
File-based release versioning: Just like Changesets.
Nx Cloud
Nx also offers a product, called Nx Cloud. A bunch of it is completely free, and the paid portions include a pretty generous free tier.
Hosted run results: As far as I can tell, this is just entirely free. At the completion of a run, whether local or in CI, you get a URL to a page that displays the results of your tasks. You get to use this without even signing up for anything.
Remote caching (Nx Replay): Cached task results are stored in Nx Cloud. That means that even if you haven't run and cached a task yet, if someone else on the team has, you can simply use their cached result. For instance, when you check out a new branch and build it, that build may already have been cached by the person who was working on it, making your initial build super fast.
Distributed task execution (Nx Agents): Rather than run CI tasks on GH Actions itself, we can send the tasks off to Nx Agents. These can efficiently divide the individual tasks to run in parallel. That's more effective than trying to manually split the tasks and projects across separate GH Actions workflows in advance.
And what's great is that if we try the free tier and run out of free credits, it simply turns off the Nx Cloud features and reverts to entirely local runs. These are just enhancements layered on top of Nx. And, if it does fit our project and the value proposition is correct, the cost of the Nx Agents should be less than the cost it saves us in GH Actions minutes, not to mention developer time. (A minute on an Nx Cloud Linux Medium costs $0.00055/cr × 10cr = $0.0055; a minute on a GitHub Linux 2-Core costs $0.008.)
Bonus
If we really like it, we might even consider bringing our Go code in here too! But that's a discussion for another time.