Reduce friction needed for custom software distributions based on Homebrew

[jafd]

I have a peculiar use case which, as of now, requires a lot of menial work that I believe can be reduced somehow.

There are a few pieces of in-house software distributed within the company. They pull in quite a bit of open-source dependencies, which to date are best managed with Homebrew. The installation I'm making is a Homebrew installation which lives in a prefix of its own and is completely self-contained — that is, it doesn't interfere with a regular installation the user might have.

To scale such an installation, building bottles is necessary. And this is the moment where an ungodly amount of work comes in.

There are formulae which are non-relocatable. For them, I make my own tap. The bottles containing these formulae are being downloaded from an alternative locations, and are installed into /opt/my-stuff. The most notable of these are pkg-config, [email protected], gettext, and [email protected]. But then, once I build these bottles, I'd like other formulae which use those as dependencies, to latch onto my builds and not those from core... which is impossible unless I pull them into my tap and correct their dependencies. And then the formulae which depend on these, and so it goes. If I miss one dependency, it will start to build the whole chain from source, starting with pkg-config, and then create problems on the way because it's going to bump into my-stuff/pkg-config.

Note that above also applies to relocatable formulae, which otherwise would need no rebuilding!

At the end of the day, I ended up having to bottle and vendor into my tap around 50 formulae, including Rust compiler. Managing updates to those is far from being fun.

Also, there seems to exist a bug in how Homebrew makes bottles, because custom prefixes don't get into the bottles themselves, and then Homebrew denies to pour such bottles, claiming they have been built for the default prefix (/usr/local). I mitigated this by patching Homebrew so that its default prefix can be changed using an environment variable.

Still I think that a lot of work could be avoided if Homebrew supported custom prefixes as first-class citizens, and also enabled some kind of overrides which would give an alternative view of formulae — so that instead of pulling in 50 formulae, only a few would need it.

Mildly related:

• homebrew tap override flag legacy-homebrew#24238
• Multiple cellars in a single Homebrew installation (i.e. Homebrew "virtual environments") brew#11787

[SMillerDev]

If you ask me your main issue: You're using homebrew as a dependency manager, while it is not.

The installation I'm making is a Homebrew installation which lives in a prefix of its own and is completely self-contained

I think this where all your issues begin. There is a reason https://docs.brew.sh/Installation#untar-anywhere says "One of the reasons Homebrew just works relative to the competition is because we recommend installing here. Pick another prefix at your peril!".

The right solution might be to manage the required packages with homebrew, but having more than one homebrew install is a pain. Even for the one use case where we support it (arm with Rosetta) there is a lot of custom code involved to make it work.

[jafd]

So do/would you also install RPM packages in a separate location for Linux users?

I can create a new RPM database, change just a couple of top-level macros, and rebuild my specs otherwise unchanged. It's a lot less human work than Homebrew currently requires for the same results (the machine still has to rebuild them all, but it's a secondary concern).

[SMillerDev]

I'm sure you can, and you have proven that you can do similar things with brew, but do you do this for Linux? Because I still maintain my earlier point "I don't think this is a supported case for other package managers". The fact that you have to rebuild every RPM is a testament to that IYAM.

[jafd]

As a matter of fact, I do. I backport a lot of newer software (like the whole KDE) onto the stable Fedora release I'm using, so I can use the stable base platform and enjoy the newest Plasma fixes, for example. However, while it takes a long time to build, I'm doing it in two commands written once. I don't have to jiggle dependencies in all the specs.

Linux, however, can sidestep the whole issue since there are at least five flavors of containers which all have a very low overhead to choose from: docker, lxc, systemd-nspawn, crun, plain chroot if you want to. Unfortunately, macOS doesn't have them (or has Docker which carries a huge overhead of a whole Linux VM), so I'm doing the kind of "isolation" I can.

[danielnachun]

I personally think that Homebrew/brew#10846, with enough effort and consideration, will eventually make all or almost all binaries relocatable. The key idea there is that you have to build the binaries with a very long prefix (Anaconda uses 255 characters), and then replace that prefix with the target prefix, with null padding to keep the actual string the same length.

This strategy is highly effective in Anaconda so I believe it will work for the vast majority of use cases. I will be actively hunting for corner cases where this does not work once the code is implemented, and I do hope that it will eventually make uses cases like yours easier.

The key caveat in all of this that you'd still have to use brew to install the software, even if in an alternate cellar. I've been asked a few times if this patching strategy could be used to deliver relocatable binaries built with brew with need brew itself to do the installation. Unfortunately that's not possible because the patching code would be part of brew (just as it is in Anaconda).

Once we have this working consistently in Homebrew, I'm thinking about writing a small portable tool in something like Rust which could be redistributed with standalone software packages and patch the prefixes that wouldn't require Homebrew or Anaconda. As you've noted, containers aren't always a great option on macOS, and even on Linux they are not without their drawbacks.

[MikeMcQuaid]

I personally think that Homebrew/brew#10846, with enough effort and consideration, will eventually make all or almost all binaries relocatable. The key idea there is that you have to build the binaries with a very long prefix (Anaconda uses 255 characters), and then replace that prefix with the target prefix, with null padding to keep the actual string the same length.

I think it may make some (in the short term) or many (in the long term) relocatable. I do not think it will handle every case because we have a much wider tail of software than Anaconda and we have quite an unusual packaging approach (and unwillingness to indefinitely carry patches).

Once we have this working consistently in Homebrew, I'm thinking about writing a small portable tool in something like Rust which could be redistributed with standalone software packages and patch the prefixes that wouldn't require Homebrew or Anaconda. As you've noted, containers aren't always a great option on macOS, and even on Linux they are not without their drawbacks.

As long as this effort is not supported by Homebrew: I'm all for it. If it is/was: this tool should be part of Homebrew itself.

[carlocab]

Also, there seems to exist a bug in how Homebrew makes bottles, because custom prefixes don't get into the bottles themselves, and then Homebrew denies to pour such bottles, claiming they have been built for the default prefix (/usr/local).

Could you explain how to reproduce this? I tried to, and found that brew was, in fact, recording build prefixes correctly. See Homebrew/brew#11787 (comment).

[MikeMcQuaid]

It seems like bottle blocks should still allow you to specify a prefix, or assume one based on the specified cellar. We'll review a pull request that implements this.

They are assumed based on the cellar. I don't want to allow customisation of prefix again because it was broken for a long time and no-one noticed (hence wasn't widely used).

However, I can't see why the cellar must be a subdirectory of HOMEBREW_PREFIX (see also the original issue pondering multiple cellars, I can't see why not).

Some software assumes this and breaks without. It's not something we're likely to support again as we don't have the ability to test it.

[jafd]

Oh, @MikeMcQuaid, good to see you here!

Current Homebrew is not deriving bottle's prefix from its Cellar, unfortunately, as you can see above. Maybe someone needs to produce a PR that does it? Right now it just assigns Homebrew::DEFAULT_PREFIX, there's no way to modify it later.

Also, the rest of the code doesn't derive prefixes from the cellar (it's rather the other way around), although I'd rather like it more if it did.

[MikeMcQuaid]

Current Homebrew is not deriving bottle's prefix from its Cellar, unfortunately, as you can see above. Maybe someone needs to produce a PR that does it? Right now it just assigns Homebrew::DEFAULT_PREFIX, there's no way to modify it later.

Gotcha. Yeh, we'd accept a PR to fix this for sure 👍🏻. Rather than modifying it I think it's worth just having some logic like "if cellar is set to a non-default value, prefix should be expected to be the parent of the cellar.

[jafd]

Some software assumes this and breaks without. It's not something we're likely to support again as we don't have the ability to test it.

I think this is the most major blocker for the original "let us just have many cellars!" issue. Like, someone implements it, it's going to work for 85% of cases, and for the remaining 15% it will explode right in your face.

[MikeMcQuaid]

I think this is the most major blocker for the original "let us just have many cellars!" issue. Like, someone implements it, it's going to work for 85% of cases, and for the remaining 15% it will explode right in your face.

Indeed. My personal philosophy of how good software should be means "don't try to support this" because I think this sort of dice-rolling is fundamentally user-hostile.

[MikeMcQuaid]

The bottles containing these formulae are being downloaded from an alternative locations, and are installed into /opt/my-stuff. The most notable of these are pkg-config, [email protected], gettext, and [email protected]. But then, once I build these bottles, I'd like other formulae which use those as dependencies, to latch onto my builds and not those from core... which is impossible unless I pull them into my tap and correct their dependencies.

This is intentional, sorry. We don't want it to be easy to swap out e.g. openssl for hopefully obvious security reasons.

At the end of the day, I ended up having to bottle and vendor into my tap around 50 formulae, including Rust compiler. Managing updates to those is far from being fun.

This would be what I'd recommend in this situation. I appreciate it's not fun but, yes, this is probably the best bet if you want to built binaries for a non-default cellar/prefix.

[jafd]

We don't want it to be easy to swap out e.g. openssl for hopefully obvious security reasons.

Don't underestimate the ingenuity of fools ;)

Back in the day, we had people reading the so called "mod files" for phpBB which were in essence manually-applied diffs and followed them to a tee, I suspect there could be mods that introduced intentional holes with people applying them being none the wiser. A lot of people running sites didn't really know how the LAMP stack worked, they just copied files into the FTP directory.

Later, we had this trend of tutorials that involved installing random Docker images off the internet. I don't know how many people ended up mining bitcoins for some unknown dudes, just because they followed a tutorial.

Same goes for third-party taps: they can sure swap OpenSSL, or introduce backdoors into less-important-sounding packages (think readline), but the general wisdom should be that you better verify the supplier and don't just blindly include poisoned taps. Still won't work for many people :(

[jafd]

I appreciate it's not fun but, yes, this is probably the best bet if you want to built binaries for a non-default cellar/prefix.

I can understand this. However, there's a reason I named the discussion "reducing friction" and not "1st party support" :). There can be small changes made (like the one about deriving prefix from the cellar), or tools produced, that make the work easier if you need it done. I just want to learn more about the possibilities.

[MikeMcQuaid]

There can be small changes made (like the one about deriving prefix from the cellar), or tools produced, that make the work easier if you need it done.

👍🏻 all game for this.