Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use virtual package mechanism for BoringSSL package #686

Merged
merged 3 commits into from
Aug 12, 2020

Conversation

ilammy
Copy link
Collaborator

@ilammy ilammy commented Jul 28, 2020

Buildbot started failing to build development packages after PRs #682 and #683 have been merged into master. This PR resolves the issues that cause the failures, and improves BoringSSL packaging a bit.

Why alternative packages are not good

Alternative package dependencies are great, but they have several issues for the use case we need. First of all, 'boolean package dependencies' are not supported on CentOS 7 which we need to support. RPM >= 4.13 is required to build and install such packages correctly. Unfortunately, CentOS 7 uses RPM 4.11 so we're out of luck here.

The second issue is that if some other package previously depended on, say, libthemis (>= 0.13), the new libthemis-boringssl cannot satisfy that dependency. It is able to, but the package manager will not allow it to be installed for it is not libthemis.

What is the alternative to them? (heh)

Alternative (boolean) package dependencies may work for our packages since we make them to, but they won't work for other packages.

In fact, alternative dependencies are not the best solution here. We would like the BoringSSL flavor to be able to stand in for the OpenSSL flavor because it provides the same API and ABI. dpkg and RPM have a mechanism for this – the Provides field in the package specs.

Provides, Conflicts, and Replaces

If BoringSSL package provides libthemis then it can satisfy the need for libthemis of any other package, like libthemis-jni, for example. Similarly, libthemis-boringssl-dev can stand in for libthemis-dev.

Note, however, that libthemis and libthemis-boringssl are still mutually exclusive so they need to conflict with each other. In fact, all flavors of libthemis conflict with each other and the same is true for the development packages as well.

In addition to that, if we declare them as replacing one other, the package manager will be able to tell that it can remove the old conflicting package and install an alternative to it, without bothering the user.

Caveats

FPM discards versioning information from --provides arguments. While RPM does not seem to have any issues with that, dpkg does have issues which break conflict resolution if the provided versions are not specified. We have to insert DEB fields directly to fix this. Otherwise the users will see the following errors

dpkg: dependency problems prevent configuration of libthemis-jni:
 libthemis-jni depends on libthemis (>= 0.13.0+xenial); however:
  Package libthemis is not installed.
  Version of libthemis on system, provided by libthemis-boringssl:amd64, is <none>.

dpkg: error processing package libthemis-jni (--install):
 dependency problems - leaving unconfigured

when installing packages that depend on libthemis when it is replaced by libthemis-boringssl.

Remaining issues

Although packages can replace each other, there are still some issues with that. For example, on Debian I am not able to cleanly replace both libthemis and libthemis-dev simultaneously with libthemis-boringssl and libthemis-boringssl-dev, while it is still possible to do it one package at a time, if done in correct order. RPM on the other hand is fine.

Also, it is currently possible to install libthemis-boringssl and libthemis-dev simultaneously without conflicts. While technically it still works, this may be surprising for the users, especially if they try building something with static libraries. It would be nice to disallow such installation in some way.

OpenSSL dependency fix for Ubuntu 16.04

On some systems (e.g., Ubuntu 16.04) the libssl-dev package depends not only on libsslX.Y.Z, but on other packages too (zlib1g-dev in the case of Ubuntu). The current spell for finding out the name of the OpenSSL library package does not take that into account, and this leads to incorrect command line passed to FPM.

Improve the spell to react only on libssl packages, which there should be only one in the dependency list.

Checklist

  • Change is covered by automated tests (only on Buildbot)
  • The coding guidelines are followed
  • Changelog is updated

ilammy added 2 commits July 28, 2020 15:46
Why alternative packages are not good
-------------------------------------

Alternative package dependencies are great, but they have several issues
for the use case we need. First of all, 'boolean package dependencies'
are not supported on CentOS 7 which we need to support. RPM >= 4.13 is
required to build *and* install such packages correctly [1].
Unfortunately, CentOS 7 uses RPM 4.11 so we're out of luck here.

[1]: https://rpm.org/user_doc/boolean_dependencies.html

The second issue is that if some other package previously depended on,
say, "libthemis (>= 0.13)", the new "libthemis-boringssl" cannot satisfy
that depenency. It is able to, but the package manager will not allow it
to be installed for it is not "libthemis".

What is the alternative to them? (heh)
--------------------------------

Alternative (boolean) package dependencies may work for our packages
since we make them to, but they won't work for other packages.

In fact, alternative dependencies are not the best solution here. We
would like the BoringSSL flavor to be able to stand in for the OpenSSL
flavor because it provides the same API and ABI. dpkg and RPM have a
mechanism for this -- the "Provides" field in the package specs.

"Provides", "Conflicts", and "Replaces"
---------------------------------------

If BoringSSL package *provides* "libthemis" then it can satify the need
for "libthemis" of any other package, like "libthemis-jni", for example.
Similarly, "libthemis-boringssl-dev" provides "libthemis-dev".

Note, however, that "libthemis" and "libthemis-boringssl" are still
mutually exclusive so they need to conflict with each other. In fact,
all flavors of "libthemis" conflict with each other and the same is true
for the development packages as well.

In addition to that, if we declare them as *replacing* one other, the
package manager will be able to tell that it can remove the old
conflicting package and install an alternative to it, without bothering
the user.

Caveats
-------

FPM discards versioning information from "--provides" arguments. While
RPM does not seem to have any issues with that, dpkg does have issues
which break conflict resolution if the provided versions are not
specificed. We have to insert DEB fields directly to fix this.
Otherwise the users will see the following errors

    dpkg: dependency problems prevent configuration of libthemis-jni:
     libthemis-jni depends on libthemis (>= 0.13.0+xenial); however:
      Package libthemis is not installed.
      Version of libthemis on system, provided by libthemis-boringssl:amd64, is <none>.

    dpkg: error processing package libthemis-jni (--install):
     dependency problems - leaving unconfigured

when installing packages that depend on "libthemis" when it is replaced
by "libthemis-boringssl".

Remaining issues
----------------

Although packages can replace each other, there are still some issues
with that. For example, on Debian I am not able to cleanly replace both
"libthemis" and "libthemis-dev" simultaneously with "libthemis-boringssl"
and "libthemis-boringssl-dev", while it is still to do it one package
at a time, if done in correct order. RPM on the other had is fine.

Also, it is currently possible to install "libthemis-boringssl" and
"libthemis-dev" simultaneously without conflicts. While technically it
still works, this may be surprising for the users, especially if they
try building something with static libraries. It would be nice to
disallow such installation in some way.
On some systems (e.g., Ubuntu 16.04) the "libssl-dev" package depends
not only on "libsslX.Y.Z", but on other packages too ("zlib1g" in the
casse of Ubuntu). The current spell for finding out the name of the
OpenSSL library package does not take that into account, and this leads
to incorrect command line passed to FPM.

Improve the spell to react only on "libssl" packages, which there should
be only one in the dependency list.
@ilammy ilammy added core Themis Core written in C, its packages infrastructure Automated building and packaging O-Linux 🐧 Operating system: Linux installation Installation of Themis core and wrapper packages labels Jul 28, 2020
@ilammy ilammy requested review from vixentael, Lagovas and shadinua July 28, 2020 12:54
@ilammy
Copy link
Collaborator Author

ilammy commented Jul 28, 2020

Here's a diagram of the new package interdependencies to make it more clear what the hell does this PR do:

package interdependencies

libthemis-dev works only with libthemis of the same version (ditto for -boringssl), but libthemis-jni can use either libthemis or libthemis-boringssl of at least the version is was built with. Finally, libthemis-boringssl conflicts with libthemis so it cannot be installed along with it (or vice versa).

I thought about outlining how this works with later upgrades, but it became so complex that I scrapped the idea.

Copy link
Contributor

@vixentael vixentael left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shadinua is the best reviewer here

Copy link
Contributor

@shadinua shadinua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the lengthy review, but it was necessary to analyze and test the behavior in different environments.
Great work, thanks!

@ilammy ilammy merged commit 960f000 into cossacklabs:master Aug 12, 2020
@ilammy ilammy deleted the package-provides branch August 12, 2020 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Themis Core written in C, its packages infrastructure Automated building and packaging installation Installation of Themis core and wrapper packages O-Linux 🐧 Operating system: Linux
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants