Document how to build portable, cross-platform binaries

[springmeyer]

Creating C++ binaries portable to a given platform (osx, linux, windows) or platform version (ubuntu precise or trusty) is absolutely feasible but a poorly documented topic. Often our goal at Mapbox is to support the latest C++11 and C++14 features but maintain code portable to older operating systems like Windows 7, OS X 10.8, and Ubuntu Precise, or RHEL 6/7. This is also feasible. For example its possible to run C++14 binaries on AWS lambda (built on ubuntu precise) even though you can't easily compile them on AWS Lambda.

We should document how Mapbox writes and distributes portable C?C++ code to make clear when this approach is advantageous and when it is not. And to ensure the techniques are robust and replicable.

This issue is to track:

• getting this all documented here
• Moving the content from https://github.com/mapbox/node-pre-gyp/wiki/External-libraries to this future doc
• Moving binary portability FAQ from https://github.com/mapbox/node-pre-gyp/wiki/FAQ#frequently-ask-questions-about-node-pre-gyp to here

[springmeyer]

Initial brainstorming notes:

OS X

• Using the latest version of XCode for clang++ (or latest clang++ via https://github.com/mapbox/mason/)
• Only supporting OS X 10.8 and above that ship with libc++
• Using static linking or shared libs with relative @loader_path
• Consistently compiling all binaries with the -mmacosx-version-min flag.

Linux

• Using a non-buggy version of clang++ that can produce binaries compatible with gcc
• Using static linking or shared libs with relative @rpath
• Ensuring your binaries do not depend in a GLIBC or GLIBCXX version greater than your target linux platform
• For binaries targeting libstdc++:
  • Upgrading libstdc++ to support C++11 or c++14 features but being careful to install the oldest possible g++ headers that your code can compile with to avoid unintended GLIBCXX symbol dependence
  • Using a consistent -D_GLIBCXX_USE_CXX11_ABI flag when targeting C++11 or C++14 and versioning binaries on this if you need to support both.
• For binaries targeting libc++:
  • Statically linking libc++

Windows

• Using
• Linking with /MT or using /MD and ensuring that the C++ redistributable is installed
• Placing any .dll beside the executable so they are found relatively
• Setting PATH dynamically to prefer local .dll over global if there might be conflicts

C++ Language specific portability limitations

• handling different endianess when working with binary data
• cross-platform filesystem io: boost::filesystem, std::experimental::filesystem, libuv

[springmeyer]

Working on https://github.com/springmeyer/glibcxx-symbol-versioning to get a good handle on how symbol versioning works on linux and what triggers your binaries to require certain GLIBCXX symbols. /cc @mapsam @GretaCB

[apendleton]

@springmeyer out of curiosity do you see writing code that handles different endianness properly to be part of this discussion, or a different issue?

[springmeyer]

@apendleton absolutely, let's document anything we've hit as a real-world issue, even if only partially solved. Protozero, for example, has basic support for handling endianess. We should get @joto to document the design and status of that (https://github.com/mapbox/protozero/blob/f5ea37d5b431024de6f87a95c9b0226b850d2ef4/include/protozero/config.hpp#L21-L47)

[springmeyer]

@apendleton I've added a section to the outline above called C++ Language specific portability limitations. Please add anything else you think of that we should document our experience with.

[joto]

@springmeyer There is no "design" in the endianness handling of protozero. I checked what boost was doing and then built the simplest thing that made everything work on the test cases we had (specifically Debian builds on unusual architectures in their build cluster) without using boost.

Generally I recommend creating Debian packages for everything. Not only are they useful for lots of people, but they also force good code (like endianness correctness as seen here) and they also check licenses and are really nitpicky about every detail.