Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make tarball target work with arm64 cross-compilation and simplify Makefile/Dockerfile #1805

Merged
merged 4 commits into from
Nov 28, 2023

Conversation

mtardy
Copy link
Member

@mtardy mtardy commented Nov 27, 2023

I tested on my amd64 VM and it generated the correct binaries for each architecture with cross-compilation. Can you check as well that it works for you with:

# on an amd64 machine, first should generate the amd64 tarball and second arm64 tarball
make tarball 
make TARGET_ARCH=arm64 tarball
Add arm64 support for the tarball via cross-compilation on amd64.

This removes the tetragon-image target that was just an alias to doing
`make tetragon tetra` and also removes the cross-compiler part since
tetragon no longer use CGO to compile.

Signed-off-by: Mahe Tardy <[email protected]>
@mtardy mtardy added the release-note/misc This PR makes changes that have no direct user impact. label Nov 27, 2023
@mtardy mtardy requested a review from tixxdz November 27, 2023 15:18
@mtardy mtardy requested a review from a team as a code owner November 27, 2023 15:18
Copy link

netlify bot commented Nov 27, 2023

Deploy Preview for tetragon ready!

Name Link
🔨 Latest commit 088a24f
🔍 Latest deploy log https://app.netlify.com/sites/tetragon/deploys/6564b3595af1530008f3c205
😎 Deploy Preview https://deploy-preview-1805--tetragon.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@tixxdz
Copy link
Member

tixxdz commented Nov 27, 2023

Hi @mtardy thank you for handling this.

I tested it , make tarball then make TARGET_ARCH=arm64 tarbal, it fails with:

=> CANCELED [tetragon-builder 3/4] COPY . .                                                                                                                                                           8.6s
------                                                                                                                                                                                                      
 > [base-build 2/7] RUN apk add iproute2:                                                                                                                                                                   
1.437 exec /bin/sh: exec format error
------
Dockerfile:72
--------------------
  70 |     # retrieves (cross-)compiled binaries from builders
  71 |     FROM docker.io/library/alpine:3.18.4@sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978 as base-build
  72 | >>> RUN apk add iproute2
  73 |     RUN mkdir /var/lib/tetragon/ && \
  74 |         mkdir -p /etc/tetragon/tetragon.conf.d/ && \
--------------------
ERROR: failed to solve: process "/bin/sh -c apk add iproute2" did not complete successfully: exit code: 1
make: *** [Makefile:275: image] Error 1

So it needs arm emulation here, if there is no easy way and it is complicated, then would it be possible at the last stage of copying the binaries make targetplatform same as buildplatform? so we run in amd64 context and we copy binaries from other arm64 builds?

Also we can add an environment variable TARBALL_CROSS_COMPILE , so if set we use the above where targetplatform becomes buildplatform as the aim is to just have the arm binaries copied into the final one layer image. We don't want to run that image... just use it to dump into tarball. If TARBALL_CROSS_COMPILE is not set then no change current solution is perfect.

@mtardy
Copy link
Member Author

mtardy commented Nov 27, 2023

Hi @mtardy thank you for handling this.

I tested it , make tarball then make TARGET_ARCH=arm64 tarbal, it fails with:


=> CANCELED [tetragon-builder 3/4] COPY . .                                                                                                                                                           8.6s

------                                                                                                                                                                                                      

 > [base-build 2/7] RUN apk add iproute2:                                                                                                                                                                   

1.437 exec /bin/sh: exec format error

------

Dockerfile:72

--------------------

  70 |     # retrieves (cross-)compiled binaries from builders

  71 |     FROM docker.io/library/alpine:3.18.4@sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978 as base-build

  72 | >>> RUN apk add iproute2

  73 |     RUN mkdir /var/lib/tetragon/ && \

  74 |         mkdir -p /etc/tetragon/tetragon.conf.d/ && \

--------------------

ERROR: failed to solve: process "/bin/sh -c apk add iproute2" did not complete successfully: exit code: 1

make: *** [Makefile:275: image] Error 1

So it needs arm emulation here, if there is no easy way and it is complicated, then would it be possible at the last stage of copying the binaries make targetplatform same as buildplatform? so we run in amd64 context and we copy binaries from other arm64 builds?

Also we can add an environment variable TARBALL_CROSS_COMPILE , so if set we use the above where targetplatform becomes buildplatform as the aim is to just have the arm binaries copied into the final one layer image. We don't want to run that image... just use it to dump into tarball. If TARBALL_CROSS_COMPILE is not set then no change current solution is perfect.

You just need to install qemu-user-static on your amd64 machine.

You need emulation for a tiny fraction of the steps just to execute some Docker RUN operation for the last stages of the Dockerfile. Qemu is part of a functional Docker install with buildx to build multi-arch images, see details here https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/.

@tixxdz
Copy link
Member

tixxdz commented Nov 28, 2023

Thank you Mahé, yes it seems the arm emulation got better, it used to fail past years due to some complex instructions... anyway just tested tetra arm64 it works, didn't test daemon.

Could you please test the generated tarball? and as a later step I will add a ci job for it.

Feel free to merge ;-)

@mtardy
Copy link
Member Author

mtardy commented Nov 28, 2023

Thank you Mahé, yes it seems the arm emulation got better, it used to fail past years due to some complex instructions... anyway just tested tetra arm64 it works, didn't test daemon.

Could you please test the generated tarball? and as a later step I will add a ci job for it.

Feel free to merge ;-)

I tested it manually on my arm64 VM and it worked fine. Thanks let's merge this.

@mtardy mtardy added the arch/arm64 arm64 issues label Nov 28, 2023
@mtardy mtardy merged commit 7d7d6e8 into main Nov 28, 2023
37 checks passed
@mtardy mtardy deleted the pr/mtardy/dockerfile-makefile-fixes branch November 28, 2023 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch/arm64 arm64 issues release-note/misc This PR makes changes that have no direct user impact.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants