Export layers to "regular" docker

[nmattia]

Hi, thanks for the great action!

I've been trying to figure out how to export a buildx build to Docker that includes the layers. Right now I use load: true which allows me to run e.g. docker run subsequently, however now I'm trying to figure out how to export the layers as well.

I have a 2-step build. The first step is a build-push-action build that is cached between runs (using cache-to and saved with actions/cache). My second build is not cached, and re-uses the layers from the first build with cache-from. This second build is saved to docker (through load: true), so I can use docker run and other commands (build file below).

While I need build-push-action in the first build step (in order to export and cache the layers), I'd like to not use it in the second step, and use a regular docker build instead. The reason is that the second step wastes about a minute importing the output with cache-from. Unfortunately, I noticed that when I save the image with load: true, a subsequent docker build won't be able to re-use the layers, but will see the image as a whole. Is there a way to export the layers from buildx so that I can use them in a regular docker build?

- uses: actions/checkout@v2

- name: Set up docker buildx
  uses: docker/setup-buildx-action@v1

- name: Cache docker layers
  uses: actions/cache@v2
  id: docker-cache
  with:
    path: /tmp/.buildx-cache
    key: ${{ runner.os }}-buildx-${{ hashFiles( '.dockerignore', 'Dockerfile', ...) }}

- name: Build base Docker image
  uses: docker/build-push-action@v2
  with:
    tags: my-image-cached
    cache-from: type=local,src=/tmp/.buildx-cache
    cache-to: ${{ (! steps.docker-cache.outputs.cache-hit) && 'type=local,dest=/tmp/.buildx-cache-new,mode=max' || '' }}
    target: cached # the cached bits are e.g. FROM ubuntu AS cached

# https://github.com/docker/build-push-action/issues/252
- name: Move cache
  run: |
    if [ -d /tmp/.buildx-cache-new ]
    then
      rm -rf /tmp/.buildx-cache
      mv /tmp/.buildx-cache-new /tmp/.buildx-cache
    fi

- name: Build final docker image
  uses: docker/build-push-action@v2
  with:
    tags: my-image
    load: true # effectively makes the image usable in 'docker run'
    cache-from: type=local,src=/tmp/.buildx-cache

- run: docker run my-image ...

[crazy-max]

@nmattia

Is there a way to export the layers from buildx so that I can use them in a regular docker build?

There are two things to take into account. When you use the cache exporter (cache-to) it effectively exports cache for all stages across your build (mode=max). There is also another internal cache for BuildKit state that is created on docker/setup-buildx-action@v1 step when the builder is created. This cache lives and is available across all the underlying steps of the job so you can reuse the docker/build-push-action@v2 at any steps in your job. There is a pending issue to also handle this cache: #482 (comment)

The reason is that the second step wastes about a minute importing the output with cache-from.

I agree and I would suggest to use the GitHub cache exporter instead.

Here is what I would do:

- uses: actions/checkout@v2

- name: Set up docker buildx
  uses: docker/setup-buildx-action@v1

- name: Build base Docker image
  uses: docker/build-push-action@v2
  with:
    tags: my-image-cached
    cache-from: type=gha,scope=cached-stage
    cache-to: type=gha,scope=cached-stage,mode=max
    target: cached

- name: Build final docker image
  uses: docker/build-push-action@v2
  with:
    tags: my-image
    load: true
    cache-from: type=gha,scope=cached-stage

- run: docker run my-image ...

Also Build final docker image uses the Git context with your current workflow which makes uses: actions/checkout@v2 not necessary.

[crazy-max]

You could also improve your workflow and have a dedicated job that will just build and cache the cached stage and make another job that would be dedicated to the build part and that depends on the cache job:

jobs:
  base:
    runs-on: ubuntu-latest
    steps:
      -
        name: Checkout
        uses: actions/checkout@v2
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      -
        name: Build base Docker image
        uses: docker/build-push-action@v2
        with:
          context: .
          cache-from: type=gha,scope=cached-stage
          cache-to: type=gha,scope=cached-stage,mode=max
          target: cached

  build:
    needs:
      - base
    runs-on: ubuntu-latest
    steps:
      -
        name: Checkout
        uses: actions/checkout@v2
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      -
        name: Build final docker image
        uses: docker/build-push-action@v2
        with:
          context: .
          tags: my-image
          load: true
          cache-from: type=gha,scope=cached-stage
      -
        run: docker run my-image ...

[nmattia]

@crazy-max thanks for the quick reply!

There is also another internal cache for BuildKit state [..]. This cache lives and is available across all the underlying steps of the job

So you mean that I actually do not need to cache-from in "Build final docker image"? I thought I tried without it, and the build failed (edit: the build started from scratch), but maybe I'm wrong. I'll try right now!

I would suggest to use the GitHub cache exporter instead.

Thanks for the suggestion. I looked into it but saw it was labeled as "experimental" in the buildx docs, and moreover couldn't quite understand how it works; does it store artefacts locally, or will it upload the cache after each build? Similarly, won't splitting my build in two different jobs cause more download and upload? One fact that I didn't mention is that my base image is ~3GB, which means that uploading and downloading takes some time.

[crazy-max]

@nmattia

So you mean that I actually do not need to cache-from in "Build final docker image"? I thought I tried without it, and the build failed, but maybe I'm wrong. I'll try right now!

Yes it should work. Let me know if you encounter an issue.

and moreover couldn't quite understand how it works; does it store artefacts locally, or will it upload the cache after each build?

It uses the same API as actions/cache@v2. See https://github.com/moby/buildkit#github-actions-cache-experimental

Similarly, won't splitting my build in two different jobs cause more download and upload? One fact that I didn't mention is that my base image is ~3GB, which means that uploading and downloading takes some time.

Yes in that case it would be more effective to have a single job agree but using the GitHub cache exporter of BuildKit is more effective than actions/cache@v2 because there is no compression/decompression steps.

[nmattia]

@crazy-max Ok so I confirm: if I remove cache-from in "Build final docker image", the build unfortunately starts from scratch! See here.

Also, is there any way to have something akin to docker run with buildx? Then I wouldn't need to save the image with loaded: true, which right now takes more time than the build itself (when there's a cache hit)!

[crazy-max]

@nmattia

Ok so I confirm: if I remove cache-from in "Build final docker image", the build unfortunately starts from scratch! See here.

Can you declare a type=cacheonly as output and let me know if it works?

      -
        name: Build base Docker image
        uses: docker/build-push-action@v2
        with:
          context: .
          cache-from: type=gha,scope=cached-stage
          cache-to: type=gha,scope=cached-stage,mode=max
          outputs: type=cacheonly
          target: cached

Also, is there any way to have something akin to docker run with buildx? Then I wouldn't need to save the image with loaded: true, which right now takes more time than the build itself (when there's a cache hit)!

You could use the docker driver with setup-buildx-action but this driver does not support cache-from/to unfortunately. Is there too much overhead with load?

[nmattia]

@crazy-max Base image building now, will report when it's done. I'm assuming I should try without cache-from in the second step, and it should use the internal buildx cache state. Correct?

Is there too much overhead with load?

There's a fair bit of overhead. When there's a cache hit, the actual (second step, non-cached) build takes 2mn (125s), but the OCI export step takes an additional 3.5mn (200s).

[crazy-max]

Base image building now, will report when it's done. I'm assuming I should try without cache-from in the second step, and it should use the internal buildx cache state. Correct?

Yes that's it.

[nmattia]

Unfortunately omitting cache-from in the second step still doesn't work , but that's ok. The gha support works very well, thanks for the pointer! It's much simpler than using actions/cache, though I notice it takes a little longer. That shouldn't be a problem though, thanks to your comment here! Looking into that now.

[crazy-max]

You're welcome, see also dfinity/nns-dapp#357 (review) to export artifacts as it seems it's what you want so it removes the load overhead.

[nmattia]

Thanks for all the help here, really appreciated! The workflow is much simpler and about twice as fast. Got yourself a new sponsor :)