Switch Dockerfile image to wolfi and add pipeline for vulnerability scanning

[kostasb]

https://github.com/elastic/search-developer-productivity/issues/3547

Description

The goal of this PR is to:

1. move "extensible Dockerfiles" to a distroless base image. Dockerfile and Dockerfile.ftest are not used in Elastic's docker image releases but are rather provided as a stepping stone for users to build custom images on.
2. add a pipeline for vulnerability scanning of built images

This is based on @acrewdson 's draft. It uses wolfi-base images and adds buildkite pipelines to build, test and scan the resulting docker images using trivy.

A notification method for vulnerability reports from Trivy is TBD.

Checklists

Pre-Review Checklist

• this PR does NOT contain credentials of any kind, such as API keys or username/passwords (double check config.yml.example)
• this PR has a meaningful title
• this PR links to all relevant github issues that it fixes or partially addresses
• this PR has a thorough description
• Covered the changes with automated tests
• Tested the changes locally
• Added a label for each target release version (example: v7.13.2, v7.14.0, v8.0.0)
• Considered corresponding documentation changes
• Contributed any configuration settings changes to the configuration reference
• if you added or changed Rich Configurable Fields for a Native Connector, you made a corresponding PR in Kibana

Changes Requiring Extra Attention

• Security-related changes (encryption, TLS, SSRF, etc)
• New external service dependencies added.

Release Note

The extensible Dockerfile provided in the connectors repo has been updated to use a different base image for improved security. Users who build custom docker images based on this Dockerfile may need to review their configuration for compatibility with the new base image.

[acrewdson]

great start! I added a few comments

[acrewdson]

wondering if we should consider running this less frequently? I think it would depend on what @elastic/search-extract-and-transform prefer, but the scenario I picture is something like:

• this job runs once a week, or maybe even once every two weeks
• if Trivy identifies any vulnerabilities, a message is sent to a Slack channel, or a GitHub issue is created

let's see what folks think 👍

[oli-g]

• if Trivy identifies any vulnerabilities, a message is sent to a Slack channel, or a GitHub issue is created

I think these notifications could potentially be a duplicate of existing Snyk issues detected on the official container image. What's the value added by these new notifications?

[seanstory]

is Snyk going to scan the images produced by these CI jobs?
I think this PR is doing 3 things, but maybe doesn't need to.

1. switches our Dockerfile to use chainguard (has customer value)
2. Adds CI to validate the Dockerfile works (has customer value)
3. Adds CI to fail if a vuln is found (internal value, with indirect customer value)

I think (3) could probably be done separately, and reuse whatever machinery we use to scan other artifacts, especially since (1) and (2) will significantly cut down on false-positives that we'd have if we tried to do (3) on its own today.

[kostasb]

The CI jobs in this PR do not push images to the docker registry, instead it relies on Trivy for vulnerability scanning, which runs within the context of the pipeline.

The alternative approach is to push the images to an internal namespace on the docker registry and request for these to be added to Snyk. This would result in duplicate reports though as we're already scanning the docker.elastic.co/integrations/elastic-connectors with Snyk built from the Dockerfile.wolfi image which is technically the exact same base OS build.

[acrewdson]

What's the value added by these new notifications?

...

which is technically the exact same base OS build.

These are really good points that I hadn't considered. Given that with this PR we are:

• switching the Dockerfile to become distroless, and making the resultant image considerably more secure
• making the Dockerfile essentially the same as the Dockerfile.wofli, the latter of which produces the images that are pushed and scanned by Snyk
• adding nothing in the Dockerfile except the bare minimum that we need to run connectors (assuming we get rid of git and make 👍)

... then I actually don't see much value in adding the Trivy step (let alone adding notifications based on its results). I wish I'd had these realizations sooner, apologies @kostasb.

What do you all think?

[acrewdson]

@kostasb thanks, I like the idea of leaving the Trivy step there, but not having it send any notifications. But I'll defer to E&T folks (@seanstory @artem-shelkovnikov) to say what they prefer 👍

[seanstory]

I have no objection to leaving it (without notifications) but also I don't think a human will be manually looking at the outputs often. So if we're ok as treating it essentially as a source of debug logs that may or may not be examined, great.

[oli-g]

I understand the following might be a big direction change for this PR, but considering the feedback above, I think it might be worthy to explore: what if, instead of adding a new pipeline (with informative value only), we add the Trivy scan step to the existing pipeline? I think the informative value would be higher for developers working in PRs: the Trivy scan would be part of their feedback loop, and not something relegated to a different pipeline.

Similarly to what we did for ent-search, I would run the Trivy scan step in PRs to main, and on merge on main and other current release branches. What do you all think?

[kostasb]

Merging the extensible Dockerfile build/test/scan steps into the main pipeline makes sense to me, we can proceed this way if the team agrees.

[kostasb]

I pushed a commit that merges the extensible Dockerfiles pipeline into the main pipeline (easy to revert if the team @seanstory @artem-shelkovnikov has a different opinion after all).

As a bonus point, this also enables testing as part of this PR (the new dedicated pipeline wouldn't be picked up until after merged).

[kostasb]

Since in this PR we are building a Dockerfile on a different base image (cgr.dev/chainguard/wolfi-base) there are package discrepancies comparing to the base image used in the official built images wolfi/python:3.11-dev.

For reference, here is a list of packages which, among others (like bash and curl), are available in the wolfi python-3.11 image used for the production image builds.

Do we want this list of packages added to the wolfi-base image? Are connectors requiring some, or all, of them? @elastic/search-extract-and-transform

[artem-shelkovnikov]

I'm not sure if we need the packages mentioned, we really only need dependencies for 3.11 python + base64 available.

[kostasb]

I'm not sure if we need the packages mentioned, we really only need dependencies for 3.11 python + base64 available.

Thanks, then I'll stick with the bare minimum packages and we can iterate if needed.

I added both amd64 and arm64 image builds, mainly for the purpose of smoke testing (with .buildkite/publish/test-docker.sh) on both architectures. I don't expect deviations in the vulnerability testing between architectures.

Do we need some sort of release note for the new base image? It could be a "breaking" change for those users who build their images on top of our Dockerfiles.

[kostasb]

Update re. the packages included with the base image vs the python one that we do docker image builds on: @oli-g suggested to check whether installing python-3.11 on the wolfi-base image installes those as dependencies. It does install most of them as part of the Dockerfile step that adds python (RUN apk add --no-cache python3=~${python_version} make git), with some exceptions (linux-headers) which are unlikely to be relevant for connectors. That said, there shouldn'be any concerns regarding the base image diff.

[artem-shelkovnikov]

It could be a "breaking" change for those users who build their images on top of our Dockerfiles.

Just to check - cgr.dev/chainguard/wolfi-base is publicly available, right?

[kostasb]

It could be a "breaking" change for those users who build their images on top of our Dockerfiles.

Just to check - cgr.dev/chainguard/wolfi-base is publicly available, right?

Yes, we've confirmed that this is a publicly available image that can be downloaded by unauthenticated (not logged in to any registry) docker instances.

[artem-shelkovnikov]

Do we need some sort of release note for the new base image? It could be a "breaking" change for those users who build their images on top of our Dockerfiles.

We can mention it in our release notes indeed. I don't expect a lot of customers to build customer docker images and likely it's just gonna work for them, but it would not hurt to mention that we've updated our docker images recently to make them more secure with potential incompatible changes that the customers will have to fix

[kostasb]

Do we need some sort of release note for the new base image? It could be a "breaking" change for those users who build their images on top of our Dockerfiles.

We can mention it in our release notes indeed. I don't expect a lot of customers to build customer docker images and likely it's just gonna work for them, but it would not hurt to mention that we've updated our docker images recently to make them more secure with potential incompatible changes that the customers will have to fix

Thank you, I added a release note in the issue description.