FEATURE: Support static web hosting

[smil2k]

When the maven repository format was introduced, there was a design goal, that maven repository could be hosted on a static hosting server.
I was wondering whether it would make sense to lower the hosting requirements of a OCI images by using the same principle.

This would allow skopeo users (mainly podman and oci-o) to lower the cost of hosting of OCI containers directly utilizing blobs like s3 or azure blob to store the images. The layout of the files could be similar to what local directories has, the only change would be to add transport mechanisms (wagons in maven lingo) to get them. Eg. s3 wagon, https or azure blob etc..

Supporting only download and upload is fairly trivial to implement.

Search is a better question. Typically blobs have directory listing features, but I would not mind not having it at all. All my use cases use absolute names.

What is your thoughts about this feature?

[mtrmac]

Thanks for reaching out.

Tools to build such web servers already exist, e.g. https://github.com/NicolasT/static-container-registry uses Skopeo underneath. So far that didn’t need specialized Skopeo support.

WRT search and the like, once contemplating adding server-side request handlers at all, I don’t know why one would stop at any particular point, and not build a full-featured registry server. It just seems simpler to me to deploy a dedicated full-featured registry server than try to … deploy a dedicated not-full-featured registry server; I can see no benefit to trying to be in the middle of the spectrum, it has the disadvantages of both extremes.

[smil2k]

Thank you for the quick answer.

The docker authorization and repo is a bit overcompkicated for my taste. Also it does not lend itself for hosting it as a blob. My aim is to have serverless repos, so I only have to pay for network and storage. No VMs.

This is why I was thinking in adding other protocols to read these "local files" besides fopen.

Regarding search: s3 and other object stores have search options, so it is possible to implement them in the client if necessary. This could allow supporting podman search eventually.

I would be happy to contribute as a pull request if it is welcome.

[mtrmac]

The docker authorization and repo is a bit overcompkicated for my taste.

That seems simply incompatible with pure static hosting: The code needs to return a properly-formatted 401 without credentials, and a 200 with content otherwise. That would strongly rely on the capabilities of the static web server.

The linked static server project, similarly, spends bulk of the code on specifically interfacing with Nginx — more than doing anything that specialized about container images.

It’s just that, correspondingly, bulk of the CI and infrastructure would need to be focused on specifically interfacing with that web server, and testing that interoperability.

Clearly that’s software that can be written, but there’s ~0 overlap with the existing Skopeo CI environment and requirements, and little overlap with the feature set, so I don’t see much benefit for embedding any specific web server knowledge into Skopeo itself, or into the Skopeo repo.

[smil2k]

I think I was not clear enough about my intentions. Maybe this stems from my lack of knowledge on skopeo.

My understanding is:

• Skopeo is the library which is used by podman and co to download images. (copy images from remote repository to local repo).
• Skopeo is a command line tool to use the library as a cli.

Currently the library supports remote repository format Docker v2 (OCI remote), and local image stores container/storage, docker, oci and "non-standard" dir.

My suggestion would be to enable a new remote repository format option: remote "OCI layout" repository. The OCI directory format has an index.json which contains the tags and images available (it is possible to use for search) and it defines a file structure which can be used as is in a static web hosting environment.

So the only addition would be adding a transport medium for OCI local repository layouts and omit all the "401" per directory complexities docker registries do. This is the thing I referred as "wagons" in maven world. Transport mechanisms to download and upload to remote oci layouts.

Making a generic read only repository is easy as using only GET requests and search using the index.json. (+basic auth)

Adding push support is a bit more challenging because of the index.json describes everything: it needs to be downloaded and reuploaded. S3 and google offer ETag based optimistic locking, azure blobs provides file locking to solve concurrency.

This type of remote can be placed under any directory, so it is possible to reduce size of index.json and the number of write collisions to the minimum by placing each image under a separated directory.

[mtrmac]

It isn’t that simple, because the registry is not that trivial a static host; the protocol requires it to set various HTTTP headers in data responses. Content-Type at least (more specific than just “JSON”), and perhaps Docker-Content-Digest.

So, the writer would need to be HTTP-server-specific.

[smil2k]

Thank you for your detailed feedback. I appreciate the time you’ve taken to engage with my proposal. Let me clarify my intentions and address your concerns directly, as I think there might be a slight misalignment in how I’ve explained the idea.

Summary of Proposal
My main goal is to enable serverless, cost-efficient hosting of OCI images by leveraging static object stores (e.g., S3, Azure Blob, Google Cloud Storage) as repositories for OCI layout-based images. This approach focuses on:

Minimizing Infrastructure Overhead: By avoiding full-featured registry servers, we can reduce costs and complexity.
Static Hosting Compatibility: Using the OCI layout format (with index.json and accompanying blobs), images can be stored and served directly from an object store or any other static file host.
Basic Transport Mechanisms: Adding support for remote OCI layout repositories in Skopeo, with a focus on simple GET/PUT operations, would enable this workflow.

Addressing Key Concerns

1. Protocol Compliance and HTTP Headers
You mentioned that static hosting is not entirely trivial because certain protocol requirements—like setting specific HTTP headers (Content-Type, Docker-Content-Digest)—are needed.

These headers can often be addressed by configuring object storage services (e.g., S3 allows setting Content-Type metadata per file). But the client (Skopeo) could handle reasonable defaults when headers are missing or inconsistent. Please note: filesystems has no such concept as content-types. What I would like to achieve is to be able to store OCI layout on a static web. As OCI layout is fully supported by Skopeo with just files, so I think changing from local file access to HTTP GET should not be breaking anything.

2. Benefits vs. Complexity
I understand your hesitation about adding server-specific logic to Skopeo or embedding new features that diverge from its core scope. However, my proposal does not aim to reimplement a registry or add extensive server-specific code. Instead, it aims to:

Extend Skopeo’s transport support to recognize remote OCI layout repositories, treating them as remote equivalents of the existing local OCI directory format.
Use existing object storage features and a minimal set of client-side logic to handle these repositories.
The idea is not to replace full-featured registries but to offer a lighter alternative for users whose use cases align with static hosting and simple transport needs.

I could even understand if these remote repositories would be entirely read only, and pushing would be done by something else as pushing might open a box of pandora on file locking problems.

3. Alignment with Skopeo’s CI and Testing
I understand that testing and maintaining such a feature within the current Skopeo CI setup would require additional effort. I would be happy to contribute not just the implementation but also any necessary tests, documentation, and examples to demonstrate the feasibility and ensure smooth integration with Skopeo’s ecosystem.

Thank you again for your feedback!

[mtrmac]

Static Hosting Compatibility: Using the OCI layout format (with index.json and accompanying blobs), images can be stored and served directly from an object store or any other static file host.

This seems to be making an incorrect assumption that the OCI image layout is the same as the on-registry URL layout. That’s not the case.

As you note, Skopeo can already create/consume OCI image layout filesystems. That is not immediately useful for with statically serving the registry protocol.

Or, alternatively, this can be read as a proposal to add a new, orthogonal, HTTP transport that uses OCI image layout-like URL paths, independently from the registry procotol. That would be very confusing for users, and I don’t see any benefit significant enough to be worth that confusion.

[mtrmac]

a new, orthogonal, HTTP transport … That would be very confusing for users

E.g. consider all of the ecosystem that uses registry references like example.com/ns/repo:tag all over the place. There is no obvious and clearly-compatible way to add a protocol marker into those strings, especially as various tools tend to invent/add extra syntax within the same string field. (I have already encountered – and, in some cases, eliminated — ambiguities like dir:foo meaning either dir:$(pwd)/foo or docker://docker.io/library/dir:foo, depending on which exact code path is involved — without anyone writing the code realizing this.)

So, this proposal would either require an ecosystem-wide effort to support a new syntax, or be inherently restricted to a very small set of tools, dooming it to obscurity.