Convert to OCI builder #125
Conversation
There was a problem hiding this comment.
This is a good start! There are a lot of little nitpicks, but I tried to stay away from them and focus on the higher-level things first so we can hash out what the end state needs to look like before we get too buried in the nuts and bolts / implementation (although it's definitely useful to have an explicit implementation to point at for articulating and discussing those 👍).
I think at a really high level, this probably needs to be integrated deeper into the existing build process, especially so we can have it be fully reproducible (by building it inside that container/image). 🤔
|
|
||
| created="2025-01-21T23:32:32Z" | ||
|
|
||
| arches=( *"/$image/hello" ) |
There was a problem hiding this comment.
| schemaVersion: 2, | ||
| mediaType: "application/vnd.oci.image.index.v1+json", | ||
| manifests: $manifests | ||
| }' > "$ociroot/index.json" |
There was a problem hiding this comment.
Lumping all the architectures together in one big index.json won't actually work in the oci-import builder -- it requires exactly/only one image in index.json or (as I noted in #124) that we specify File: pointing to a JSON file that contains a single image descriptor.
As much as I'd like to lump all these into a single OCI layout here (it's very cute!), I think we'll actually have a much better time downstream if we instead do a separate OCI layout per architecture (amd64/oci/index.json, i386/oci/index.json, etc), especially since the sourceId for the image/build will currently be calculated based on the full contents of the OCI layout, not just the one architecture, so any update to any binaries will result in "rebuilding" all of them (which, to be fair, is something we should try to fix downstream, but implementing that correctly is very complex).
| } | tojson' | ||
| } | ||
|
|
||
| created="2025-01-21T23:32:32Z" |
There was a problem hiding this comment.
Any particular reason for this specific value? 😅
(long-term we probably need to scrape an appropriate value per-architecture from the latest Git commit to touch the relevant hello binary or something like that, but that might get recursive if we want to update the binary and the OCI layouts in one step 🤔 there's also an argument we could make that committing the binaries and the tarballs/OCI layouts is overkill and we should only commit the latter)
There was a problem hiding this comment.
Hahaha, that was s just the value from the current one in registry and forgot to change it
|
|
||
| mkdir -p $blobs | ||
|
|
||
| toBlob() { |
There was a problem hiding this comment.
You might be interested in https://gist.github.com/tianon/35528d9851ebf0b0767c04cec5e01bcf, particularly https://gist.github.com/tianon/35528d9851ebf0b0767c04cec5e01bcf#file-oci-put 😅
| "com.docker.official-images.bashbrew.arch": $arch, | ||
| "org.opencontainers.image.base.name": "scratch", | ||
| "org.opencontainers.image.created": $timestamp, | ||
| "org.opencontainers.image.revision": $revision, | ||
| "org.opencontainers.image.source": "https://github.com/docker-library/hello-world.git#\($revision):\($arch)/hello-world", | ||
| "org.opencontainers.image.url": "https://hub.docker.com/_/hello-world", | ||
| "org.opencontainers.image.version": $os |
There was a problem hiding this comment.
All these annotations are added automatically in the index by meta-scripts, so we shouldn't need them at all here. 🙈
| local arch="$1"; shift | ||
|
|
||
| path="./$arch/hello-world" | ||
| tar -C $path -cf - hello > tmp.tar |
There was a problem hiding this comment.
If we're going to commit these tarballs (and even if we're not, really), we should go out of our way to make sure they are generated reproducibly like we do in BusyBox, which is probably going to involve/require we build them inside a container or image (such as the Dockerfile.build we already have 😅): https://github.com/docker-library/busybox/blob/69136da0a606fde9f84b593f975060819960f42b/build.sh#L44-L53
| platform="" | ||
|
|
||
| case "$arch" in | ||
| arm64v8) | ||
| platform=$(jq -r -n --arg os $os --arg arch "arm64" --arg variant "v8" '{ | ||
| architecture: $arch, | ||
| os: $os, | ||
| variant: $variant | ||
| } | tojson') | ||
| ;; | ||
| arm32v*) | ||
| platform=$(jq -r -n --arg os $os --arg arch "arm" --arg variant "${arch/arm32/}" '{ | ||
| architecture: $arch, | ||
| os: $os, | ||
| variant: $variant | ||
| } | tojson') | ||
| ;; | ||
| *) | ||
| platform=$(jq -r -n --arg os $os --arg arch $arch '{ | ||
| architecture: $arch, | ||
| os: $os | ||
| } | tojson') | ||
| ;; | ||
| esac |
There was a problem hiding this comment.
We should let bashbrew interpret these for us instead, like we do in BusyBox: https://github.com/docker-library/busybox/blob/69136da0a606fde9f84b593f975060819960f42b/build.sh#L11-L13 😅
(however, we don't actually need this for index.json - only for the config blob)
| "created": $timestamp, | ||
| } | tojson')" | ||
|
|
||
| toBlob $content |
There was a problem hiding this comment.
Having all these functions go straight into blobs/sha256/xxx is valid, but it makes the end result hard for us humans to navigate and make sense out of, which is why BusyBox takes the approach of human-readable names in the OCI directory like rootfs.tar.gz, image-config.json, and image-manifest.json, and then makes symlinks into blobs/sha256/xxx instead: https://github.com/docker-library/busybox/blob/69136da0a606fde9f84b593f975060819960f42b/build.sh#L88
| manifests=$(jq -r -n --argjson manifests $manifests --argjson manifest $indexdescriptor '$manifests | . += [$manifest] | tojson') | ||
| done | ||
|
|
||
| # TODO: Windows |
There was a problem hiding this comment.
Unfortunately, we won't be able to use OCI import for Windows (the only remotely tenable means would be foreign layers for the Windows layers, but I believe the oci-import builder explicitly blocks/clears those), so they'll need to stay as Dockerfile-based builds for the foreseeable future.
Addresses #124