Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert to OCI builder #125

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
**/*.txt eol=lf
Makefile eol=lf
hello.c eol=lf
/oci/*/** linguist-generated
3 changes: 0 additions & 3 deletions Dockerfile-linux.template

This file was deleted.

3 changes: 0 additions & 3 deletions amd64/hello-world/Dockerfile

This file was deleted.

3 changes: 0 additions & 3 deletions arm32v5/hello-world/Dockerfile

This file was deleted.

3 changes: 0 additions & 3 deletions arm32v6/hello-world/Dockerfile

This file was deleted.

3 changes: 0 additions & 3 deletions arm32v7/hello-world/Dockerfile

This file was deleted.

3 changes: 0 additions & 3 deletions arm64v8/hello-world/Dockerfile

This file was deleted.

3 changes: 0 additions & 3 deletions i386/hello-world/Dockerfile

This file was deleted.

3 changes: 0 additions & 3 deletions mips64le/hello-world/Dockerfile

This file was deleted.

205 changes: 205 additions & 0 deletions oci.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
#!/usr/bin/env bash
set -Eeuo pipefail

image="${1:-hello-world}"

hash="sha256"

self="$(basename "$BASH_SOURCE")"
cd "$(dirname "$(readlink -f "$BASH_SOURCE")")"

ociroot="./oci"
blobs="$ociroot/blobs/$hash"

rm -rf $ociroot

mkdir -p $blobs

toBlob() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

local content="$1"; shift

digest="$(echo -n $content | sha256sum | cut -d " " -f1)"
echo -n $content > "$blobs/$digest"
echo $digest
}

annotations() {
local os="$1"; shift
local arch="$1"; shift
local timestamp="$1"; shift
local revision="$1"; shift

jq -r -n --arg os $os --arg arch $arch --arg timestamp $timestamp --arg revision $revision '{
"com.docker.official-images.bashbrew.arch": $arch,
"org.opencontainers.image.base.name": "scratch",
"org.opencontainers.image.created": $timestamp,
"org.opencontainers.image.revision": $revision,
"org.opencontainers.image.source": "https://github.com/docker-library/hello-world.git#\($revision):\($arch)/hello-world",
"org.opencontainers.image.url": "https://hub.docker.com/_/hello-world",
"org.opencontainers.image.version": $os
LaurentGoderre marked this conversation as resolved.
Show resolved Hide resolved
} | tojson'
}

rootfs() {
local os="$1"; shift
local arch="$1"; shift

path="./$arch/hello-world"
tar -C $path -cf - hello > tmp.tar
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're going to commit these tarballs (and even if we're not, really), we should go out of our way to make sure they are generated reproducibly like we do in BusyBox, which is probably going to involve/require we build them inside a container or image (such as the Dockerfile.build we already have 😅): https://github.com/docker-library/busybox/blob/69136da0a606fde9f84b593f975060819960f42b/build.sh#L44-L53

digest="$(cat tmp.tar | sha256sum | cut -d " " -f1)"
mv tmp.tar "$blobs/$digest"
echo $digest
}

config() {
local digest="$1"; shift
local os="$1"; shift
local arch="$1"; shift
local timestamp="$1"; shift

configArch=""

case "$arch" in
arm64v8)
configArch="arm64"
;;
arm32v*)
configArch="arm"
;;
*)
configArch="$arch"
;;
esac

content="$(jq -r -n --arg digest "$hash:$digest" --arg os $os --arg arch $configArch --arg timestamp $timestamp '{
config: {
Cmd: [
"/hello"
],
Env: [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
WorkingDir: "/"
},
os: $os,
architecture: $arch,
rootfs: {
diff_ids: [
$digest
],
type: "layers"
},
"created": $timestamp,
} | tojson')"

toBlob $content
LaurentGoderre marked this conversation as resolved.
Show resolved Hide resolved
}

manifest() {
local configdigest="$1"; shift
local rootfsdigest="$1"; shift
local annotations="$1"; shift

configsize="$(wc -c "$blobs/$configdigest" | awk '{print $1}')"
rootfssize="$(wc -c "$blobs/$rootfsdigest" | awk '{print $1}')"

content="$(jq -r -n \
--arg configdigest "$hash:$configdigest" \
--arg configsize $configsize \
--arg rootfsdigest "$hash:$rootfsdigest" \
--arg rootfssize $rootfssize \
--argjson annotations $annotations \
'{
schemaVersion: 2,
mediaType: "application/vnd.oci.image.manifest.v1+json",
config: {
mediaType: "application/vnd.oci.image.config.v1+json",
digest: $configdigest,
size: $configsize | tonumber
},
layers: [
{
mediaType: "application/vnd.oci.image.layer.v1.tar",
digest: $rootfsdigest,
size: $rootfssize | tonumber
}
],
annotations: $annotations
} | tojson')"

toBlob $content
}

indexdescriptor() {
local manifestdigest="$1"; shift
local os="$1"; shift
local arch="$1"; shift
local annotations="$1"; shift

manifestsize="$(wc -c "$blobs/$manifestdigest" | awk '{print $1}')"

platform=""

case "$arch" in
arm64v8)
platform=$(jq -r -n --arg os $os --arg arch "arm64" --arg variant "v8" '{
architecture: $arch,
os: $os,
variant: $variant
} | tojson')
;;
arm32v*)
platform=$(jq -r -n --arg os $os --arg arch "arm" --arg variant "${arch/arm32/}" '{
architecture: $arch,
os: $os,
variant: $variant
} | tojson')
;;
*)
platform=$(jq -r -n --arg os $os --arg arch $arch '{
architecture: $arch,
os: $os
} | tojson')
;;
esac
Comment on lines +121 to +144
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should let bashbrew interpret these for us instead, like we do in BusyBox: https://github.com/docker-library/busybox/blob/69136da0a606fde9f84b593f975060819960f42b/build.sh#L11-L13 😅

(however, we don't actually need this for index.json - only for the config blob)


jq -r -n \
--arg digest "$hash:$manifestdigest" \
--arg manifestsize $manifestsize \
--argjson platform $platform \
--argjson annotations $annotations \
'{
mediaType: "application/vnd.oci.image.manifest.v1+json",
digest: $digest,
size: $manifestsize | tonumber,
platform: $platform,
annotations: $annotations
} | tojson'
}

created="2025-01-21T23:32:32Z"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any particular reason for this specific value? 😅

(long-term we probably need to scrape an appropriate value per-architecture from the latest Git commit to touch the relevant hello binary or something like that, but that might get recursive if we want to update the binary and the OCI layouts in one step 🤔 there's also an argument we could make that committing the binaries and the tarballs/OCI layouts is overkill and we should only commit the latter)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hahaha, that was s just the value from the current one in registry and forgot to change it


arches=( *"/$image/hello" )
LaurentGoderre marked this conversation as resolved.
Show resolved Hide resolved

manifests="[]"

for path in "${arches[@]}"; do
os="linux"
arch="${path/\/hello\-world\/hello/}"

annotations=$(annotations $os $arch $created "foo")
rootfs=$(rootfs $os $arch)
config=$(config $rootfs $os $arch $created)
manifest=$(manifest $config $rootfs $annotations)
indexdescriptor=$(indexdescriptor $manifest $os $arch $annotations)

manifests=$(jq -r -n --argjson manifests $manifests --argjson manifest $indexdescriptor '$manifests | . += [$manifest] | tojson')
done

# TODO: Windows
LaurentGoderre marked this conversation as resolved.
Show resolved Hide resolved

jq -r -n --argjson manifests $manifests '{
schemaVersion: 2,
mediaType: "application/vnd.oci.image.index.v1+json",
manifests: $manifests
}' > "$ociroot/index.json"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lumping all the architectures together in one big index.json won't actually work in the oci-import builder -- it requires exactly/only one image in index.json or (as I noted in #124) that we specify File: pointing to a JSON file that contains a single image descriptor.

As much as I'd like to lump all these into a single OCI layout here (it's very cute!), I think we'll actually have a much better time downstream if we instead do a separate OCI layout per architecture (amd64/oci/index.json, i386/oci/index.json, etc), especially since the sourceId for the image/build will currently be calculated based on the full contents of the OCI layout, not just the one architecture, so any update to any binaries will result in "rebuilding" all of them (which, to be fair, is something we should try to fix downstream, but implementing that correctly is very complex).

Binary file not shown.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file not shown.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file not shown.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file not shown.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file not shown.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file not shown.
Binary file not shown.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file not shown.
Binary file not shown.
Binary file not shown.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading