OSTree Plugin (#28)

* adds plugins.md as starting point for plugin how-to documentation links README.md to plugins.md * adds IsTLSMiddleware for convenience refactors repository.Manager to non-generic form to allow nil without needing to satisfy generic constraints refactors code depending on generic repository.Manager to new non-generic form various comments * ignore intellij project state files * refactors service to a non-generic form * initial implementation of ostree plugin (without mirroring) refactors yum plugin executable to use new non-generic Serve refactors static plugin executable to use new non-generic Serve * gofumpt refactors beskctl to use cobra adds beskarctl ostree sub-command tree adds beskarctl static file sub-command tree documents beskarctl * gofumpt fixes kun command typos adds summary.sig as known tag changes regex separator from file to repo refactors beskarctl to use cobra refactors beskarctl subcommands into packages implements beskarctl ostree push fixes ostree router.rego request body issue * adds plugins.md as starting point for plugin how-to documentation links README.md to plugins.md * adds IsTLSMiddleware for convenience refactors repository.Manager to non-generic form to allow nil without needing to satisfy generic constraints refactors code depending on generic repository.Manager to new non-generic form various comments * ignore intellij project state files * refactors service to a non-generic form * initial implementation of ostree plugin (without mirroring) refactors yum plugin executable to use new non-generic Serve refactors static plugin executable to use new non-generic Serve * gofumpt refactors beskctl to use cobra adds beskarctl ostree sub-command tree adds beskarctl static file sub-command tree documents beskarctl * gofumpt fixes kun command typos adds summary.sig as known tag changes regex separator from file to repo refactors beskarctl to use cobra refactors beskarctl subcommands into packages implements beskarctl ostree push fixes ostree router.rego request body issue * various changes from PR review fixes for linter * more fixes for linter adds ostree helm chart regens ostree api for future work * adds libostree testdata to gitignore * libostree finished ostree plugin finished (needs testing) * fixes various bugs discovered after manual testing renames ostree to libostree to better represent what it does implements GCancellable to allow c functions to respect go context.Context * gofumpt linter fixes * removes unused dockerfile * fixes unit test build failure by introducing buildEnv and buildExecStmts from binaryConfigs attempts to ignore ALL intellij files this time renames free freeFunc to deferFree deferredFreeFn for a more clear api bumps beskar-ostree go version to match others * removes mistaken C import causing mage to no longer see test targets. removes intellij files fixes timeFormat constant per coderabbit suggestion * fixes logic error in Depth() libostree option * rebase upstream main fixes copyright headers * removed comments again... * fixes linter issues * adds libostree testdata adds documentation about generating libostree testdata * adds libostree testdata changes libostree to write to /tmp instead of ./testdata * adds --priviledged option to test:unit container
ctrliq · Jan 15, 2024 · 01976d3 · 01976d3
1 parent 6aec20b
commit 01976d3
Show file tree

Hide file tree

Showing 93 changed files with 4,750 additions and 163 deletions.
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -69,4 +69,18 @@ jobs:
       - name: Release beskar-static image
         run: ./scripts/mage ci:image ghcr.io/ctrliq/beskar-static:${{ github.ref_name }} "${{ github.actor }}" "${{ secrets.GITHUB_TOKEN }}"
       - name: Release beskar-static helm chart
-        run: ./scripts/mage ci:chart ghcr.io/ctrliq/helm-charts/beskar-static:${{ github.ref_name }} "${{ github.actor }}" "${{ secrets.GITHUB_TOKEN }}"
+        run: ./scripts/mage ci:chart ghcr.io/ctrliq/helm-charts/beskar-static:${{ github.ref_name }} "${{ github.actor }}" "${{ secrets.GITHUB_TOKEN }}"
+
+  release-beskar-ostree:
+    name: release beskar-ostree
+    needs: [lint, tests]
+    runs-on: ubuntu-22.04
+    steps:
+      - uses: actions/setup-go@v3
+        with:
+          go-version: '1.21'
+      - uses: actions/checkout@v3
+      - name: Release beskar-ostree image
+        run: ./scripts/mage ci:image ghcr.io/ctrliq/beskar-ostree:${{ github.ref_name }} "${{ github.actor }}" "${{ secrets.GITHUB_TOKEN }}"
+      - name: Release beskar-ostree helm chart
+        run: ./scripts/mage ci:chart ghcr.io/ctrliq/helm-charts/beskar-ostree:${{ github.ref_name }} "${{ github.actor }}" "${{ secrets.GITHUB_TOKEN }}"
diff --git a/.gitignore b/.gitignore
@@ -3,3 +3,7 @@ build/output
 .envrc.local
 vendor
 go.work.sum
+
+
+*/.idea
+/.idea/
diff --git a/.idea/.gitignore b/.idea/.gitignore
diff --git a/.idea/beskar.iml b/.idea/beskar.iml
diff --git a/.idea/modules.xml b/.idea/modules.xml
diff --git a/README.md b/README.md
@@ -5,9 +5,10 @@ It's designed to support various artifacts and expose them through dedicated plu
 
 ### Features
 
-* Modular/Extensible via plugins
+* Modular/Extensible via [plugins](docs/plugins.md)
 * Support for YUM repositories (beskar-yum)
 * Support for static file repositories (beskar-static)
+* Support for OSTree repositories ([beskar-ostree](internal/plugins/ostree/README.md))
 
 ### Docker images
 
@@ -41,6 +42,12 @@ For beskar-static helm chart:
 helm pull oci://ghcr.io/ctrliq/helm-charts/beskar-static --version 0.0.1 --untar
 ```
 
+For beskar-static helm chart:
+
+```
+helm pull oci://ghcr.io/ctrliq/helm-charts/beskar-ostree --version 0.0.1 --untar
+```
+
 ### Compilation
 
 Binaries are not provided as part of releases, you can compile it yourself by running:

diff --git a/build/mage/build.go b/build/mage/build.go
@@ -55,13 +55,16 @@ type binaryConfig struct {
 	buildTags         []string
 	baseImage         string
 	integrationTest   *integrationTest
+	buildEnv          map[string]string
+	buildExecStmts    [][]string
 }
 
 const (
 	BeskarBinary       = "beskar"
 	BeskarctlBinary    = "beskarctl"
 	BeskarYUMBinary    = "beskar-yum"
 	BeskarStaticBinary = "beskar-static"
+	BeskarOSTreeBinary = "beskar-ostree"
 )
 
 var binaries = map[string]binaryConfig{
@@ -101,7 +104,7 @@ var binaries = map[string]binaryConfig{
 		},
 		useProto: true,
 		// NOTE: restore in case alpine createrepo_c package is broken again
-		//baseImage: "debian:bullseye-slim",
+		// baseImage: "debian:bullseye-slim",
 		integrationTest: &integrationTest{
 			isPlugin: true,
 			envs: map[string]string{
@@ -130,6 +133,46 @@ var binaries = map[string]binaryConfig{
 			},
 		},
 	},
+	BeskarOSTreeBinary: {
+		configFiles: map[string]string{
+			"internal/plugins/ostree/pkg/config/default/beskar-ostree.yaml": "/etc/beskar/beskar-ostree.yaml",
+		},
+		genAPI: &genAPI{
+			path:          "pkg/plugins/ostree/api/v1",
+			filename:      "api.go",
+			interfaceName: "OSTree",
+		},
+		useProto: true,
+		execStmts: [][]string{
+			{
+				"apk", "add", "ostree", "ostree-dev",
+			},
+		},
+		buildExecStmts: [][]string{
+			{
+				// pkg-config is needed to compute CFLAGS
+				"apk", "add", "pkgconfig",
+			},
+			{
+				// Install gcc. Could have installed gc directly but this seems to be the recommended way for alpine.
+				"apk", "add", "build-base",
+			},
+			{
+				// Install ostree development libraries
+				"apk", "add", "ostree", "ostree-dev",
+			},
+		},
+		buildEnv: map[string]string{
+			"CGO_ENABLED": "1",
+		},
+		excludedPlatforms: map[dagger.Platform]struct{}{
+			"linux/arm64":   {},
+			"linux/s390x":   {},
+			"linux/ppc64le": {},
+			"linux/arm/v6":  {},
+			"linux/arm/v7":  {},
+		},
+	},
 }
 
 type Build mg.Namespace
@@ -170,6 +213,7 @@ func (b Build) Plugins(ctx context.Context) {
 		ctx,
 		mg.F(b.Plugin, BeskarYUMBinary),
 		mg.F(b.Plugin, BeskarStaticBinary),
+		mg.F(b.Plugin, BeskarOSTreeBinary),
 	)
 }
 
@@ -274,6 +318,14 @@ func (b Build) build(ctx context.Context, name string) error {
 			golang = golang.WithEnvVariable(key, value)
 		}
 
+		for key, value := range binaryConfig.buildEnv {
+			golang = golang.WithEnvVariable(key, value)
+		}
+
+		for _, execStmt := range binaryConfig.buildExecStmts {
+			golang = golang.WithExec(execStmt)
+		}
+
 		path := filepath.Join("/output", binary)
 		inputCmd := filepath.Join("cmd", name)
 

diff --git a/build/mage/lint.go b/build/mage/lint.go
@@ -36,6 +36,18 @@ func (Lint) Go(ctx context.Context) error {
 		WithWorkdir("/src").
 		With(goCache(client))
 
+	// Set up the environment for the linter per the settings of each binary.
+	// This could lead to conflicts if the binaries have different settings.
+	for _, config := range binaries {
+		for key, value := range config.buildEnv {
+			golangciLint = golangciLint.WithEnvVariable(key, value)
+		}
+
+		for _, execStmt := range config.buildExecStmts {
+			golangciLint = golangciLint.WithExec(execStmt)
+		}
+	}
+
 	golangciLint = golangciLint.WithExec([]string{
 		"golangci-lint", "-v", "run", "--modules-download-mode", "readonly", "--timeout", "5m",
 	})

diff --git a/build/mage/test.go b/build/mage/test.go
@@ -2,11 +2,10 @@ package mage
 
 import (
 	"context"
-	"fmt"
-	"strings"
-
 	"dagger.io/dagger"
+	"fmt"
 	"github.com/magefile/mage/mg"
+	"strings"
 )
 
 type Test mg.Namespace
@@ -29,8 +28,20 @@ func (Test) Unit(ctx context.Context) error {
 		WithWorkdir("/src").
 		With(goCache(client))
 
+	for _, config := range binaries {
+		for key, value := range config.buildEnv {
+			unitTest = unitTest.WithEnvVariable(key, value)
+		}
+
+		for _, execStmt := range config.buildExecStmts {
+			unitTest = unitTest.WithExec(execStmt)
+		}
+	}
+
 	unitTest = unitTest.WithExec([]string{
 		"go", "test", "-v", "-count=1", "./...",
+	}, dagger.ContainerWithExecOpts{
+		InsecureRootCapabilities: true,
 	})
 
 	return printOutput(ctx, unitTest)

diff --git a/charts/beskar-ostree/.helmignore b/charts/beskar-ostree/.helmignore
@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/
diff --git a/charts/beskar-ostree/Chart.yaml b/charts/beskar-ostree/Chart.yaml
@@ -0,0 +1,12 @@
+apiVersion: v2
+description: A Helm chart for Beskar OSTree Repository Plugin
+name: beskar-ostree
+version: 0.0.1
+appVersion: 0.0.1
+home: https://github.com/ctrliq/beskar
+maintainers:
+- email: [email protected]
+  name: CtrlIQ Inc.
+  url: https://github.com/ctrliq/beskar
+sources:
+- https://github.com/ctrliq/beskar
-Original file line number
+Diff line change
@@ Expand Up / @@ -3,3 +3,7 @@ build/output @@
     .envrc.local
     vendor
     go.work.sum
+    */.idea
+    /.idea/