tap-nsp-gitops

This repo contains instructions on how to deploy TAP using GitOps on Google Cloud and create resources in your Developer namespaces on your TAP cluster using Namespace Provisioner (NSP) in full GitOps mode.

Usage

Fork/Clone this repo and update your fork with your changes. This repo serves as a base to give users a headstart.

What's used?

This tutorial is using the following:

• Tanzu Application Platform (TAP) 1.5 RC
• Namespace Provisioner (NSP) for TAP for provisioning resources in our developer namespaces.
  • Namespace Provisioner is installed as part of TAP 1.5 profile installation.
  • Creates 2 namespaces dev and qa.
  • Installs Tekton Pipelines for 3 languages (java, python and golang).
  • Installs Snyk scanner in addition to Grype scanner that is installed OOTB.
  • Gets the Snyk scanner secret from Google Secrets Manager using External Secrets Operator.
  • Installs RabbitMQ operator and Topology operator on the TAP cluster.
• Google Secrets Manager (GSM) for storing all our sensitive secrets.
• External Secrets Operator (ESO) to pull the secrets from Google Secrets Manager into our TAP Cluster.

GKE cluster setup

For this setup, we need a

• GKE cluster and a kubeconfig to it. If using gcloud command, kubeconfig is automatically added to ~/.kube/config.
• Workload identity enabled on the GKE cluster.
• Install Cluster Essentials on it.

You can use a method of your choice to get this infrastructure up, I will be using tappr, A CLI I made that helps in K8s cluster creation and TAP installation and management.

if using tappr (need version >=0.13.0), run the following command to create a GKE cluster in the project where your gcloud is pointing to. This cluster will have workload identity enabled by default.

tappr cluster create gke --cluster-name {cluster-name} --channel RAPID
tappr tap install-cluster-essentials

Create required secrets in Google Secrets Manager

• Create a secret named sync-git-ssh containing the SSH private key that has access to this git repo and associated known hosts:

{
  "ssh-privatekey": "-----BEGIN OPENSSH PRIVATE KEY-----\nb3B................................................................tZW\nQyN................................................................6XZ\nMQA................................................................x+w\nAAA................................................................0pR\na6I..........................xQF\n-----END OPENSSH PRIVATE KEY-----\n",
  "ssh-knownhosts": "github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl"
}

• Next, a secret with creds for Tanzu Network so we can install TAP. Create a secret tanzunet-dockerconfig in Google secrets manager with following value:

{
  "auths": {
    "registry.tanzu.vmware.com": {
      "username": "[email protected]",
      "password": ""
    }
  }
}

• Finally, create a YAML format secret called sensitive-values in Google Secrets manager with all the sensitive values you have for your setup. Bare-minimum is the registry credentials that has pull/push access to a repo.

# Use your creds
shared:
 image_registry:
   project_path: "gcr.io/adhol-playground/tap"
   username: "_json_key"
   password: |
     { ... put actual service account JSON, here ... }

Install TAP from this GitOps repo

The following set of commands will do the following:

• Create a sync app in tanzu_sync that

  • Install External Secrets Operator
  • Installs TAP Package Repository and required Sync secrets
  • Installs required Tanzu Network secret
  • Install TAP from the configs in the GitOps repository.

• Creates GCP IAM service accounts for Sync app and setup the Kubernetes Service Accounts created by the Sync app with Workload identity pointing to those GCP IAM service accounts.

• Pulls the secrets created in previous setup from Google secrets manager using the External secrets operator and make it available in the cluster for the sync app.

• Starts TAP installation based on the values provided in the clusters/tap15/cluster-config/values/tap-values.yaml.

• Setup Required Environment variables Set this following env vars which are required to bootstrap the cluster and create the sync app which enables the whole GitOps TAP installer workflow.

# Update this with your registry if you relocated TAP packages
export INSTALL_REGISTRY_HOSTNAME=registry.tanzu.vmware.com
# Install registry creds. If you have not relocated TAP packages, use Tanzu Network Creds
export [email protected]
export INSTALL_REGISTRY_PASSWORD=""
# KUBECONTEXT of the k8s cluster created above
export KAPP_KUBECONFIG_CONTEXT=""
# GCP project information
export GCP_PROJECT=adhol-playground
# GKE cluster information
export GKE_CLUSTER_NAME=""
export GKE_CLUSTER_REGION=us-east4
# TAP information
export TAP_VERSION=1.5.0-build.37

• Update the configs in the Git Repo to match your cluster

cd clusters/tap15
./tanzu-sync/scripts/bootstrap.sh
./tanzu-sync/scripts/configure.sh
git add cluster-config/ tanzu-sync/
git commit -m "fix: Configure install of TAP 1.5.0"
git push

• Create required IAM Service accounts on GCP project and Workload identity mappings

./tanzu-sync/scripts/gcp/create-sa.sh

• Deploy the Tanzu Sync App based on the config in your GitOps repo

./tanzu-sync/scripts/deploy.sh

Namespace Provisioner Setup

Next step is to update the Namespace Provisioner config so we can pull in the following additional sources which contains all the resources we want to create in our namespaces. You can choose to omit some of these resources if it does not fit your needs or update the ytt templated resource yaml files in the following folders to match your needs.

These following paths in this GitOps repo are imported with the following NSP config:

• scan folder contains
  • A lax and strict Scan Policies for Grype, Snyk, CarbonBlack and Prisma scanners in TAP 1.4.
  • All the secrets like snyk-token-secret that needs to be created for the scanners to work and these secrets are synced from our external secrets manager using ExternalSecret CR using the ESO setup that we did in the previous step.
• test folder contains
  • A single ytt templated, parameterized tekton test Pipeline to run tests on the testing_scanning supply chain.
  • The base image to use in the Tekton Pipeline and the cmd to run in the container are both passed as params in desired-namespaces ConfigMap.
• constraints folder contains
  • A default LimitRange object that is applied to all namespaces, but is overridable using the parameters in the desired-namespaces ConfigMap.
• extras folder contains all random stuff that does not really fit in one of the above and its fun for testing.

Update the TAP values with the following config for NSP.

namespace_provisioner:
  # We are setting this to false as we will manage the desired-namespaces configmap using GitOps. All the namespaces we want to create and their params are in ns folder in the https://github.com/atmandhol/tap-nsp-gitops.git repo.
  controller: false
  additional_sources:
  # Add scanners and scanpolicies
  - git:
      ref: origin/main
      subPath: scan
      url: https://github.com/atmandhol/tap-nsp-gitops.git
    path: _ytt_lib/scansetup
  # Add parameterized tekton test pipeline
  - git:
      ref: origin/main
      subPath: test
      url: https://github.com/atmandhol/tap-nsp-gitops.git
    path: _ytt_lib/testsetup
  # Add default resource constraints on your namespaces and pods
  - git:
      ref: origin/main
      subPath: constraints
      url: https://github.com/atmandhol/tap-nsp-gitops.git
    path: _ytt_lib/resourceconstraintssetup
  # Extras
  - git:
      ref: origin/main
      subPath: extras
      url: https://github.com/atmandhol/tap-nsp-gitops.git
    path: _ytt_lib/extras

Manage Desired namespaces via GitOps

We will use kapp App located here to sync the desired namespaces from our GitOps repo to our TAP cluster. Namespace provisioner also uses kapp App called provisioner and owns the desired-namespaces ConfigMap on the cluster (A behavior that will be fixed in TAP 1.5). We will add an annotation kapp.k14s.io/exists: "" to the Namespace provisioner OOTB desired-namespaces ConfigMap using a Package overlay so the 2 kapp Apps don't fight over ownership issues.

Create an Overlay secret

kubectl apply -f https://raw.githubusercontent.com/atmandhol/tap-nsp-gitops/main/tap/02-desired-namespace-overlay.yaml

Update the TAP Config with NSP Package Overlay

Add the following to the tap values config

package_overlays:
- name: namespace-provisioner
  secrets:
  - name: desired-namespaces-overlay

Create the kapp App that will sync desired-namespaces from GitOps repo

We are now ready to create our kapp App that:

• Creates the desired-namespaces ConfigMap from this GitOps Repo and owns that ConfigMap.
• Creates all the namespaces mentioned in the desired-ns-list.yaml in the ns folder in our GitOps repo.
• Creates all the Scanner Package Install for all namespace mentioned in the desired-ns-list.yaml as the PackageInstall for Scanner are not currently Cluster Scoped and not properly Namespaced either.

kubectl apply -f https://github.com/atmandhol/tap-nsp-gitops/raw/main/apps/01-desired-namespaces-sync.yaml