Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(docs): update rbac guide for shared cluster #3443

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

chore(docs): update rbac guide for shared cluster #3443

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
305 changes: 135 additions & 170 deletions docs/advanced/rbac-config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,173 +3,138 @@ order: 4
title: Minimal RBAC Configuration for Development Clusters
---

The following describes the minimal RBAC roles and permissions required for day-to-day use by developers for Garden when using the `kubernetes` plugin. These should be created along with the kubeconfig/kubecontext for the user in their namespace, replacing the `<username>`, `<service-accounts-namespace>` and `<project-namespace>` values as appropriate.

```
---
# The user service account
apiVersion: v1
kind: ServiceAccount
metadata:
name: user-<username>
namespace: <service-accounts-namespace>

---

# Project namespaces
apiVersion: v1
kind: Namespace
metadata:
name: <project-namespace>
# Some required annotations
annotations:
garden.io/version: "0.11.3"

---

# Allow reading namespaces and persistent volumes, which are cluster-scoped
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: user-<username>
rules:
- apiGroups: [""]
resources: ["namespaces", "persistentvolumes"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: user-<username>
namespace: <service-accounts-namespace>
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: user-<username>
subjects:
- namespace: <service-accounts-namespace>
kind: ServiceAccount
name: user-<username>

---

# Full permissions within the <project-namespace>
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: <project-namespace>
namespace: <project-namespace>
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: <project-namespace>
namespace: <project-namespace>
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: <project-namespace>
subjects:
- namespace: <service-accounts-namespace>
kind: ServiceAccount
name: user-<username>

---

# Required access for the garden-system namespace
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: garden-system
name: user-<username>-common
rules:
# Allow port forward to build-sync services
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
# Note: An upcoming release will remove the requirement
- apiGroups: [""]
resources: ["pods/portforward"]
verbs: ["get", "list", "create"]
# Allow storing and reading test results
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list", "create"]
# Allow getting status of shared services
- apiGroups: [""]
resources:
- "configmaps"
- "services"
- "serviceaccounts"
- "persistentvolumeclaims"
- "pods/log"
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["configmaps", "services", "serviceaccounts"]
verbs: ["get", "list"]
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["roles", "rolebindings"]
verbs: ["get", "list"]
- apiGroups: ["extensions", "apps"]
resources: ["deployments", "daemonsets"]
verbs: ["get", "list"]
# Note: We do not store anything sensitive in secrets, aside from registry auth,
# which users anyway need to be able to read and push built images.
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: user-<username>-common
namespace: garden-system
roleRef:
kind: Role
name: user-<username>-common
apiGroup: ""
subjects:
- namespace: <service-accounts-namespace>
kind: ServiceAccount
name: user-<username>

---

# Allow building with kaniko in-cluster
# Note: An upcoming release will remove this required role
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: garden-system
name: user-<username>-kaniko
rules:
- apiGroups: [""]
resources: ["pods"]
verbs:
- "get"
- "list"
- "create"
- "delete"

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: user-<username>-kaniko
namespace: garden-system
roleRef:
kind: Role
name: user-<username>-kaniko
apiGroup: ""
subjects:
- namespace: <service-accounts-namespace>
kind: ServiceAccount
name: user-<username>
```
If there is no dedicated development k8s cluster, the following configurations can be used to limit the permissions, of developers on a shared cluster using RBAC (Role-Based Access Control), to specific dev namespaces only. This can help to ensure that developers have the appropriate access to resources on the shared cluster.

> We have verified to ensure that the following configuration is effective for an Azure IAM group that includes all developers. Similar approach can be followed for any other k8s group mapping. (Contributions, to this doc, are welcome)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer this guide to cover AWS, Azure and GCP. I think the only places where the configuration differs is the mapping between IAM and Kubernetes RBAC.

Copy link
Author

@ghost ghost Dec 15, 2022 •
edited by ghost
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before I go ahead and make changes, I feel we are repeating ourselves especially with this doc.
In my opinion an optimal solution, to make this doc to be valid for all three major* cloud providers, is not keeping it in git book but rather having it as a README in three different example projects (per cloud provider) and having cross reference to that README in the gitbook doc. This way we can also serve with updated config or so. It also makes quite easier to understand when we look at an example than going through the doc. Wdyt?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like a good idea too. My thinking was that we get the differences covered here and the rest can more or less stay as is. But the example projects are also a good idea, i will leave that to you :-)


In order to achieve scoped permissions for desired namespaces only, we are using benefits of [hierarchical namespaces](https://kubernetes.io/blog/2020/08/14/introducing-hierarchical-namespaces/)

1. Install hierarchical namespace controller in the k8s cluster.
```
kubectl apply -f https://github.com/kubernetes-sigs/hierarchical-namespaces/releases/download/v1.0.0/default.yaml
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we show how to get the latest version of the controller and use that version instead of hardcoding it? Afraid that the version will be outdated pretty soon.

```
2. Create an Azure IAM group for the developers who are allowed to deploy to dev namespaces only. Add all the Users to the group and note the group ID.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
2. Create an Azure IAM group for the developers who are allowed to deploy to dev namespaces only. Add all the Users to the group and note the group ID.
2. Next we will need to take care to map your Cloud Provider's IAM groups or users to Kubernetes RBAC. This differs slightly between different cloud providers. The following should give you a brief overview, but might not cover all use cases. Please consider your Cloud Provider's documentation for further info.
For Azure: Create an IAM group for the developers who are allowed to deploy to dev namespaces only. Add all the Users to the group and note the group ID. This group ID can be directly used in `roleBindings` and `clusterRoleBindings`. For more info see [here](https://learn.microsoft.com/en-us/azure/aks/azure-ad-rbac).
For AWS: Add your users to the [aws-auth configmap](https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html) and add them to an `apiGroup` that you want to use, we will call it `developers` here. This will be your group ID going forward.
For GCP: You can [create a Google group or add IAM users](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control#rolebinding) directly to `roleBindings` and `clusterRoleBindings` We recommend creating a group, as adding single users to several `roleBindings` and `clusterRoleBindings` can become rather tedious.


3. Create a root namespace (`webdev-root` for example) which will have a role that can be inherited by sub-namespaces (all dev namespaces).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we could call it just dev-root? Then its also fine for non-webdev projects

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

another option could be garden-dev-root


4. Create a clusterRole and ClusterRoleBinding.
```
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
```
```
# Allow reading namespaces and persistent volumes, which are cluster-scoped

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: namespacesAndPVCsCreateAndList
rules:
- apiGroups:
- ""
resources:
- namespaces
- persistenvolumes
verbs:
- create
- get
- list
- watch
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: webdevsClusterRoleBinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: namespacesAndPVCsCreateAndList
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: <Azure IAM group ID>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: <Azure IAM group ID>
name: <group created in step 2>

```
5. Create a Role and RoleBinding in the root namespace (`webdev-root` for example). This will allow all the members of group to have full access to all sub-namespaces.
```
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: webdevNamespacesFullAccess
namespace: webdev-root
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: webdevsRoleBinding
namespace: webdev-root
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: webdevNamespaceFullAccess
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: <Azure IAM group ID>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: <Azure IAM group ID>
name: <group created in step 2>

```
6. Create Role and RoleBinding for providing full access to `garden-system namespace`.
```
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: gardenSystemFullAccess
namespace: garden-system
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: gardenSystemFullAccessRoleBinding
namespace: garden-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: gardenSystemFullAccess
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: <Azure IAM group ID>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: <Azure IAM group ID>
name: <group created in step 2>

```

7. Update project.garden.yaml in order to make sure that sub-namespaces have an annotation (`hnc.x-k8s.io/subnamespace-of: webdev-root` for example) of root namespace where permissions are inherited from.
```
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think providing an example garden project is a great idea, but let's remove the unneeded parts like the local kubernetes env and providers and everything else that is not strictly required here.

kind: Project
name: demo-project

# defaultEnvironment: "remote" # Uncomment if the remote environment is preferred to be the default for this project.

environments:
- name: local
- name: remote
defaultNamespace: webdev-${var.userId}
providers:
- name: local-kubernetes
environments: [local]
- name: kubernetes
environments: [remote]
# Replace the below values as appropriate
context: rbac-test
namespace:
name: webdev-${var.userId}
annotations:
hnc.x-k8s.io/subnamespace-of: webdev-root
defaultHostname: webdev-${var.userId}.sys.garden
buildMode: kaniko
deploymentRegistry:
hostname: eu.gcr.io # <- set this according to the region in which k8s cluster runs
namespace: garden-demo-324810
imagePullSecrets:
# Make sure this matches with the name and namespace of the imagePullSecret created to authenticate with the registry (if needed)
- name: gcr-json-key
namespace: webdev-root
variables:
userId: ${local.username}
```