Skip to content

Commit

Permalink
Add kafka connector version concepts
Browse files Browse the repository at this point in the history 
Signed-off-by: obaydullahmhs <[email protected]>
  • Loading branch information
@obaydullahmhs
obaydullahmhs committed May 3, 2024
1 parent f406bc9 commit 11ebe89
Show file tree
Hide file tree
Showing 15 changed files with 570 additions and 119 deletions.
9 changes: 5 additions & 4 deletions docs/guides/kafka/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,13 @@ aliases:
|----------------------------------------------------------------|
| Clustering - Combined (shared controller and broker nodes) |
| Clustering - Topology (dedicated controllers and broker nodes) |
| Kafka Connect Cluster |
| Connectors |
| Kafka Connect Cluster |
| Connectors |
| Custom Docker Image |
| Authentication & Authorization |
| Persistent Volume |
| Custom Volume |
| TLS: using ( [Cert Manager](https://cert-manager.io/docs/) ) |
| TLS: using ( [Cert Manager](https://cert-manager.io/docs/) ) |
| Reconfigurable Health Checker |
| Externally manageable Auth Secret |
| Monitoring with Prometheus & Grafana |
Expand All @@ -39,10 +39,11 @@ KubeDB supports The following Kafka versions. Supported version are applicable f
- `3.3.2`
- `3.4.1`
- `3.5.1`
- `3.5.2`
- `3.6.0`
- `3.6.1`

> The listed KafkaVersions are tested and provided as a part of the installation process (ie. catalog chart), but you are open to create your own [KafkaVersion](/docs/guides/kafka/concepts/catalog.md) object with your custom Kafka image.
> The listed KafkaVersions are tested and provided as a part of the installation process (ie. catalog chart), but you are open to create your own [KafkaVersion](/docs/guides/kafka/concepts/kafkaversion.md) object with your custom Kafka image.
## Lifecycle of Kafka Object

Expand Down
18 changes: 9 additions & 9 deletions docs/guides/kafka/cli/cli.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,15 +47,15 @@ cat kafka.yaml | kubectl create -f -
```bash
$ kubectl get kafka
NAME TYPE VERSION STATUS AGE
kafka kubedb.com/v1alpha2 3.4.0 Ready 36m
kafka kubedb.com/v1alpha2 3.6.1 Ready 36m
```

You can also use short-form (`kf`) for kafka CR.

```bash
$ kubectl get kf
NAME TYPE VERSION STATUS AGE
kafka kubedb.com/v1alpha2 3.4.0 Ready 36m
kafka kubedb.com/v1alpha2 3.6.1 Ready 36m
```

To get YAML of an object, use `--output=yaml` or `-oyaml` flag. Use `-n` flag for referring namespace.
Expand All @@ -67,7 +67,7 @@ kind: Kafka
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"kubedb.com/v1alpha2","kind":"Kafka","metadata":{"annotations":{},"name":"kafka","namespace":"demo"},"spec":{"authSecret":{"name":"kafka-admin-cred"},"enableSSL":true,"healthChecker":{"failureThreshold":3,"periodSeconds":20,"timeoutSeconds":10},"keystoreCredSecret":{"name":"kafka-keystore-cred"},"storageType":"Durable","terminationPolicy":"DoNotTerminate","tls":{"certificates":[{"alias":"server","secretName":"kafka-server-cert"},{"alias":"client","secretName":"kafka-client-cert"}],"issuerRef":{"apiGroup":"cert-manager.io","kind":"Issuer","name":"kafka-ca-issuer"}},"topology":{"broker":{"replicas":3,"resources":{"limits":{"memory":"1Gi"},"requests":{"cpu":"500m","memory":"1Gi"}},"storage":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"standard"},"suffix":"broker"},"controller":{"replicas":3,"resources":{"limits":{"memory":"1Gi"},"requests":{"cpu":"500m","memory":"1Gi"}},"storage":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"standard"},"suffix":"controller"}},"version":"3.4.0"}}
{"apiVersion":"kubedb.com/v1alpha2","kind":"Kafka","metadata":{"annotations":{},"name":"kafka","namespace":"demo"},"spec":{"authSecret":{"name":"kafka-admin-cred"},"enableSSL":true,"healthChecker":{"failureThreshold":3,"periodSeconds":20,"timeoutSeconds":10},"keystoreCredSecret":{"name":"kafka-keystore-cred"},"storageType":"Durable","terminationPolicy":"DoNotTerminate","tls":{"certificates":[{"alias":"server","secretName":"kafka-server-cert"},{"alias":"client","secretName":"kafka-client-cert"}],"issuerRef":{"apiGroup":"cert-manager.io","kind":"Issuer","name":"kafka-ca-issuer"}},"topology":{"broker":{"replicas":3,"resources":{"limits":{"memory":"1Gi"},"requests":{"cpu":"500m","memory":"1Gi"}},"storage":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"standard"},"suffix":"broker"},"controller":{"replicas":3,"resources":{"limits":{"memory":"1Gi"},"requests":{"cpu":"500m","memory":"1Gi"}},"storage":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"standard"},"suffix":"controller"}},"version":"3.6.1"}}
creationTimestamp: "2023-03-29T07:01:29Z"
finalizers:
- kubedb.com
Expand Down Expand Up @@ -136,7 +136,7 @@ spec:
storage: 1Gi
storageClassName: standard
suffix: controller
version: 3.4.0
version: 3.6.1
status:
conditions:
- lastTransitionTime: "2023-03-29T07:01:29Z"
Expand Down Expand Up @@ -181,7 +181,7 @@ $ kubectl get kf kafka -n demo -ojson
"kind": "Kafka",
"metadata": {
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"kubedb.com/v1alpha2\",\"kind\":\"Kafka\",\"metadata\":{\"annotations\":{},\"name\":\"kafka\",\"namespace\":\"demo\"},\"spec\":{\"authSecret\":{\"name\":\"kafka-admin-cred\"},\"enableSSL\":true,\"healthChecker\":{\"failureThreshold\":3,\"periodSeconds\":20,\"timeoutSeconds\":10},\"keystoreCredSecret\":{\"name\":\"kafka-keystore-cred\"},\"storageType\":\"Durable\",\"terminationPolicy\":\"DoNotTerminate\",\"tls\":{\"certificates\":[{\"alias\":\"server\",\"secretName\":\"kafka-server-cert\"},{\"alias\":\"client\",\"secretName\":\"kafka-client-cert\"}],\"issuerRef\":{\"apiGroup\":\"cert-manager.io\",\"kind\":\"Issuer\",\"name\":\"kafka-ca-issuer\"}},\"topology\":{\"broker\":{\"replicas\":3,\"resources\":{\"limits\":{\"memory\":\"1Gi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"1Gi\"}},\"storage\":{\"accessModes\":[\"ReadWriteOnce\"],\"resources\":{\"requests\":{\"storage\":\"1Gi\"}},\"storageClassName\":\"standard\"},\"suffix\":\"broker\"},\"controller\":{\"replicas\":3,\"resources\":{\"limits\":{\"memory\":\"1Gi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"1Gi\"}},\"storage\":{\"accessModes\":[\"ReadWriteOnce\"],\"resources\":{\"requests\":{\"storage\":\"1Gi\"}},\"storageClassName\":\"standard\"},\"suffix\":\"controller\"}},\"version\":\"3.4.0\"}}\n"
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"kubedb.com/v1alpha2\",\"kind\":\"Kafka\",\"metadata\":{\"annotations\":{},\"name\":\"kafka\",\"namespace\":\"demo\"},\"spec\":{\"authSecret\":{\"name\":\"kafka-admin-cred\"},\"enableSSL\":true,\"healthChecker\":{\"failureThreshold\":3,\"periodSeconds\":20,\"timeoutSeconds\":10},\"keystoreCredSecret\":{\"name\":\"kafka-keystore-cred\"},\"storageType\":\"Durable\",\"terminationPolicy\":\"DoNotTerminate\",\"tls\":{\"certificates\":[{\"alias\":\"server\",\"secretName\":\"kafka-server-cert\"},{\"alias\":\"client\",\"secretName\":\"kafka-client-cert\"}],\"issuerRef\":{\"apiGroup\":\"cert-manager.io\",\"kind\":\"Issuer\",\"name\":\"kafka-ca-issuer\"}},\"topology\":{\"broker\":{\"replicas\":3,\"resources\":{\"limits\":{\"memory\":\"1Gi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"1Gi\"}},\"storage\":{\"accessModes\":[\"ReadWriteOnce\"],\"resources\":{\"requests\":{\"storage\":\"1Gi\"}},\"storageClassName\":\"standard\"},\"suffix\":\"broker\"},\"controller\":{\"replicas\":3,\"resources\":{\"limits\":{\"memory\":\"1Gi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"1Gi\"}},\"storage\":{\"accessModes\":[\"ReadWriteOnce\"],\"resources\":{\"requests\":{\"storage\":\"1Gi\"}},\"storageClassName\":\"standard\"},\"suffix\":\"controller\"}},\"version\":\"3.6.1\"}}\n"
},
"creationTimestamp": "2023-03-29T07:01:29Z",
"finalizers": [
Expand Down Expand Up @@ -282,7 +282,7 @@ $ kubectl get kf kafka -n demo -ojson
"suffix": "controller"
}
},
"version": "3.4.0"
"version": "3.6.1"
},
"status": {
"conditions": [
Expand Down Expand Up @@ -342,15 +342,15 @@ demo pod/kafka-broker-1 1/1 Running 0 45m 10.24
demo pod/kafka-broker-2 1/1 Running 0 45m 10.244.0.57 kind-control-plane <none> <none>
demo pod/kafka-controller-0 1/1 Running 0 45m 10.244.0.51 kind-control-plane <none> <none>
demo pod/kafka-controller-1 1/1 Running 0 45m 10.244.0.55 kind-control-plane <none> <none>
demo pod/kafka-controller-2 1/1 Running 3 (45m ago) 45m 10.244.0.58 kind-control-plane <none> <none>
demo pod/kafka-controller-2 1/1 Running 0 45m 10.244.0.58 kind-control-plane <none> <none>
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
demo service/kafka-broker ClusterIP None <none> 9092/TCP,29092/TCP 46m app.kubernetes.io/instance=kafka,app.kubernetes.io/managed-by=kubedb.com,app.kubernetes.io/name=kafkas.kubedb.com,kubedb.com/role=broker
demo service/kafka-controller ClusterIP None <none> 9093/TCP 46m app.kubernetes.io/instance=kafka,app.kubernetes.io/managed-by=kubedb.com,app.kubernetes.io/name=kafkas.kubedb.com,kubedb.com/role=controller
NAMESPACE NAME READY AGE CONTAINERS IMAGES
demo statefulset.apps/kafka-broker 3/3 45m kafka docker.io/kubedb/kafka-kraft:3.4.0@sha256:f059db2929e3cfe388f50e82e168a9ce94b012e413e056eda2838df48632048a
demo statefulset.apps/kafka-controller 3/3 45m kafka docker.io/kubedb/kafka-kraft:3.4.0@sha256:f059db2929e3cfe388f50e82e168a9ce94b012e413e056eda2838df48632048a
demo statefulset.apps/kafka-broker 3/3 45m kafka ghcr.io/appscode-images/kafka-kraft:3.6.1@sha256:e251d3c0ceee0db8400b689e42587985034852a8a6c81b5973c2844e902e6d11
demo statefulset.apps/kafka-controller 3/3 45m kafka ghcr.io/appscode-images/kafka-kraft:3.6.1@sha256:e251d3c0ceee0db8400b689e42587985034852a8a6c81b5973c2844e902e6d11
NAMESPACE NAME TYPE VERSION AGE
demo appbinding.appcatalog.appscode.com/kafka kubedb.com/kafka 3.4.0 45m
Expand Down
36 changes: 18 additions & 18 deletions docs/guides/kafka/clustering/combined-cluster/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ demo Active 9s
## Create Standalone Kafka Cluster

Here, we are going to create a standalone (ie. `replicas: 1`) Kafka cluster in Kraft mode. For this demo, we are going to provision kafka version `3.3.2`. To learn more about Kafka CR, visit [here](/docs/guides/kafka/concepts/kafka.md). visit [here](/docs/guides/kafka/concepts/catalog.md) to learn more about KafkaVersion CR.
Here, we are going to create a standalone (i.e. `replicas: 1`) Kafka cluster in Kraft mode. For this demo, we are going to provision kafka version `3.6.1`. To learn more about Kafka CR, visit [here](/docs/guides/kafka/concepts/kafka.md). visit [here](/docs/guides/kafka/concepts/kafkaversion.md) to learn more about KafkaVersion CR.

```yaml
apiVersion: kubedb.com/v1alpha2
Expand All @@ -47,7 +47,7 @@ metadata:
namespace: demo
spec:
replicas: 1
version: 3.3.2
version: 3.6.1
storage:
accessModes:
- ReadWriteOnce
Expand All @@ -71,12 +71,12 @@ Watch the bootstrap progress:
```bash
$ kubectl get kf -n demo -w
NAME TYPE VERSION STATUS AGE
kafka-standalone kubedb.com/v1alpha2 3.3.2 Provisioning 8s
kafka-standalone kubedb.com/v1alpha2 3.3.2 Provisioning 14s
kafka-standalone kubedb.com/v1alpha2 3.3.2 Provisioning 35s
kafka-standalone kubedb.com/v1alpha2 3.3.2 Provisioning 35s
kafka-standalone kubedb.com/v1alpha2 3.3.2 Provisioning 36s
kafka-standalone kubedb.com/v1alpha2 3.3.2 Ready 41s
kafka-standalone kubedb.com/v1alpha2 3.6.1 Provisioning 8s
kafka-standalone kubedb.com/v1alpha2 3.6.1 Provisioning 14s
kafka-standalone kubedb.com/v1alpha2 3.6.1 Provisioning 35s
kafka-standalone kubedb.com/v1alpha2 3.6.1 Provisioning 35s
kafka-standalone kubedb.com/v1alpha2 3.6.1 Provisioning 36s
kafka-standalone kubedb.com/v1alpha2 3.6.1 Ready 41s
```

Hence, the cluster is ready to use.
Expand All @@ -94,7 +94,7 @@ NAME READY AGE
statefulset.apps/kafka-standalone 1/1 8m56s

NAME TYPE VERSION AGE
appbinding.appcatalog.appscode.com/kafka-standalone kubedb.com/kafka 3.3.2 8m56s
appbinding.appcatalog.appscode.com/kafka-standalone kubedb.com/kafka 3.6.1 8m56s

NAME TYPE DATA AGE
secret/kafka-standalone-admin-cred kubernetes.io/basic-auth 2 8m59s
Expand All @@ -116,7 +116,7 @@ metadata:
namespace: demo
spec:
replicas: 3
version: 3.3.2
version: 3.6.1
storage:
accessModes:
- ReadWriteOnce
Expand All @@ -139,12 +139,12 @@ Watch the bootstrap progress:

```bash
$ kubectl get kf -n demo -w
kafka-multinode kubedb.com/v1alpha2 3.3.2 Provisioning 9s
kafka-multinode kubedb.com/v1alpha2 3.3.2 Provisioning 14s
kafka-multinode kubedb.com/v1alpha2 3.3.2 Provisioning 18s
kafka-multinode kubedb.com/v1alpha2 3.3.2 Provisioning 2m6s
kafka-multinode kubedb.com/v1alpha2 3.3.2 Provisioning 2m8s
kafka-multinode kubedb.com/v1alpha2 3.3.2 Ready 2m14s
kafka-multinode kubedb.com/v1alpha2 3.6.1 Provisioning 9s
kafka-multinode kubedb.com/v1alpha2 3.6.1 Provisioning 14s
kafka-multinode kubedb.com/v1alpha2 3.6.1 Provisioning 18s
kafka-multinode kubedb.com/v1alpha2 3.6.1 Provisioning 2m6s
kafka-multinode kubedb.com/v1alpha2 3.6.1 Provisioning 2m8s
kafka-multinode kubedb.com/v1alpha2 3.6.1 Ready 2m14s
```

Hence, the cluster is ready to use.
Expand All @@ -164,7 +164,7 @@ NAME READY AGE
statefulset.apps/kafka-multinode 3/3 6m2s

NAME TYPE VERSION AGE
appbinding.appcatalog.appscode.com/kafka-multinode kubedb.com/kafka 3.3.2 6m2s
appbinding.appcatalog.appscode.com/kafka-multinode kubedb.com/kafka 3.6.1 6m2s

NAME TYPE DATA AGE
secret/kafka-multinode-admin-cred kubernetes.io/basic-auth 2 6m7s
Expand Down Expand Up @@ -310,6 +310,6 @@ $ kubectl delete namespace demo
- Deploy [dedicated topology cluster](/docs/guides/kafka/clustering/topology-cluster/index.md) for Apache Kafka
- Monitor your Kafka cluster with KubeDB using [`out-of-the-box` Prometheus operator](/docs/guides/kafka/monitoring/using-prometheus-operator.md).
- Detail concepts of [Kafka object](/docs/guides/kafka/concepts/kafka.md).
- Detail concepts of [KafkaVersion object](/docs/guides/kafka/concepts/catalog.md).
- Detail concepts of [KafkaVersion object](/docs/guides/kafka/concepts/kafkaversion.md).
- Learn to use KubeDB managed Kafka objects using [CLIs](/docs/guides/kafka/cli/cli.md).
- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md).
26 changes: 13 additions & 13 deletions docs/guides/kafka/clustering/topology-cluster/index.md
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
title: Kafka Topology Cluster
menu:
docs_{{ .version }}:
identifier: kf-topology-cluster
name: Topology Cluster
parent: kf-clustering
weight: 20
docs_{{ .version }}:
identifier: kf-topology-cluster
name: Topology Cluster
parent: kf-clustering
weight: 20
menu_name: docs_{{ .version }}
section_menu_id: guides
---
Expand Down Expand Up @@ -80,7 +80,7 @@ issuer.cert-manager.io/kafka-ca-issuer created

### Provision TLS secure Kafka

For this demo, we are going to provision kafka version `3.3.2` with 3 controllers and 3 brokers. To learn more about Kafka CR, visit [here](/docs/guides/kafka/concepts/kafka.md). visit [here](/docs/guides/kafka/concepts/catalog.md) to learn more about KafkaVersion CR.
For this demo, we are going to provision kafka version `3.6.1` with 3 controllers and 3 brokers. To learn more about Kafka CR, visit [here](/docs/guides/kafka/concepts/kafka.md). visit [here](/docs/guides/kafka/concepts/kafkaversion.md) to learn more about KafkaVersion CR.

```yaml
apiVersion: kubedb.com/v1alpha2
Expand All @@ -89,7 +89,7 @@ metadata:
name: kafka-prod
namespace: demo
spec:
version: 3.3.2
version: 3.6.1
enableSSL: true
tls:
issuerRef:
Expand Down Expand Up @@ -131,10 +131,10 @@ Watch the bootstrap progress:
```bash
$ kubectl get kf -n demo -w
NAME TYPE VERSION STATUS AGE
kafka-prod kubedb.com/v1alpha2 3.3.2 Provisioning 6s
kafka-prod kubedb.com/v1alpha2 3.3.2 Provisioning 14s
kafka-prod kubedb.com/v1alpha2 3.3.2 Provisioning 50s
kafka-prod kubedb.com/v1alpha2 3.3.2 Ready 68s
kafka-prod kubedb.com/v1alpha2 3.6.1 Provisioning 6s
kafka-prod kubedb.com/v1alpha2 3.6.1 Provisioning 14s
kafka-prod kubedb.com/v1alpha2 3.6.1 Provisioning 50s
kafka-prod kubedb.com/v1alpha2 3.6.1 Ready 68s
```

Hence, the cluster is ready to use.
Expand All @@ -159,7 +159,7 @@ statefulset.apps/kafka-prod-broker 3/3 4m10s
statefulset.apps/kafka-prod-controller 3/3 4m8s
NAME TYPE VERSION AGE
appbinding.appcatalog.appscode.com/kafka-prod kubedb.com/kafka 3.3.2 4m8s
appbinding.appcatalog.appscode.com/kafka-prod kubedb.com/kafka 3.6.1 4m8s
NAME TYPE DATA AGE
secret/kafka-prod-admin-cred kubernetes.io/basic-auth 2 4m14s
Expand Down Expand Up @@ -313,6 +313,6 @@ $ kubectl delete namespace demo
- Deploy [dedicated topology cluster](/docs/guides/kafka/clustering/topology-cluster/index.md) for Apache Kafka
- Monitor your Kafka cluster with KubeDB using [`out-of-the-box` Prometheus operator](/docs/guides/kafka/monitoring/using-prometheus-operator.md).
- Detail concepts of [Kafka object](/docs/guides/kafka/concepts/kafka.md).
- Detail concepts of [KafkaVersion object](/docs/guides/kafka/concepts/catalog.md).
- Detail concepts of [KafkaVersion object](/docs/guides/kafka/concepts/kafkaversion.md).
- Learn to use KubeDB managed Kafka objects using [CLIs](/docs/guides/kafka/cli/cli.md).
- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md).
Loading

0 comments on commit 11ebe89

Please sign in to comment.