Add Connector

Signed-off-by: obaydullahmhs <[email protected]>
kubedb · May 6, 2024 · c888b47 · c888b47
1 parent 11ebe89
commit c888b47
Show file tree

Hide file tree

Showing 14 changed files with 271 additions and 50 deletions.
diff --git a/docs/examples/kafka/clustering/kf-multinode.yaml b/docs/examples/kafka/clustering/kf-multinode.yaml
@@ -5,7 +5,7 @@ metadata:
   namespace: demo
 spec:
   replicas: 3
-  version: 3.3.2
+  version: 3.6.1
   storage:
     accessModes:
       - ReadWriteOnce

diff --git a/docs/examples/kafka/clustering/kf-standalone.yaml b/docs/examples/kafka/clustering/kf-standalone.yaml
@@ -5,7 +5,7 @@ metadata:
   namespace: demo
 spec:
   replicas: 1
-  version: 3.3.2
+  version: 3.6.1
   storage:
     accessModes:
       - ReadWriteOnce

diff --git a/docs/examples/kafka/clustering/kf-topology.yaml b/docs/examples/kafka/clustering/kf-topology.yaml
@@ -4,7 +4,7 @@ metadata:
   name: kafka-prod
   namespace: demo
 spec:
-  version: 3.3.2
+  version: 3.6.1
   enableSSL: true
   tls:
     issuerRef:

diff --git a/docs/examples/kafka/monitoring/kf-with-monitoring.yaml b/docs/examples/kafka/monitoring/kf-with-monitoring.yaml
@@ -11,7 +11,7 @@ spec:
       name: kafka-ca-issuer
       kind: Issuer
   replicas: 3
-  version: 3.4.0
+  version: 3.6.1
   storage:
     accessModes:
       - ReadWriteOnce

diff --git a/docs/guides/kafka/README.md b/docs/guides/kafka/README.md
@@ -17,21 +17,41 @@ aliases:
 
 ## Supported Kafka Features
 
+| Features                                                                           | Kafka    | ConnectCluster |
+|------------------------------------------------------------------------------------|----------|----------------|
+| Clustering - Combined (shared controller and broker nodes)                         | &#10003; | &#45;          |
+| Clustering - Topology (dedicated controllers and broker nodes)                     | &#10003; | &#45;          |
+| Connectors                                                                         | &#45;    | &#10003;       |
+| Custom Configuration                                                               | &#10003; | &#10003;       |
+| Automated Version Update                                                           | &#10003; | &#10007;       |
+| Automatic Vertical Scaling                                                         | &#10003; | &#10007;       |
+| Automated Horizontal Scaling                                                       | &#10003; | &#10007;       |
+| Automated Volume Expansion                                                         | &#10003; | &#45;          |
+| Custom Docker Image                                                                | &#10003; | &#10003;       |
+| Authentication & Authorization                                                     | &#10003; | &#10003;       |
+| Persistent Volume                                                                  | &#10003; | &#45;          |
+| Custom Volume                                                                      | &#10003; | &#10003;       |
+| TLS: Add, Remove, Update, Rotate ( [Cert Manager](https://cert-manager.io/docs/) ) | &#10003; | &#10003;       |
+| Reconfigurable Health Checker                                                      | &#10003; | &#10003;       |
+| Externally manageable Auth Secret                                                  | &#10003; | &#10003;       |
+| Monitoring with Prometheus & Grafana                                               | &#10003; | &#10003;       |
+| Autoscaling (vertically, volume)	                                                  | &#10003; | &#10007;       |
 
-| Features                                                       | 
-|----------------------------------------------------------------|
-| Clustering - Combined (shared controller and broker nodes)     | 
-| Clustering - Topology (dedicated controllers and broker nodes) |
-| Kafka Connect Cluster                                          |
-| Connectors                                                     |
-| Custom Docker Image                                            | 
-| Authentication & Authorization                                 | 
-| Persistent Volume                                              | 
-| Custom Volume                                                  | 
-| TLS: using ( [Cert Manager](https://cert-manager.io/docs/) )   |
-| Reconfigurable Health Checker                                  | 
-| Externally manageable Auth Secret                              | 
-| Monitoring with Prometheus & Grafana                           |
+## Lifecycle of Kafka Object
+
+<!---
+ref : https://cacoo.com/diagrams/4PxSEzhFdNJRIbIb/0281B
+--->
+
+<p align="center">
+<img alt="lifecycle"  src="/docs/images/kafka/Kafka-CRD-Lifecycle.png">
+</p>
+
+## Lifecycle of ConnectCluster Object
+
+<p align="center">
+<img alt="lifecycle"  src="/docs/images/kafka/connectcluster/connectcluster-crd-lifecycle.png">
+</p>
 
 ## Supported Kafka Versions
 
@@ -45,21 +65,30 @@ KubeDB supports The following Kafka versions. Supported version are applicable f
 
 > The listed KafkaVersions are tested and provided as a part of the installation process (ie. catalog chart), but you are open to create your own [KafkaVersion](/docs/guides/kafka/concepts/kafkaversion.md) object with your custom Kafka image.
 
-## Lifecycle of Kafka Object
+## Supported KafkaConnector Versions
 
-<!---
-ref : https://cacoo.com/diagrams/4PxSEzhFdNJRIbIb/0281B
---->
+| Connector Plugin     | Type   | Version     | Connector Class                                            |
+|----------------------|--------|-------------|------------------------------------------------------------|
+| mongodb-1.11.0       | Source | 1.11.0      | com.mongodb.kafka.connect.MongoSourceConnector             |
+| mongodb-1.11.0       | Sink   | 1.11.0      | com.mongodb.kafka.connect.MongoSinkConnector               |
+| mysql-2.4.2.final    | Source | 2.4.2.Final | io.debezium.connector.mysql.MySqlConnector                 |
+| postgres-2.4.2.final | Source | 2.4.2.Final | io.debezium.connector.postgresql.PostgresConnector         |
+| jdbc-2.6.1.final     | Sink   | 2.6.1.Final | io.debezium.connector.jdbc.JdbcSinkConnector               |
+| s3-2.15.0            | Sink   | 2.15.0      | io.aiven.kafka.connect.s3.AivenKafkaConnectS3SinkConnector |
+| gcs-0.13.0           | Sink   | 0.13.0      | io.aiven.kafka.connect.gcs.GcsSinkConnector                |
 
-<p align="center">
-<img alt="lifecycle"  src="/docs/images/kafka/Kafka-CRD-Lifecycle.png">
-</p>
 
 ## User Guide 
-- [Quickstart Kafka](/docs/guides/kafka/quickstart/overview/index.md) with KubeDB Operator.
+- [Quickstart Kafka](/docs/guides/kafka/quickstart/overview/kafka/index.md) with KubeDB Operator.
+- [Quickstart ConnectCluster](/docs/guides/kafka/quickstart/overview/connectcluster/index.md) with KubeDB Operator.
 - Kafka Clustering supported by KubeDB
   - [Combined Clustering](/docs/guides/kafka/clustering/combined-cluster/index.md)
   - [Topology Clustering](/docs/guides/kafka/clustering/topology-cluster/index.md)
+- Monitor your Kafka database with KubeDB using [out-of-the-box Prometheus and Grafana](/docs/guides/kafka/monitoring/using-prometheus-operator.md).
 - Use [kubedb cli](/docs/guides/kafka/cli/cli.md) to manage databases like kubectl for Kubernetes.
 - Detail concepts of [Kafka object](/docs/guides/kafka/concepts/kafka.md).
+- Detail concepts of [ConnectCluster object](/docs/guides/kafka/concepts/connectcluster.md).
+- Detail concepts of [Connector object](/docs/guides/kafka/concepts/connector.md).
+- Detail concepts of [KafkaVersion object](/docs/guides/kafka/concepts/kafkaversion.md).
+- Detail concepts of [KafkaConnectorVersion object](/docs/guides/kafka/concepts/kafkaconnectorversion.md).
 - Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md).
diff --git a/docs/guides/kafka/concepts/appbinding.md b/docs/guides/kafka/concepts/appbinding.md
@@ -5,7 +5,7 @@ menu:
     identifier: kf-appbinding-concepts
     name: AppBinding
     parent: kf-concepts-kafka
-    weight: 30
+    weight: 35
 menu_name: docs_{{ .version }}
 section_menu_id: guides
 ---

diff --git a/docs/guides/kafka/concepts/connector.md b/docs/guides/kafka/concepts/connector.md
@@ -0,0 +1,74 @@
+---
+title: Connector CRD
+menu:
+  docs_{{ .version }}:
+    identifier: kf-connector-concepts
+    name: Connector
+    parent: kf-concepts-kafka
+    weight: 20
+menu_name: docs_{{ .version }}
+section_menu_id: guides
+---
+
+> New to KubeDB? Please start [here](/docs/README.md).
+
+# Connector
+
+## What is Connector
+
+`Connector` is a Kubernetes `Custom Resource Definitions` (CRD). It provides declarative configuration for [Connector](https://kafka.apache.org/) in a Kubernetes native way. You only need to describe the desired configuration in a `Connector` object, and the KubeDB operator will create Kubernetes objects in the desired state for you.
+
+## Connector Spec
+
+As with all other Kubernetes objects, a Connector needs `apiVersion`, `kind`, and `metadata` fields. It also needs a `.spec` section. Below is an example Connector object.
+
+```yaml
+apiVersion: kafka.kubedb.com/v1alpha1
+kind: Connector
+metadata:
+  name: mongodb-source-connector
+  namespace: demo
+spec:
+  configSecret:
+    name: mongodb-source-config
+  connectClusterRef:
+    name: connectcluster-quickstart
+    namespace: demo
+  terminationPolicy: WipeOut
+```
+
+### spec.configSecret
+
+`spec.configSecret` is a required field that specifies the name of the secret containing the configuration for the Connector. The secret should contain a key `config.properties` which contains the configuration for the Connector.
+```yaml
+spec:
+  configSecret:
+    name: <config-secret-name>
+```
+
+### spec.connectClusterRef
+
+`spec.connectClusterRef` is a required field that specifies the name and namespace of the `ConnectCluster` object that the `Connector` object is associated with. This is an appbinding reference for `ConnectCluster` object.
+```yaml
+spec:
+  connectClusterRef:
+    name: <connectcluster-appbinding-name>
+    namespace: <connectcluster-appbinding-namespace>
+```
+
+### spec.terminationPolicy
+
+`spec.terminationPolicy` gives flexibility whether to `nullify`(reject) the delete operation of `Connector` CR or which resources KubeDB should keep or delete when you delete `ConnectCluster` CR. KubeDB provides following four termination policies:
+
+- Delete
+- DoNotTerminate
+- WipeOut
+
+When `terminationPolicy` is `DoNotTerminate`, KubeDB takes advantage of `ValidationWebhook` feature in Kubernetes 1.9.0 or later clusters to implement `DoNotTerminate` feature. If admission webhook is enabled, `DoNotTerminate` prevents users from deleting the resource as long as the `spec.terminationPolicy` is set to `DoNotTerminate`.
+
+## Next Steps
+
+- Learn how to use KubeDB to run a Apache Kafka Connect cluster [here](/docs/guides/kafka/quickstart/overview/connectcluster/index.md).
+- Detail concepts of [KafkaConnectorVersion object](/docs/guides/kafka/concepts/kafkaconnectorversion.md).
+- Learn to use KubeDB managed Kafka objects using [CLIs](/docs/guides/kafka/cli/cli.md).
+- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md).
diff --git a/docs/guides/kafka/concepts/kafkaconnectorversion.md b/docs/guides/kafka/concepts/kafkaconnectorversion.md
@@ -5,7 +5,7 @@ menu:
     identifier: kf-kafkaconnectorversion-concepts
     name: KafkaConnectorVersion
     parent: kf-concepts-kafka
-    weight: 25
+    weight: 30
 menu_name: docs_{{ .version }}
 section_menu_id: guides
 ---

diff --git a/docs/guides/kafka/concepts/kafkaversion.md b/docs/guides/kafka/concepts/kafkaversion.md
@@ -5,7 +5,7 @@ menu:
     identifier: kf-catalog-concepts
     name: KafkaVersion
     parent: kf-concepts-kafka
-    weight: 20
+    weight: 25
 menu_name: docs_{{ .version }}
 section_menu_id: guides
 ---

diff --git a/docs/guides/kafka/monitoring/overview.md b/docs/guides/kafka/monitoring/overview.md
@@ -31,16 +31,16 @@ When a user creates a Kafka crd with `spec.monitor` section configured, KubeDB o
 
 In order to enable monitoring for a database, you have to configure `spec.monitor` section. KubeDB provides following options to configure `spec.monitor` section:
 
-|                Field                               |    Type    |                                                                                     Uses                                                       |
-| -------------------------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| Field                                              | Type       | Uses                                                                                                                                    |
+|----------------------------------------------------|------------|-----------------------------------------------------------------------------------------------------------------------------------------|
 | `spec.monitor.agent`                               | `Required` | Type of the monitoring agent that will be used to monitor this database. It can be `prometheus.io/builtin` or `prometheus.io/operator`. |
-| `spec.monitor.prometheus.exporter.port`            | `Optional` | Port number where the exporter side car will serve metrics.                                                                                    |
-| `spec.monitor.prometheus.exporter.args`            | `Optional` | Arguments to pass to the exporter sidecar.                                                                                                     |
-| `spec.monitor.prometheus.exporter.env`             | `Optional` | List of environment variables to set in the exporter sidecar container.                                                                        |
-| `spec.monitor.prometheus.exporter.resources`       | `Optional` | Resources required by exporter sidecar container.                                                                                              |
-| `spec.monitor.prometheus.exporter.securityContext` | `Optional` | Security options the exporter should run with.                                                                                                 |
-| `spec.monitor.prometheus.serviceMonitor.labels`    | `Optional` | Labels for `ServiceMonitor` crd.                                                                                                               |
-| `spec.monitor.prometheus.serviceMonitor.interval`  | `Optional` | Interval at which metrics should be scraped.                                                                                                   |
+| `spec.monitor.prometheus.exporter.port`            | `Optional` | Port number where the exporter side car will serve metrics.                                                                             |
+| `spec.monitor.prometheus.exporter.args`            | `Optional` | Arguments to pass to the exporter sidecar.                                                                                              |
+| `spec.monitor.prometheus.exporter.env`             | `Optional` | List of environment variables to set in the exporter sidecar container.                                                                 |
+| `spec.monitor.prometheus.exporter.resources`       | `Optional` | Resources required by exporter sidecar container.                                                                                       |
+| `spec.monitor.prometheus.exporter.securityContext` | `Optional` | Security options the exporter should run with.                                                                                          |
+| `spec.monitor.prometheus.serviceMonitor.labels`    | `Optional` | Labels for `ServiceMonitor` crd.                                                                                                        |
+| `spec.monitor.prometheus.serviceMonitor.interval`  | `Optional` | Interval at which metrics should be scraped.                                                                                            |
 
 ## Sample Configuration
 
@@ -60,7 +60,7 @@ spec:
       name: kafka-ca-issuer
       kind: Issuer
   replicas: 3
-  version: 3.4.0
+  version: 3.6.1
   storage:
     accessModes:
       - ReadWriteOnce

diff --git a/docs/guides/kafka/quickstart/overview/_index.md b/docs/guides/kafka/quickstart/overview/_index.md
@@ -5,6 +5,6 @@ menu:
     identifier: kf-overview-kafka
     name: Overview
     parent: kf-quickstart-kafka
-    weight: 15
+    weight: 10
 menu_name: docs_{{ .version }}
 ---