Skip to content

Commit

Permalink
Add Connector
Browse files Browse the repository at this point in the history 
Signed-off-by: obaydullahmhs <[email protected]>
  • Loading branch information
@obaydullahmhs
obaydullahmhs committed May 6, 2024
1 parent 11ebe89 commit dc83bfa
Show file tree
Hide file tree
Showing 9 changed files with 242 additions and 39 deletions.
43 changes: 27 additions & 16 deletions docs/guides/kafka/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,21 +17,26 @@ aliases:
## Supported Kafka Features


| Features |
|----------------------------------------------------------------|
| Clustering - Combined (shared controller and broker nodes) |
| Clustering - Topology (dedicated controllers and broker nodes) |
| Kafka Connect Cluster |
| Connectors |
| Custom Docker Image |
| Authentication & Authorization |
| Persistent Volume |
| Custom Volume |
| TLS: using ( [Cert Manager](https://cert-manager.io/docs/) ) |
| Reconfigurable Health Checker |
| Externally manageable Auth Secret |
| Monitoring with Prometheus & Grafana |
| Features |
|------------------------------------------------------------------------------------|
| Clustering - Combined (shared controller and broker nodes) |
| Clustering - Topology (dedicated controllers and broker nodes) |
| Kafka Connect Cluster |
| Connectors |
| Custom Configuration |
| Automated Version Update |
| Automatic Vertical Scaling |
| Automated Horizontal Scaling |
| Automated Volume Expansion |
| Custom Docker Image |
| Authentication & Authorization |
| Persistent Volume |
| Custom Volume |
| TLS: Add, Remove, Update, Rotate ( [Cert Manager](https://cert-manager.io/docs/) ) |
| Reconfigurable Health Checker |
| Externally manageable Auth Secret |
| Monitoring with Prometheus & Grafana |
| Autoscaling (vertically, volume) |

## Supported Kafka Versions

Expand All @@ -55,8 +60,14 @@ ref : https://cacoo.com/diagrams/4PxSEzhFdNJRIbIb/0281B
<img alt="lifecycle" src="/docs/images/kafka/Kafka-CRD-Lifecycle.png">
</p>

## Lifecycle of ConnectCluster Object

<p align="center">
<img alt="lifecycle" src="/docs/images/kafka/connectcluster/connectcluster-crd-lifecycle.png">
</p>

## User Guide
- [Quickstart Kafka](/docs/guides/kafka/quickstart/overview/index.md) with KubeDB Operator.
- [Quickstart Kafka](/docs/guides/kafka/quickstart/overview/kafka/index.md) with KubeDB Operator.
- Kafka Clustering supported by KubeDB
- [Combined Clustering](/docs/guides/kafka/clustering/combined-cluster/index.md)
- [Topology Clustering](/docs/guides/kafka/clustering/topology-cluster/index.md)
Expand Down
74 changes: 74 additions & 0 deletions docs/guides/kafka/concepts/connector.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
---
title: Connector CRD
menu:
docs_{{ .version }}:
identifier: kf-connector-concepts
name: Connector
parent: kf-concepts-kafka
weight: 20
menu_name: docs_{{ .version }}
section_menu_id: guides
---

> New to KubeDB? Please start [here](/docs/README.md).
# Connector

## What is Connector

`Connector` is a Kubernetes `Custom Resource Definitions` (CRD). It provides declarative configuration for [Connector](https://kafka.apache.org/) in a Kubernetes native way. You only need to describe the desired configuration in a `Connector` object, and the KubeDB operator will create Kubernetes objects in the desired state for you.

## Connector Spec

As with all other Kubernetes objects, a Connector needs `apiVersion`, `kind`, and `metadata` fields. It also needs a `.spec` section. Below is an example Connector object.

```yaml
apiVersion: kafka.kubedb.com/v1alpha1
kind: Connector
metadata:
name: mongodb-source-connector
namespace: demo
spec:
configSecret:
name: mongodb-source-config
connectClusterRef:
name: connectcluster-quickstart
namespace: demo
terminationPolicy: WipeOut
```
### spec.configSecret
`spec.configSecret` is a required field that specifies the name of the secret containing the configuration for the Connector. The secret should contain a key `config.properties` which contains the configuration for the Connector.
```yaml
spec:
configSecret:
name: <config-secret-name>
```

### spec.connectClusterRef

`spec.connectClusterRef` is a required field that specifies the name and namespace of the `ConnectCluster` object that the `Connector` object is associated with. This is an appbinding reference for `ConnectCluster` object.
```yaml
spec:
connectClusterRef:
name: <connectcluster-appbinding-name>
namespace: <connectcluster-appbinding-namespace>
```

### spec.terminationPolicy

`spec.terminationPolicy` gives flexibility whether to `nullify`(reject) the delete operation of `Connector` CR or which resources KubeDB should keep or delete when you delete `ConnectCluster` CR. KubeDB provides following four termination policies:

- Delete
- DoNotTerminate
- WipeOut

When `terminationPolicy` is `DoNotTerminate`, KubeDB takes advantage of `ValidationWebhook` feature in Kubernetes 1.9.0 or later clusters to implement `DoNotTerminate` feature. If admission webhook is enabled, `DoNotTerminate` prevents users from deleting the resource as long as the `spec.terminationPolicy` is set to `DoNotTerminate`.

## Next Steps

- Learn how to use KubeDB to run a Apache Kafka Connect cluster [here](/docs/guides/kafka/quickstart/overview/connectcluster/index.md).
- Detail concepts of [KafkaConnectorVersion object](/docs/guides/kafka/concepts/kafkaconnectorversion.md).
- Learn to use KubeDB managed Kafka objects using [CLIs](/docs/guides/kafka/cli/cli.md).
- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md).
2 changes: 1 addition & 1 deletion docs/guides/kafka/concepts/kafkaconnectorversion.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ menu:
identifier: kf-kafkaconnectorversion-concepts
name: KafkaConnectorVersion
parent: kf-concepts-kafka
weight: 25
weight: 30
menu_name: docs_{{ .version }}
section_menu_id: guides
---
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/kafka/concepts/kafkaversion.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ menu:
identifier: kf-catalog-concepts
name: KafkaVersion
parent: kf-concepts-kafka
weight: 20
weight: 25
menu_name: docs_{{ .version }}
section_menu_id: guides
---
Expand Down
20 changes: 10 additions & 10 deletions docs/guides/kafka/monitoring/overview.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,16 +31,16 @@ When a user creates a Kafka crd with `spec.monitor` section configured, KubeDB o

In order to enable monitoring for a database, you have to configure `spec.monitor` section. KubeDB provides following options to configure `spec.monitor` section:

| Field | Type | Uses |
| -------------------------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| Field | Type | Uses |
|----------------------------------------------------|------------|-----------------------------------------------------------------------------------------------------------------------------------------|
| `spec.monitor.agent` | `Required` | Type of the monitoring agent that will be used to monitor this database. It can be `prometheus.io/builtin` or `prometheus.io/operator`. |
| `spec.monitor.prometheus.exporter.port` | `Optional` | Port number where the exporter side car will serve metrics. |
| `spec.monitor.prometheus.exporter.args` | `Optional` | Arguments to pass to the exporter sidecar. |
| `spec.monitor.prometheus.exporter.env` | `Optional` | List of environment variables to set in the exporter sidecar container. |
| `spec.monitor.prometheus.exporter.resources` | `Optional` | Resources required by exporter sidecar container. |
| `spec.monitor.prometheus.exporter.securityContext` | `Optional` | Security options the exporter should run with. |
| `spec.monitor.prometheus.serviceMonitor.labels` | `Optional` | Labels for `ServiceMonitor` crd. |
| `spec.monitor.prometheus.serviceMonitor.interval` | `Optional` | Interval at which metrics should be scraped. |
| `spec.monitor.prometheus.exporter.port` | `Optional` | Port number where the exporter side car will serve metrics. |
| `spec.monitor.prometheus.exporter.args` | `Optional` | Arguments to pass to the exporter sidecar. |
| `spec.monitor.prometheus.exporter.env` | `Optional` | List of environment variables to set in the exporter sidecar container. |
| `spec.monitor.prometheus.exporter.resources` | `Optional` | Resources required by exporter sidecar container. |
| `spec.monitor.prometheus.exporter.securityContext` | `Optional` | Security options the exporter should run with. |
| `spec.monitor.prometheus.serviceMonitor.labels` | `Optional` | Labels for `ServiceMonitor` crd. |
| `spec.monitor.prometheus.serviceMonitor.interval` | `Optional` | Interval at which metrics should be scraped. |

## Sample Configuration

Expand All @@ -60,7 +60,7 @@ spec:
name: kafka-ca-issuer
kind: Issuer
replicas: 3
version: 3.4.0
version: 3.6.1
storage:
accessModes:
- ReadWriteOnce
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/kafka/quickstart/overview/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@ menu:
identifier: kf-overview-kafka
name: Overview
parent: kf-quickstart-kafka
weight: 15
weight: 10
menu_name: docs_{{ .version }}
---
Loading

0 comments on commit dc83bfa

Please sign in to comment.