Skip to content

Commit

Permalink
DOC-924 Clarify and distinguish between Redpanda Connectors and Redpa…
Browse files Browse the repository at this point in the history
…nda Connect (#953)
  • Loading branch information
JakeSCahill authored Jan 17, 2025
1 parent 27a7db2 commit f9a1a06
Show file tree
Hide file tree
Showing 7 changed files with 31 additions and 29 deletions.
3 changes: 2 additions & 1 deletion modules/ROOT/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -254,7 +254,8 @@
**** xref:reference:k-redpanda-helm-spec.adoc[Redpanda]
**** xref:reference:k-operator-helm-spec.adoc[Redpanda Operator]
**** xref:reference:k-console-helm-spec.adoc[Redpanda Console]
**** xref:reference:k-connector-helm-spec.adoc[Redpanda Connectors]
**** xref:reference:k-connector-helm-spec.adoc[Kafka Connect]
**** xref:redpanda-connect:reference:k-connect-helm-spec.adoc[Redpanda Connect]
*** xref:reference:k-crd-index.adoc[]
**** xref:reference:k-crd.adoc[]
** xref:reference:monitor-metrics.adoc[Monitoring Metrics]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
= Deploy Redpanda Connectors in Docker
= Deploy Kafka Connect in Docker
:description: Learn how to use the Docker image to configure connectors for Redpanda.
:page-context-links: [{"name": "Docker", "to": "deploy:deployment-option/self-hosted/docker-image.adoc" },{"name": "Kubernetes", "to": "deploy:deployment-option/self-hosted/kubernetes/k-deploy-connectors.adoc" } ]
:page-toclevels: 1
Expand All @@ -7,6 +7,9 @@

include::shared:partial$community-supported-connectors.adoc[]

The https://hub.docker.com/r/redpandadata/connectors/tags[Redpanda Connectors Docker image^] includes a pre-configured instance of https://redpanda.com/guides/kafka-tutorial/what-is-kafka-connect[Kafka Connect] that works
with Redpanda. This image contains _only_ the MirrorMaker2 connector but you can build a custom image to install additional connectors.

The latest Docker image contains:

* Red Hat Enterprise Linux 8.9
Expand Down
Original file line number Diff line number Diff line change
@@ -1,16 +1,19 @@
= Deploy Redpanda Connectors in Kubernetes
:description: Learn how to deploy and configure Redpanda Connectors using Helm. Redpanda Connectors is a package that includes Kafka Connect and built-in connectors that you can create and configure in Redpanda Console.
= Deploy Kafka Connect in Kubernetes
:description: Learn how to deploy and configure Kafka Connect using Helm.
:link-dockerhub: https://hub.docker.com/r/redpandadata/connectors/tags
:page-context-links: [{"name": "Docker", "to": "deploy:deployment-option/self-hosted/docker-image.adoc" },{"name": "Kubernetes", "to": "deploy:deployment-option/self-hosted/kubernetes/k-deploy-connectors.adoc" } ]
:page-toclevels: 1
:env-kubernetes: true
:page-categories: Deployment

This topic describes how to use the Redpanda Helm chart to configure and deploy Redpanda Connectors in Kubernetes.
This topic describes how to use the Redpanda Helm chart to configure and deploy Kafka Connect in Kubernetes.

include::shared:partial$community-supported-connectors.adoc[]

Redpanda Connectors is a package that includes Kafka Connect and the following built-in connectors, sometimes known as plugins. You can create and configure the built-in connectors in Redpanda Console.
The Redpanda Connectors Helm chart includes a pre-configured instance of https://redpanda.com/guides/kafka-tutorial/what-is-kafka-connect[Kafka Connect] that works
with Redpanda. The underlying Docker image contains _only_ the MirrorMaker2 connector but you can build a custom image to install additional connectors.

TIP: Try xref:redpanda-connect:home:index.adoc[Redpanda Connect] for a faster way to build streaming data pipelines. It's fully compatible with the Kafka API but eliminates the complex setup and maintenance of Kafka Connect. Redpanda Connect also comes with built-in connectors to support AI integrations.

[cols="1a,2a"]
|===
Expand Down Expand Up @@ -57,15 +60,15 @@ helm version

== Deploy the Helm chart

The Redpanda Helm chart includes Redpanda Connectors as a subchart so that you can deploy a Redpanda cluster, Redpanda Connectors, and Redpanda Console using a single chart. You can enable and configure the Redpanda Connectors subchart in the `connectors` section of the Helm values.
The Redpanda Helm chart includes Kafka Connect (the Redpanda Connectors Helm chart) as a subchart so that you can deploy a Redpanda cluster, Kafka Connect, and Redpanda Console using a single chart. You can enable and configure the subchart in the `connectors` section of the Helm values.

The subchart includes a Pod that runs Kafka Connect and the built-in connectors. The Pod is managed by a Deployment that you can configure in the Helm values under `connectors.deployment`. Redpanda Console connects to Kafka Connect through the default `redpanda-connectors` Service. Kafka Connect connects to the Redpanda brokers through the default `redpanda` Service.

image::shared:k-connectors-architecture.png['Redpanda Connectors deployed in a Kubernetes cluster with three worker nodes.']

NOTE: Do not schedule Pods that run Kafka Connect on the same nodes as Redpanda brokers. Redpanda brokers require access to all node resources. See <<Tolerations>> and <<Affinity rules>>.

When deploying Redpanda Connectors with Helm, you can choose between two modes:
When deploying Kafka Connect with Helm, you can choose between two modes:

[cols="1a,2a,2a", options="header"]
|===
Expand All @@ -90,7 +93,7 @@ When `connectors.deployment.create` is `true`, you are responsible for configuri

=== Automatic mode

In automatic mode, the Redpanda Connectors subchart is automatically configured using the values in the Redpanda Helm chart. You don't need to add any additional configuration. The chart automatically configures the Deployment resource with the values needed for Kafka Connect to communicate with your Redpanda cluster and for Redpanda Console to communicate with Kafka Connect.
In automatic mode, the subchart is automatically configured using the values in the Redpanda Helm chart. You don't need to add any additional configuration. The chart automatically configures the Deployment resource with the values needed for Kafka Connect to communicate with your Redpanda cluster and for Redpanda Console to communicate with Kafka Connect.

NOTE: All incoming traffic to Kafka Connect, such as from Redpanda Console, is unauthenticated (no SASL) and sent in plain text (no TLS). See <<Limitations>>.

Expand Down Expand Up @@ -258,7 +261,7 @@ This section provides advice for configuring the Redpanda Connectors subchart. A

=== Name overrides

Deploying multiple instances of the same Helm chart in a Kubernetes cluster can lead to naming conflicts. Using `nameOverride` and `fullnameOverride` helps differentiate between them. If you have a production and staging environment for Redpanda connectors, different names help to avoid confusion.
Deploying multiple instances of the same Helm chart in a Kubernetes cluster can lead to naming conflicts. Using `nameOverride` and `fullnameOverride` helps differentiate between them. If you have a production and staging environment, different names help to avoid confusion.

- Use `nameOverride` to customize:

Expand Down Expand Up @@ -290,7 +293,7 @@ For all available settings, see the xref:reference:k-connector-helm-spec.adoc#co

=== Tolerations

Tolerations and taints allow Pods to be scheduled onto nodes where they otherwise wouldn't. If you have nodes dedicated to Redpanda Connectors with a taint `dedicated=redpanda-connectors:NoSchedule`, the following toleration allows the Pods to be scheduled on them.
Tolerations and taints allow Pods to be scheduled onto nodes where they otherwise wouldn't. If you have nodes dedicated to Kafka Connect with a taint `dedicated=redpanda-connectors:NoSchedule`, the following toleration allows the Pods to be scheduled on them.

[source,yaml]
----
Expand Down Expand Up @@ -611,7 +614,7 @@ From here, you can create and configure instances of your connectors.

=== Verify with the Kafka Connect REST API

. Get the name of the Pod that's running Redpanda Connectors:
. Get the name of the Pod that's running Kafka Connect:
+
[,bash]
----
Expand Down Expand Up @@ -746,7 +749,7 @@ connectors:
+
Kafka Connect should discover the new connector automatically on startup.

. Get the name of the Pod that's running Redpanda Connectors:
. Get the name of the Pod that's running Kafka Connect:
+
[,bash]
----
Expand Down
10 changes: 5 additions & 5 deletions modules/manage/pages/kubernetes/k-manage-connectors.adoc
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
= Create and Manage Connectors in Kubernetes
= Create and Manage Kafka Connect Connectors in Kubernetes
:description: Learn how to create and manage connectors using Redpanda Console or the Kafka Connect REST API.
:page-aliases: manage:kubernetes/manage-connectors.adoc
:page-categories: Management, Integration
:env-kubernetes: true

When you have Redpanda Connectors deployed, you can create and manage connectors using Redpanda Console or the Kafka Connect REST API.
When you have Kafka Connect deployed, you can create and manage connectors using Redpanda Console or the Kafka Connect REST API.

include::shared:partial$community-supported-connectors.adoc[]

== Prerequisites

- xref:deploy:deployment-option/self-hosted/kubernetes/kubernetes-deploy.adoc[Deploy a Redpanda cluster with Redpanda Console].
- xref:deploy:deployment-option/self-hosted/kubernetes/k-deploy-connectors.adoc[Deploy Redpanda Connectors].
- xref:deploy:deployment-option/self-hosted/kubernetes/k-deploy-connectors.adoc[Deploy Kafka Connect].

== Manage connectors in Redpanda Console

Expand All @@ -34,9 +34,9 @@ You can create and manage connectors by clicking *Connectors* in the navigation

== Manage connectors with the REST API

This section provides examples of requesting data from the REST API using cURL. Execute all cURL commands in the Pod that's running Redpanda Connectors.
This section provides examples of requesting data from the REST API using cURL. Execute all cURL commands in the Pod that's running Kafka Connect.

To view the name of the Pod that's running Redpanda Connectors:
To view the name of the Pod that's running Kafka Connect:

[,bash]
----
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Monitor Connectors in Kubernetes
:description: Monitor the health of your Redpanda Connectors using managed connector metrics.
:description: Monitor the health of Kafka Connect using managed connector metrics.
:page-context-links: [{"name": "Cloud", "to": "deploy:deployment-option/cloud/managed-connectors/monitor-connectors.adoc" },{"name": "Kubernetes", "to": "manage:kubernetes/monitoring/k-monitor-connectors.adoc" } ]
:env-kubernetes:
:page-aliases: manage:kubernetes/monitoring/monitor-connectors.adoc
Expand Down
4 changes: 2 additions & 2 deletions modules/manage/partials/monitor-connectors.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

ifdef::env-kubernetes[]

You can monitor the health of your Redpanda Connectors with
You can monitor the health of Kafka Connect with
metrics that are exported through a Prometheus endpoint at the default port 9404. You can use Grafana to visualize the metrics and set up alerts.

== Prerequisites
Expand Down Expand Up @@ -32,7 +32,7 @@ can use Grafana to visualize the metrics and set up alerts.

The most important metrics to be monitored by alerts are:

* connector failed tasks
* connector failed tasks
* connector lag / connector lag rate

endif::[]
Expand Down
11 changes: 3 additions & 8 deletions modules/shared/partials/community-supported-connectors.adoc
Original file line number Diff line number Diff line change
@@ -1,9 +1,4 @@
IMPORTANT: *The Redpanda Connectors Docker image is a community-supported artifact*. For support, reach out to the Redpanda team in https://redpanda.com/slack[Redpanda Community Slack^]. +
Are you looking for xref:redpanda-connect:ROOT:about.adoc[Redpanda Connect documentation]?
:note-caption: Community
NOTE: *The Redpanda Connectors Docker image is a community-supported artifact*. Redpanda Data does not provide enterprise support for this image. For support, reach out to the Redpanda team in https://redpanda.com/slack[Redpanda Community Slack^].

This topic describes how to use the https://hub.docker.com/r/redpandadata/connectors/tags[Redpanda Connectors Docker image^], which
includes a pre-configured instance of https://redpanda.com/guides/kafka-tutorial/what-is-kafka-connect[Kafka Connect] that works
with Redpanda.

The Redpanda Connectors Docker image contains _only_ the MirrorMaker2 connector but you can build a custom image to install additional connectors.
For a smoother experience, consider using the Managed Connectors available in Redpanda Cloud.
:note-caption: Note

0 comments on commit f9a1a06

Please sign in to comment.