Skip to content

Commit

Permalink
Doc 773 single source leader pinning (#873)
Browse files Browse the repository at this point in the history
Co-authored-by: Paulo Borges <[email protected]>
Co-authored-by: Joyce Fee <[email protected]>
Co-authored-by: Jake Cahill <[email protected]>
Co-authored-by: Gellért Peresztegi-Nagy <[email protected]>
Co-authored-by: Angela Simms <[email protected]>
Co-authored-by: Kat Batuigas <[email protected]>
Co-authored-by: JakeSCahill <[email protected]>
  • Loading branch information
8 people authored Dec 3, 2024
1 parent 3559a16 commit 59aabf4
Show file tree
Hide file tree
Showing 2 changed files with 37 additions and 16 deletions.
2 changes: 1 addition & 1 deletion modules/develop/pages/consume-data/follower-fetching.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ The first fetch from a consumer is processed by a Redpanda leader broker. The le

Redpanda decides which replica a consumer fetches from. If the consumer configures its `client.rack` property, Redpanda by default selects a replica from the same rack as the consumer, if available.

ifndef::env-cloud[]
To enable follower fetching in Redpanda, configure properties for the consumer and the Redpanda cluster and broker:

ifndef::env-cloud[]
- For a Redpanda cluster, set the xref:reference:cluster-properties.adoc#enable_rack_awareness[`enable_rack_awareness`] property to `true`.
- For each Redpanda broker, set the xref:reference:node-properties.adoc#rack[`rack`] property to a rack ID.
- For each consumer, set the `client.rack` property to a rack ID.
Expand Down
51 changes: 36 additions & 15 deletions modules/develop/pages/produce-data/leader-pinning.adoc
Original file line number Diff line number Diff line change
@@ -1,45 +1,64 @@
= Leader Pinning
// tag::single-source[]
:description: Learn about Leader Pinning and how to configure a preferred partition leader location based on cloud availability zones or regions.
:description: Learn about leader pinning and how to configure a preferred partition leader location based on cloud availability zones or regions.

Produce requests that write data to Redpanda topics go through the topic partition leader, which syncs messages across its follower replicas. For a Redpanda cluster deployed across multiple availability zones (AZs), leader pinning ensures that a topic's partition leaders are geographically closer to clients, which helps decrease networking costs and guarantees lower latency.

If consumers are located in the same preferred region or AZ for leader pinning, and you have not set up xref:develop:consume-data/follower-fetching.adoc[follower fetching], leader pinning can also help reduce networking costs on consume requests.

ifndef::env-cloud[]
== Prerequisites

[NOTE]
====
include::shared:partial$enterprise-license.adoc[]
====

Produce requests that write data to Redpanda topics go through the topic partition leader, which syncs messages across its follower replicas. For a Redpanda cluster deployed across multiple availability zones (AZs), Leader Pinning ensures that a topic's partition leaders are geographically closer to clients, which helps decrease networking costs and guarantees lower latency.
Before you can enable leader pinning, you must xref:manage:rack-awareness.adoc#configure-rack-awareness[configure rack awareness] on the cluster. If the config_ref:enable_rack_awareness,true,properties/cluster-properties[] cluster configuration property is set to `false`, leader pinning is disabled across the cluster.

If consumers are located in the same preferred region or AZ for Leader Pinning, and you have not set up xref:develop:consume-data/follower-fetching.adoc[follower fetching], Leader Pinning can also help reduce networking costs on consume requests.
endif::[]

== Prerequisites
ifndef::env-cloud[]
== Configure leader pinning

Before you can enable Leader Pinning, you must xref:manage:rack-awareness.adoc#configure-rack-awareness[configure rack awareness] on the cluster. If the config_ref:enable_rack_awareness,true,properties/cluster-properties[] cluster configuration property is set to `false`, Leader Pinning is disabled across the cluster.

== Configure Leader Pinning

You can use both a topic configuration property and a cluster configuration property to configure Leader Pinning.

You can set the topic configuration property for individual topics only, or set the cluster-wide configuration property that will enable Leader Pinning by default for all topics. You can also use a combination in which a default setting applies across the cluster, and you toggle the setting on or off for specific topics.
You can use both a topic configuration property and a cluster configuration property to configure leader pinning.

You can set the topic configuration property for individual topics only, or set the cluster-wide configuration property that will enable leader pinning by default for all topics. You can also use a combination in which a default setting applies across the cluster, and you toggle the setting on or off for specific topics.

This configuration is based on the following scenario: you have Redpanda deployed in a multi-AZ or multi-region cluster, and you have configured each broker so that the config_ref:rack,true,properties/broker-properties[] configuration property contains rack IDs corresponding to the AZ IDs:

* Set the topic configuration property xref:reference:properties/topic-properties.adoc#redpandaleaderspreference[`redpanda.leaders.preference`]. The property accepts the following string values:
+
--
** `none`: Opt out the topic from Leader Pinning.
** `none`: Opt out the topic from leader pinning.
** `racks:<rack1>[,<rack2>,...]`: Specify the preferred location (rack) of all topic partition leaders. The list can contain one or more rack IDs, and you can list the IDs in any order. Spaces in the list are ignored, for example: `racks:rack1,rack2` and `racks: rack1, rack2` are equivalent. You cannot specify empty rack IDs, for example: `racks: rack1,,rack2`. If you specify multiple IDs, Redpanda tries to distribute the partition leader locations equally across brokers in these racks.
--
+
This property inherits the default value from the cluster property `default_leaders_preference`.
* Set the cluster configuration property config_ref:default_leaders_preference,true,properties/cluster-properties[], which specifies the default Leader Pinning configuration for all topics that don’t have `redpanda.leaders.preference` explicitly set. It accepts values in the same format as `redpanda.leaders.preference`. Default: `none`
* Set the cluster configuration property config_ref:default_leaders_preference,true,properties/cluster-properties[], which specifies the default leader pinning configuration for all topics that don’t have `redpanda.leaders.preference` explicitly set. It accepts values in the same format as `redpanda.leaders.preference`. Default: `none`
+
This property also affects internal topics, such as `__consumer_offsets` and transaction coordinators. All offset tracking and transaction coordination requests get placed within the preferred regions or AZs for all clients, so you see end-to-end latency and networking cost benefits.
If there is more than one broker in the preferred AZ (or AZs), Leader Pinning distributes partition leaders uniformly across brokers in the AZ.
If there is more than one broker in the preferred AZ (or AZs), leader pinning distributes partition leaders uniformly across brokers in the AZ.

endif::[]

== Leader Pinning failover across availability zones
ifdef::env-cloud[]
== Configure leader pinning

Configure leader pinning if you have Redpanda deployed in a multi-AZ or multi-region cluster and your ingress is concentrated in a particular AZ or region.

Use the topic configuration property `redpanda.leaders.preference` to configure leader pinning for individual topics. The property accepts the following string values:

** `none`: Opt out the topic from leader pinning.
** `racks:<rack1>[,<rack2>,...]`: Specify the preferred location (rack) of all topic partition leaders. The list can contain one or more rack IDs, and you can list the IDs in any order. Spaces in the list are ignored, for example: `racks:rack1,rack2` and `racks: rack1, rack2` are equivalent. You cannot specify empty rack IDs, for example: `racks: rack1,,rack2`. If you specify multiple IDs, Redpanda tries to distribute the partition leader locations equally across brokers in these racks.

If there is more than one broker in the preferred AZ (or AZs), leader pinning distributes partition leaders uniformly across brokers in the AZ.

endif::[]

== Leader pinning failover across availability zones

If there are three AZs: A, B, and C, and A becomes unavailable, the failover behavior is as follows:

Expand All @@ -50,3 +69,5 @@ If there are three AZs: A, B, and C, and A becomes unavailable, the failover beh
== Suggested reading

* xref:develop:consume-data/follower-fetching.adoc[]

// end::single-source[]

0 comments on commit 59aabf4

Please sign in to comment.