support az aware hashring and multiple sts in one hashring #129

christopherzli · 2024-01-24T00:14:17Z

What changes are proposed in this pull request?

Support az aware hashring in Thanos v0.32+
https://thanos.io/tip/components/receive.md/#az-aware-ketama-hashring-experimental
supported multiple sts in one hashring
Update dependencies including golang version, along with bingo and golangci-lint version, and fixed all lint tests

How is this tested?

main_test.go,
Make thanos-receive-controller
#127

christopherzli · 2024-01-24T09:26:10Z

.bingo/golangci-lint.mod

@@ -2,4 +2,4 @@ module _ // Auto generated by https://github.com/bwplotka/bingo. DO NOT EDIT

 go 1.17

-require github.com/golangci/golangci-lint v1.45.2 // cmd/golangci-lint


needs to update golangci-lint version and bingo version to work with golang v1.21

christopherzli · 2024-01-24T09:26:55Z

.golangci.yml

@@ -19,6 +19,11 @@ linters:
  - varnamelen
  - paralleltest
  - cyclop
+  - exhaustruct


temp disable some newly added lint checks, feel free to resolve on these lint issues

christopherzli · 2024-01-24T09:27:26Z

go.mod

-	k8s.io/apimachinery v0.26.1
-	k8s.io/client-go v0.26.1
+	github.com/prometheus/client_golang v1.17.0
+	github.com/thanos-io/thanos v0.33.0


accomodate thanos newer version

christopherzli · 2024-01-24T09:27:35Z

main.go

 	return p.listsMetric
 }

-func (p prometheusReflectorMetrics) NewListDurationMetric(name string) cache.SummaryMetric {
+func (p prometheusReflectorMetrics) NewListDurationMetric(_ string) cache.SummaryMetric {


fix lint issues

christopherzli · 2024-01-24T09:27:42Z

main.go

@@ -283,7 +289,7 @@ func newReflectorMetrics(reg *prometheus.Registry) prometheusReflectorMetrics {

 const labelParts = 2

-func splitLabel(in string) (key, value string) {
+func splitLabel(in string) (string, string) {


fix nonamed return values

christopherzli · 2024-01-24T09:28:23Z

main.go

+			statefulsets[hashring] = []*appsv1.StatefulSet{}
+		}
+		// Append the new value to the slice associated with the hashring key
+		statefulsets[hashring] = append(statefulsets[hashring], sts.DeepCopy())


support multiple sts placed in a single hashring in our setup

christopherzli · 2024-01-24T09:28:43Z

main.go

@@ -679,6 +686,40 @@ func (c *controller) populate(ctx context.Context, hashrings []receive.HashringC
 	}
 }

+func (c *controller) populateEndpoint(sts *appsv1.StatefulSet, podIndex int, err error, pod *corev1.Pod) *receive.Endpoint {


extracted this function to reduce cognitive complexity required by lint

christopherzli · 2024-01-25T10:36:14Z

tested it worked when scale up 1 node, the configmap updates the configmap too:

## What changes are proposed in this pull request? 1. Support az aware hashring in Thanos v0.32+ https://thanos.io/tip/components/receive.md/#az-aware-ketama-hashring-experimental, 2. also supported multiple sts in one hashring config 3. fix lint and go dependency issues OSS PR: observatorium#129 ## How is this tested? main_test.go, Make thanos-receive-controller pending test in deployment

philipgough

lgtm thanks for this

…rium#129) * support az aware hashring * Update receive-controller.json * support multiple statefulsets in 1 hashring * add more logs * style * fix lint issue * debug * return when encountering error * remove whitespace

* support az aware hashring and multiple sts in one hashring (observatorium#129) * support az aware hashring * Update receive-controller.json * support multiple statefulsets in 1 hashring * add more logs * style * fix lint issue * debug * return when encountering error * remove whitespace * Fix k8s permissions (observatorium#133) * Fix k8s permissions * fix ci * fix ci * sync * add pantheon migration state * Revert "Fix k8s permissions (observatorium#133)" This reverts commit e545b83. --------- Co-authored-by: Alec Rajeev <[email protected]>

* support az aware hashring and multiple sts in one hashring (observatorium#129) * support az aware hashring * Update receive-controller.json * support multiple statefulsets in 1 hashring * add more logs * style * fix lint issue * debug * return when encountering error * remove whitespace * Fix k8s permissions (observatorium#133) * Fix k8s permissions * fix ci * fix ci * sync * add pantheon migration state * Revert "Fix k8s permissions (observatorium#133)" This reverts commit e545b83. --------- Co-authored-by: Alec Rajeev <[email protected]> Signed-off-by: Yi Jin <[email protected]>

support az aware hashring

44a6a09

christopherzli mentioned this pull request Jan 24, 2024

support az aware endpoints with multiple sts in one hashring databricks/thanos-receive-controller#1

Merged

christopherzli force-pushed the dev2 branch from a7dcfa9 to 902e7fd Compare January 24, 2024 01:16

Update receive-controller.json

4f27cf4

christopherzli force-pushed the dev2 branch from 902e7fd to 4f27cf4 Compare January 24, 2024 01:24

christopherzli changed the title ~~support az aware hashring~~ support az aware hashring, update thanos and its dependencies Jan 24, 2024

christopherzli added 2 commits January 23, 2024 18:31

support multiple statefulsets in 1 hashring

f6e7a2b

add more logs

8cb9803

christopherzli changed the title ~~support az aware hashring, update thanos and its dependencies~~ support az aware hashring and multiple sts in one hashring Jan 24, 2024

style

3c8115c

christopherzli force-pushed the dev2 branch from dca47ea to 02a396b Compare January 24, 2024 09:19

fix lint issue

8e22498

christopherzli force-pushed the dev2 branch from 02a396b to 8e22498 Compare January 24, 2024 09:29

christopherzli commented Jan 24, 2024

View reviewed changes

christopherzli mentioned this pull request Jan 25, 2024

Support Multi-AZ in HashRing Config #127

Closed

debug

26f12fd

christopherzli force-pushed the dev2 branch from 07054ac to 26f12fd Compare January 25, 2024 10:40

christopherzli mentioned this pull request May 1, 2024

Fix vulnerabilities #135

Open

christopherzli added 2 commits May 8, 2024 15:08

return when encountering error

77b5f5f

remove whitespace

b206582

philipgough approved these changes May 16, 2024

View reviewed changes

philipgough merged commit 2753f3f into observatorium:main May 16, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support az aware hashring and multiple sts in one hashring #129

support az aware hashring and multiple sts in one hashring #129

christopherzli commented Jan 24, 2024 •

edited

Loading

christopherzli Jan 24, 2024

christopherzli Jan 24, 2024

christopherzli Jan 24, 2024

christopherzli Jan 24, 2024

christopherzli Jan 24, 2024

christopherzli Jan 24, 2024

christopherzli Jan 24, 2024

christopherzli commented Jan 25, 2024

philipgough left a comment

		@@ -2,4 +2,4 @@ module _ // Auto generated by https://github.com/bwplotka/bingo. DO NOT EDIT

		go 1.17

		require github.com/golangci/golangci-lint v1.45.2 // cmd/golangci-lint

support az aware hashring and multiple sts in one hashring #129

support az aware hashring and multiple sts in one hashring #129

Conversation

christopherzli commented Jan 24, 2024 • edited Loading

What changes are proposed in this pull request?

How is this tested?

christopherzli Jan 24, 2024

Choose a reason for hiding this comment

christopherzli Jan 24, 2024

Choose a reason for hiding this comment

christopherzli Jan 24, 2024

Choose a reason for hiding this comment

christopherzli Jan 24, 2024

Choose a reason for hiding this comment

christopherzli Jan 24, 2024

Choose a reason for hiding this comment

christopherzli Jan 24, 2024

Choose a reason for hiding this comment

christopherzli Jan 24, 2024

Choose a reason for hiding this comment

christopherzli commented Jan 25, 2024

philipgough left a comment

Choose a reason for hiding this comment

christopherzli commented Jan 24, 2024 •

edited

Loading