Add multi-database support to cluster mode #1671

xbasel · 2025-02-05T15:28:52Z

This commit introduces multi-database support in cluster mode while maintaining backward compatibility and requiring no API changes. Key features include:

Database-agnostic hashing: The hashing algorithm is unchanged. Identical keys map to the same slot across all databases. No changes to slot calculation. This ensures consistency in key distribution and maintains compatibility with existing single-database setups.
Implementation is fully backward compatible with no API changes.
The core structure remains an array of databases, each containing a list of hashtables (one per slot).

Cluster management commands are global commands, except for GETKEYSINSLOT and COUNTKEYSINSLOT, which run in selected-DB context.

MIGRATE command operates a selected-db context. Please note that MIGRATE command parameter destination-db is used, when migrating keys they can be migrated to a different database in the target, like in non-cluster mode.

Slot migration process changes when multiple databases are used:

	Iterate through all databases
 		SELECT database
 		keys = GETKEYSINSLOT
 		MIGRATE source target keys

Valkey-cli has been updated to support resharding across all databases.

#1319

This commit introduces multi-database support in cluster mode while maintaining backward compatibility and requiring no API changes. Key features include: - Database-agnostic hashing: The hashing algorithm is unchanged. Identical keys map to the same slot across all databases. No changes to slot calculation. This ensures consistency in key distribution and maintains compatibility with existing single-database setups. - Implementation is fully backward compatible with no API changes. - The core structure remains an array of databases, each containing a list of hashtables (one per slot). Cluster management commands are global commands, except for GETKEYSINSLOT and COUNTKEYSINSLOT, which run in selected-DB context. MIGRATE command operates a selected-db context. Please note that MIGRATE command parameter destination-db is used, when migrating keys they can be migrated to a different database in the target, like in non-cluster mode. Slot migration process changes when multiple databases are used: Iterate through all databases SELECT database keys = GETKEYSINSLOT MIGRATE source target keys Valkey-cli has been updated to support resharding across all databases. Signed-off-by: xbasel <[email protected]>

codecov · 2025-02-05T15:45:24Z

Codecov Report

Attention: Patch coverage is 94.91525% with 3 lines in your changes missing coverage. Please review.

Project coverage is 71.15%. Comparing base (2eac2cc) to head (a670e9d).
Report is 41 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/valkey-cli.c	92.59%	2 Missing ⚠️
src/cluster.c	90.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1671      +/-   ##
============================================
+ Coverage     70.97%   71.15%   +0.17%     
============================================
  Files           121      123       +2     
  Lines         65238    65542     +304     
============================================
+ Hits          46305    46638     +333     
+ Misses        18933    18904      -29

Files with missing lines	Coverage Δ
src/cluster_legacy.c	`86.17% <100.00%> (+0.27%)`	⬆️
src/config.c	`78.35% <ø> (-0.06%)`	⬇️
src/db.c	`89.94% <ø> (+0.37%)`	⬆️
src/valkey-benchmark.c	`61.75% <ø> (+1.61%)`	⬆️
src/cluster.c	`89.12% <90.00%> (-0.12%)`	⬇️
src/valkey-cli.c	`56.30% <92.59%> (+0.41%)`	⬆️

... and 32 files with indirect coverage changes

JoBeR007 · 2025-02-11T10:09:32Z

src/db.c

@@ -1728,12 +1714,6 @@ void swapMainDbWithTempDb(serverDb *tempDb) {
 void swapdbCommand(client *c) {
    int id1, id2;

-    /* Not allowed in cluster mode: we have just DB 0 there. */


Would that be enough for swapdb to work in cluster mode? What will happen in setup with 2 shards, each responsible for half of slots in db's?

With this implementation SWAPDB must be executed in all primary nodes. There are three options:

Allow SWAPDB and shift responsibility to the user – Risky, non-atomic, can cause temporary inconsistency and data corruption. Needs strong warnings.

Keep SWAPDB disabled in cluster mode – Safest, avoids inconsistency.

Make SWAPDB cluster-wide and atomic or – Complex, unclear feasibility.

I think option 2 is the safest bet. @JoBeR007 wdyt?

Is SWAPDB replicated as a single command? Then it's atomic.

If it's risky, it's risky in standslone mode with replicas too, right?

I think we can allow it. Swapping the data can only be done in some non-realtime workloads anyway I think.

I think risky because of replication and risky because of the need to execute SWAPDB on all primary nodes are unrelated just because as a user you can't control first, but user is the main risk in the second case.
I would keep SWAPDB disabled in cluster mode, if we decide to continue with this implementation

In cluster mode, consistency is per slot.

Is SWAPDB replicated as a single command? Then it's atomic.

If it's risky, it's risky in standslone mode with replicas too, right?

I think we can allow it. Swapping the data can only be done in some non-realtime workloads anyway I think.

I don’t think it’s very risky with standalone replicas. The only downside is if SWAPDB propagation to the replica takes time, a client might still access the wrong database. At least the client won’t be able to modify the wrong database, as they can only read.
In cluster mode, the same (logical) DB can be DB0 on one node and DB1 on another, but similar issues already exist today, FLUSHDB on one node doesn’t clear the entire DB since data exists in other slots/nodes. But as you said, consistency is per slot.

Yes, FLUSHDB is very similar in this regard. If a failover happens just before this command has been propagated to replicas, it's a big thing, but it's no surprise I think. The client can use WAIT or check replication offset to make sure the FLUSHDB or SWAPDB was successful on the replicas.

Regarding this, I think it is not just an issue of Multi-database but is more related to atomic slot migration. If a shard is in a stable state (not undergoing slot migration), then flushdb/flushall/swapdb are safe. However, if slot migration is in progress, it might lead to data inconsistency.

I think this needs to be considered alongside atomic-slot-migration:

During the ATM process, for slots being migrated, if we encounter flushall/flushdb, we can send a command like flushslot or flushslotall to the target shard

As for swapdb, I recommend temporarily prohibiting execution during the ATM process

@PingXie @enjoy-binbin , please also take note of this.

make sense. @murphyjacob4 FYI

I made a comment on the issue about this, but also worth mentioning it's hard to orchestrate SWAPDB. Even in steady state, flushdb and flushall are idempotent (you can send them multiple times) but swapdb isn't. If a command times out on one node, it's hard to reason about if it was successful and how to retry it. I think we should continue to disable SWAPDB in cluster mode for now, unless we introduce an idempotent way to do the swap.

Signed-off-by: zhaozhao.zz <[email protected]>

soloestoy · 2025-02-13T08:59:51Z

src/cluster.c

@@ -1102,7 +1110,7 @@ getNodeByQuery(client *c, struct serverCommand *cmd, robj **argv, int argc, int
             * NODE <node-id>. */
            int flags = LOOKUP_NOTOUCH | LOOKUP_NOSTATS | LOOKUP_NONOTIFY | LOOKUP_NOEXPIRE;
            if ((migrating_slot || importing_slot) && !pubsubshard_included) {
-                if (lookupKeyReadWithFlags(&server.db[0], thiskey, flags) == NULL)
+                if (lookupKeyReadWithFlags(c->db, thiskey, flags) == NULL)


Here, I modified it to use c->db, so for most commands, the key it wants to access can be correctly located. However, some cross-DB commands, such as COPY, still require additional checks. The ultimate solution is atomic-slot-migration I believe. Once ATM is implemented, the TRYAGAIN issue will no longer occur.

I noticed that getNodeByQuery doesn't follow selects either, so this might not be the right database. If you for example have:

SELECT 0 GET FOO SELECT 1 GET FOO

c->db won't be correct here either. COPY and move are also such problems as mentioned. I wonder if there is some way to make this correct without having ATM so we can limit the breakage if you're moving from standalone to cluster.

Generally, c->db can obtain the correct context information. Are you referring to the scenario where the select command is also used within a transaction (MULTI/EXEC)?

soloestoy · 2025-02-13T09:07:19Z

I'm happy that we did "Unified db rehash method for both standalone and cluster #12848" when developing kvstore , which made the implementation of multi-database simpler.

Signed-off-by: xbasel <[email protected]>

hpatro

We need to add history to SWAPDB, SELECT, MOVE json files to indicate it's supported since 9.0.

hpatro · 2025-02-17T18:55:00Z

src/cluster_legacy.c

+int dbHasNoKeys(void) {
+    for (int i = 0; i < server.dbnum; i++) {
+        if (kvstoreSize(server.db[i].keys) != 0) {
+            return 0;
+        }
+    }
+    return 1;
+}


I think we should move this to db.c

hpatro · 2025-02-17T18:56:14Z

tests/support/cluster.tcl

@@ -196,7 +196,7 @@ proc ::valkey_cluster::__method__masternode_notfor_slot {id slot} {
    error "Slot $slot is everywhere"
 }

-proc ::valkey_cluster::__dispatch__ {id method args} {
+proc ::valkey_cluster::__dispatch__ {id method args} {	


madolson · 2025-02-17T19:55:49Z

tests/cluster/tests/05-cluster-multidatabases.tcl

@@ -0,0 +1,481 @@
+# Tests multi-databases in cluster mode


This is the legacy clustering system. Ideally this test should be in unit/cluster

madolson · 2025-02-17T19:58:30Z

tests/cluster/tests/05-cluster-multidatabases.tcl

@@ -0,0 +1,481 @@
+# Tests multi-databases in cluster mode
+
+proc pause {{message "Hit Enter to continue ==> "}} {


I see you used

valkey/tests/support/util.tcl

Line 1214 in f85c933

proc bp {{s {}}} {

for break points in other places in the code, make sure to clean this all up.

madolson · 2025-02-17T19:59:45Z

tests/unit/cluster/cli.tcl

+        }
+    }
+}
+
 } ;# tags

 set ::singledb $old_singledb


Worth noting we no longer need this constraint, so we might want to consider removing this throughout the codebase except when we are running in external mode.

madolson · 2025-02-17T20:01:32Z

src/cluster.c

@@ -1102,7 +1110,7 @@ getNodeByQuery(client *c, struct serverCommand *cmd, robj **argv, int argc, int
             * NODE <node-id>. */
            int flags = LOOKUP_NOTOUCH | LOOKUP_NOSTATS | LOOKUP_NONOTIFY | LOOKUP_NOEXPIRE;
            if ((migrating_slot || importing_slot) && !pubsubshard_included) {
-                if (lookupKeyReadWithFlags(&server.db[0], thiskey, flags) == NULL)
+                if (lookupKeyReadWithFlags(c->db, thiskey, flags) == NULL)


I noticed that getNodeByQuery doesn't follow selects either, so this might not be the right database. If you for example have:

SELECT 0 GET FOO SELECT 1 GET FOO

c->db won't be correct here either. COPY and move are also such problems as mentioned. I wonder if there is some way to make this correct without having ATM so we can limit the breakage if you're moving from standalone to cluster.

madolson · 2025-02-17T20:07:53Z

src/db.c

@@ -1728,12 +1714,6 @@ void swapMainDbWithTempDb(serverDb *tempDb) {
 void swapdbCommand(client *c) {
    int id1, id2;

-    /* Not allowed in cluster mode: we have just DB 0 there. */


I made a comment on the issue about this, but also worth mentioning it's hard to orchestrate SWAPDB. Even in steady state, flushdb and flushall are idempotent (you can send them multiple times) but swapdb isn't. If a command times out on one node, it's hard to reason about if it was successful and how to retry it. I think we should continue to disable SWAPDB in cluster mode for now, unless we introduce an idempotent way to do the swap.

madolson · 2025-02-17T20:11:09Z

tests/unit/lazyfree.tcl

@@ -1,5 +1,12 @@
 start_server {tags {"lazyfree"}} {
    test "UNLINK can reclaim memory in background" {
+
+        # The test framework invokes "flushall", replacing kvstores even if empty.  


I would rather we did a sync flushall then in the test framework, so we don't have these random waits all over the place.

xbasel added 2 commits February 5, 2025 17:45

Reformatting

4d45a7e

Typo

a20c149

xbasel mentioned this pull request Feb 5, 2025

[NEW] Multiple DB supports in cluster mode #1319

Open

xbasel marked this pull request as draft February 6, 2025 10:01

xbasel mentioned this pull request Feb 6, 2025

[NEW] Multi-database support in cluster mode - Implementation Plan #1681

Closed

xbasel added 2 commits February 10, 2025 21:57

fix tests

1c3ea2f

fix test

0cf9b2d

xbasel marked this pull request as ready for review February 10, 2025 21:37

xbasel requested a review from zuiderkwast February 10, 2025 22:13

JoBeR007 reviewed Feb 11, 2025

View reviewed changes

soloestoy requested review from soloestoy and removed request for zuiderkwast February 12, 2025 06:28

soloestoy added 2 commits February 13, 2025 16:35

Correctly determine the DB of the key in getNodeByQuery

9fff9f1

Signed-off-by: zhaozhao.zz <[email protected]>

CLUSTER RESET should ensure primary node has no keys

0e29ea2

Signed-off-by: zhaozhao.zz <[email protected]>

soloestoy reviewed Feb 13, 2025

View reviewed changes

xbasel added 2 commits February 16, 2025 12:32

Add CLUSTER RESET test when the non-default db contain keys

64b4e9a

Signed-off-by: xbasel <[email protected]>

Expand multi-database tests in cluster mode to test SLOT-STATS

a670e9d

Signed-off-by: xbasel <[email protected]>

ranshid added the release-notes This issue should get a line item in the release notes label Feb 17, 2025

hpatro reviewed Feb 17, 2025

View reviewed changes

madolson reviewed Feb 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add multi-database support to cluster mode #1671

Add multi-database support to cluster mode #1671

xbasel commented Feb 5, 2025 •

edited

Loading

codecov bot commented Feb 5, 2025 •

edited

Loading

JoBeR007 Feb 11, 2025

xbasel Feb 11, 2025 •

edited

Loading

zuiderkwast Feb 12, 2025

JoBeR007 Feb 12, 2025

zuiderkwast Feb 12, 2025

xbasel Feb 12, 2025

zuiderkwast Feb 12, 2025

soloestoy Feb 13, 2025

PingXie Feb 17, 2025

madolson Feb 17, 2025

soloestoy Feb 13, 2025

madolson Feb 17, 2025

soloestoy Feb 18, 2025

soloestoy commented Feb 13, 2025

hpatro left a comment

hpatro Feb 17, 2025

hpatro Feb 17, 2025

madolson Feb 17, 2025

madolson Feb 17, 2025

madolson Feb 17, 2025

madolson Feb 17, 2025

madolson Feb 17, 2025

madolson Feb 17, 2025

		@@ -0,0 +1,481 @@
		# Tests multi-databases in cluster mode

		proc pause {{message "Hit Enter to continue ==> "}} {

Add multi-database support to cluster mode #1671

Are you sure you want to change the base?

Add multi-database support to cluster mode #1671

Conversation

xbasel commented Feb 5, 2025 • edited Loading

codecov bot commented Feb 5, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

xbasel Feb 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soloestoy commented Feb 13, 2025

hpatro left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xbasel commented Feb 5, 2025 •

edited

Loading

codecov bot commented Feb 5, 2025 •

edited

Loading

xbasel Feb 11, 2025 •

edited

Loading