Deprecating the offline session preloading (keycloak#26160)

Closes keycloak#25300

Signed-off-by: Alexander Schwartz <[email protected]>

common/src/main/java/org/keycloak/common/Profile.java

@@ -102,6 +102,8 @@ public enum Feature {
         TRANSIENT_USERS("Transient users for brokering", Type.EXPERIMENTAL),
 
         MULTI_SITE("Multi-site support", Type.PREVIEW),
+
+        OFFLINE_SESSION_PRELOADING("Offline session preloading", Type.DEPRECATED),
         ;
 
         private final Type type;

common/src/test/java/org/keycloak/common/ProfileTest.java

@@ -1,5 +1,7 @@
 package org.keycloak.common;
 
+import org.hamcrest.MatcherAssert;
+import org.hamcrest.Matchers;
 import org.junit.After;
 import org.junit.Assert;
 import org.junit.BeforeClass;
@@ -81,7 +83,8 @@ public void checkDefaults() {
             Profile.Feature.TOKEN_EXCHANGE,
             Profile.Feature.CLIENT_SECRET_ROTATION,
             Profile.Feature.UPDATE_EMAIL,
-            Profile.Feature.LINKEDIN_OAUTH
+            Profile.Feature.LINKEDIN_OAUTH,
+            Profile.Feature.OFFLINE_SESSION_PRELOADING
         ));
 
         // KERBEROS can be disabled (i.e. FIPS mode disables SunJGSS provider)
@@ -245,14 +248,11 @@ public void configWithMultipleResolvers() {
     }
 
     public static void assertEquals(Set<Profile.Feature> actual, Collection<Profile.Feature> expected) {
-        assertEquals(actual, expected.toArray(new Profile.Feature[0]));
+        MatcherAssert.assertThat(actual, Matchers.equalTo(expected));
     }
 
     public static void assertEquals(Set<Profile.Feature> actual, Profile.Feature... expected) {
-        Profile.Feature[] a = actual.toArray(new Profile.Feature[0]);
-        Arrays.sort(a, new FeatureComparator());
-        Arrays.sort(expected, new FeatureComparator());
-        Assert.assertArrayEquals(expected, a);
+        assertEquals(actual, new HashSet<>(Arrays.asList(expected)));
     }
 
     private static class FeatureComparator implements Comparator<Profile.Feature> {

docs/documentation/release_notes/topics/24_0_0.adoc

@@ -125,6 +125,14 @@ will be shown.
 In addition to that, a new error (`EMAIL_ALREADY_VERIFIED`) event will be fired to indicate an attempt to verify an already verified email. You can
 use this event to track possible attempts to hijack user accounts in case the link has leaked or to alert users if they do not recognize the action.
 
+= Deprecated offline session preloading
+
+The default behavior of Keycloak is to load offline sessions on demand.
+The old behavior to preload them at startup is now deprecated, as pre-loading them at startup doesn't scale well with a growing number of sessions, and increases Keycloak memory usage. The old behavior will be removed in a future release.
+
+For more details, check the
+link:{upgradingguide_link}[{upgradingguide_name}].
+
 = Infinispan metrics use labels for cache manager and cache names
 
 When enabling metrics for {project_name}'s embedded caches, the metrics now use labels for the cache manager and the cache names.

docs/documentation/server_admin/topics/sessions/preloading.adoc

@@ -9,10 +9,11 @@ Therefore, the offline sessions are lazily fetched from the database by default.
 
 However, {project_name} can be configured to preload the offline sessions from the database into the Infinispan caches during the server startup.
 It can be achieved by setting `preloadOfflineSessionsFromDatabase` property in the `userSessions` SPI to `true`.
+This functionality is currently deprecated and will be removed in a future release.
 
 The following example shows how to configure offline sessions preloading.
 
 [source,bash]
 ----
-bin/kc.[sh|bat] start --spi-user-sessions-infinispan-preload-offline-sessions-from-database=true
+bin/kc.[sh|bat] start --features-enabled offline-session-preloading --spi-user-sessions-infinispan-preload-offline-sessions-from-database=true
 ----

docs/documentation/upgrading/topics/keycloak/changes-24_0_0.adoc

@@ -155,6 +155,21 @@ Therefore, it was changed to sequential session loading.
 For offline sessions, the default in this and previous versions of Keycloak is to load those sessions on demand, which scales better with a lot of offline sessions than the attempt to preload them in parallel. Setups that use this default setup are not affected by the change of the loading strategy for offline sessions.
 Setups that have offline session preloading enabled should migrate to a setup where offline-session preloading is disabled.
 
+= Deprecated offline session preloading
+
+The default behavior of Keycloak is to load offline sessions on demand.
+The old behavior to preload them at startup is now deprecated, as preloading them at startup does not scale well with a growing number of sessions, and increases Keycloak memory usage.
+The old behavior will be removed in a future release.
+
+To re-enable old behavior while it is deprecated and not removed yet, use the feature flag and the SPI option as shown below:
+
+[source,bash]
+----
+bin/kc.[sh|bat] start --features-enabled offline-session-preloading --spi-user-sessions-infinispan-preload-offline-sessions-from-database=true
+----
+
+The API of `UserSessionProvider` deprecated the method `getOfflineUserSessionByBrokerSessionId(RealmModel realm, String brokerSessionId)`.
+Instead of this method, use `getOfflineUserSessionByBrokerUserIdStream(RealmModel, String brokerUserId)` to first get the sessions of a user, and then filter by the broker session ID as needed.
 
 = Infinispan metrics use labels for cache manager and cache names
 

model/infinispan/src/main/java/org/keycloak/models/sessions/infinispan/InfinispanUserSessionProvider.java

@@ -385,6 +385,9 @@ protected Stream<UserSessionModel> getUserSessionsStream(RealmModel realm, UserS
             }
 
             if (predicate.getBrokerSessionId() != null) {
+                if (!Profile.isFeatureEnabled(Profile.Feature.OFFLINE_SESSION_PRELOADING)) {
+                    throw new RuntimeException("The deprecated offline session preloading feature is disabled in this configuration. Read the migration guide to learn more.");
+                }
                 // TODO add support for offline user-session lookup by brokerSessionId
                 // currently it is not possible to access the brokerSessionId in offline user-session in a database agnostic way
                 throw new ModelException("Dynamic database lookup for offline user-sessions by broker session ID is currently only supported for preloaded sessions. " +
@@ -813,6 +816,9 @@ public UserSessionAdapter getOfflineUserSession(RealmModel realm, String userSes
 
     @Override
     public UserSessionModel getOfflineUserSessionByBrokerSessionId(RealmModel realm, String brokerSessionId) {
+        if (!Profile.isFeatureEnabled(Profile.Feature.OFFLINE_SESSION_PRELOADING)) {
+            throw new RuntimeException("The deprecated offline session preloading feature is disabled in this configuration. Read the migration guide to learn more.");
+        }
         return this.getUserSessionsStream(realm, UserSessionPredicate.create(realm.getId()).brokerSessionId(brokerSessionId), true)
                 .findFirst().orElse(null);
     }

model/infinispan/src/main/java/org/keycloak/models/sessions/infinispan/InfinispanUserSessionProviderFactory.java

@@ -23,6 +23,7 @@
 import org.jboss.logging.Logger;
 import org.keycloak.Config;
 import org.keycloak.cluster.ClusterProvider;
+import org.keycloak.common.Profile;
 import org.keycloak.common.util.Environment;
 import org.keycloak.common.util.Time;
 import org.keycloak.connections.infinispan.InfinispanConnectionProvider;
@@ -104,6 +105,9 @@ public InfinispanUserSessionProvider create(KeycloakSession session) {
     public void init(Config.Scope config) {
         this.config = config;
         preloadOfflineSessionsFromDatabase = config.getBoolean("preloadOfflineSessionsFromDatabase", false);
+        if (preloadOfflineSessionsFromDatabase && !Profile.isFeatureEnabled(Profile.Feature.OFFLINE_SESSION_PRELOADING)) {
+            throw new RuntimeException("The deprecated offline session preloading feature is disabled in this configuration. Read the migration guide to learn more.");
+        }
     }
 
     @Override

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testBuildHelp.unix.approved.txt

@@ -50,17 +50,18 @@ Feature:
                        admin2[:v1], authorization[:v1], ciba[:v1], client-policies[:v1],
                        client-secret-rotation[:v1], device-flow[:v1], docker[:v1], dpop[:v1],
                        dynamic-scopes[:v1], fips[:v1], impersonation[:v1], js-adapter[:v1], kerberos
-                       [:v1], linkedin-oauth[:v1], multi-site[:v1], par[:v1], preview,
-                       recovery-codes[:v1], scripts[:v1], step-up-authentication[:v1],
-                       token-exchange[:v1], transient-users[:v1], update-email[:v1], web-authn[:v1].
+                       [:v1], linkedin-oauth[:v1], multi-site[:v1], offline-session-preloading[:
+                       v1], par[:v1], preview, recovery-codes[:v1], scripts[:v1],
+                       step-up-authentication[:v1], token-exchange[:v1], transient-users[:v1],
+                       update-email[:v1], web-authn[:v1].
 --features-disabled <feature>
                      Disables a set of one or more features. Possible values are: account-api,
                        account2, account3, admin-api, admin-fine-grained-authz, admin2,
                        authorization, ciba, client-policies, client-secret-rotation, device-flow,
                        docker, dpop, dynamic-scopes, fips, impersonation, js-adapter, kerberos,
-                       linkedin-oauth, multi-site, par, preview, recovery-codes, scripts,
-                       step-up-authentication, token-exchange, transient-users, update-email,
-                       web-authn.
+                       linkedin-oauth, multi-site, offline-session-preloading, par, preview,
+                       recovery-codes, scripts, step-up-authentication, token-exchange,
+                       transient-users, update-email, web-authn.
 
 HTTP(S):
 

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testBuildHelp.windows.approved.txt

@@ -50,17 +50,18 @@ Feature:
                        admin2[:v1], authorization[:v1], ciba[:v1], client-policies[:v1],
                        client-secret-rotation[:v1], device-flow[:v1], docker[:v1], dpop[:v1],
                        dynamic-scopes[:v1], fips[:v1], impersonation[:v1], js-adapter[:v1], kerberos
-                       [:v1], linkedin-oauth[:v1], multi-site[:v1], par[:v1], preview,
-                       recovery-codes[:v1], scripts[:v1], step-up-authentication[:v1],
-                       token-exchange[:v1], transient-users[:v1], update-email[:v1], web-authn[:v1].
+                       [:v1], linkedin-oauth[:v1], multi-site[:v1], offline-session-preloading[:
+                       v1], par[:v1], preview, recovery-codes[:v1], scripts[:v1],
+                       step-up-authentication[:v1], token-exchange[:v1], transient-users[:v1],
+                       update-email[:v1], web-authn[:v1].
 --features-disabled <feature>
                      Disables a set of one or more features. Possible values are: account-api,
                        account2, account3, admin-api, admin-fine-grained-authz, admin2,
                        authorization, ciba, client-policies, client-secret-rotation, device-flow,
                        docker, dpop, dynamic-scopes, fips, impersonation, js-adapter, kerberos,
-                       linkedin-oauth, multi-site, par, preview, recovery-codes, scripts,
-                       step-up-authentication, token-exchange, transient-users, update-email,
-                       web-authn.
+                       linkedin-oauth, multi-site, offline-session-preloading, par, preview,
+                       recovery-codes, scripts, step-up-authentication, token-exchange,
+                       transient-users, update-email, web-authn.
 
 HTTP(S):
 

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testExportHelp.unix.approved.txt

@@ -61,17 +61,18 @@ Feature:
                        admin2[:v1], authorization[:v1], ciba[:v1], client-policies[:v1],
                        client-secret-rotation[:v1], device-flow[:v1], docker[:v1], dpop[:v1],
                        dynamic-scopes[:v1], fips[:v1], impersonation[:v1], js-adapter[:v1], kerberos
-                       [:v1], linkedin-oauth[:v1], multi-site[:v1], par[:v1], preview,
-                       recovery-codes[:v1], scripts[:v1], step-up-authentication[:v1],
-                       token-exchange[:v1], transient-users[:v1], update-email[:v1], web-authn[:v1].
+                       [:v1], linkedin-oauth[:v1], multi-site[:v1], offline-session-preloading[:
+                       v1], par[:v1], preview, recovery-codes[:v1], scripts[:v1],
+                       step-up-authentication[:v1], token-exchange[:v1], transient-users[:v1],
+                       update-email[:v1], web-authn[:v1].
 --features-disabled <feature>
                      Disables a set of one or more features. Possible values are: account-api,
                        account2, account3, admin-api, admin-fine-grained-authz, admin2,
                        authorization, ciba, client-policies, client-secret-rotation, device-flow,
                        docker, dpop, dynamic-scopes, fips, impersonation, js-adapter, kerberos,
-                       linkedin-oauth, multi-site, par, preview, recovery-codes, scripts,
-                       step-up-authentication, token-exchange, transient-users, update-email,
-                       web-authn.
+                       linkedin-oauth, multi-site, offline-session-preloading, par, preview,
+                       recovery-codes, scripts, step-up-authentication, token-exchange,
+                       transient-users, update-email, web-authn.
 
 Config:
 

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testExportHelpAll.unix.approved.txt

@@ -61,17 +61,18 @@ Feature:
                        admin2[:v1], authorization[:v1], ciba[:v1], client-policies[:v1],
                        client-secret-rotation[:v1], device-flow[:v1], docker[:v1], dpop[:v1],
                        dynamic-scopes[:v1], fips[:v1], impersonation[:v1], js-adapter[:v1], kerberos
-                       [:v1], linkedin-oauth[:v1], multi-site[:v1], par[:v1], preview,
-                       recovery-codes[:v1], scripts[:v1], step-up-authentication[:v1],
-                       token-exchange[:v1], transient-users[:v1], update-email[:v1], web-authn[:v1].
+                       [:v1], linkedin-oauth[:v1], multi-site[:v1], offline-session-preloading[:
+                       v1], par[:v1], preview, recovery-codes[:v1], scripts[:v1],
+                       step-up-authentication[:v1], token-exchange[:v1], transient-users[:v1],
+                       update-email[:v1], web-authn[:v1].
 --features-disabled <feature>
                      Disables a set of one or more features. Possible values are: account-api,
                        account2, account3, admin-api, admin-fine-grained-authz, admin2,
                        authorization, ciba, client-policies, client-secret-rotation, device-flow,
                        docker, dpop, dynamic-scopes, fips, impersonation, js-adapter, kerberos,
-                       linkedin-oauth, multi-site, par, preview, recovery-codes, scripts,
-                       step-up-authentication, token-exchange, transient-users, update-email,
-                       web-authn.
+                       linkedin-oauth, multi-site, offline-session-preloading, par, preview,
+                       recovery-codes, scripts, step-up-authentication, token-exchange,
+                       transient-users, update-email, web-authn.
 
 Config:
 

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testImportHelp.unix.approved.txt

@@ -61,17 +61,18 @@ Feature:
                        admin2[:v1], authorization[:v1], ciba[:v1], client-policies[:v1],
                        client-secret-rotation[:v1], device-flow[:v1], docker[:v1], dpop[:v1],
                        dynamic-scopes[:v1], fips[:v1], impersonation[:v1], js-adapter[:v1], kerberos
-                       [:v1], linkedin-oauth[:v1], multi-site[:v1], par[:v1], preview,
-                       recovery-codes[:v1], scripts[:v1], step-up-authentication[:v1],
-                       token-exchange[:v1], transient-users[:v1], update-email[:v1], web-authn[:v1].
+                       [:v1], linkedin-oauth[:v1], multi-site[:v1], offline-session-preloading[:
+                       v1], par[:v1], preview, recovery-codes[:v1], scripts[:v1],
+                       step-up-authentication[:v1], token-exchange[:v1], transient-users[:v1],
+                       update-email[:v1], web-authn[:v1].
 --features-disabled <feature>
                      Disables a set of one or more features. Possible values are: account-api,
                        account2, account3, admin-api, admin-fine-grained-authz, admin2,
                        authorization, ciba, client-policies, client-secret-rotation, device-flow,
                        docker, dpop, dynamic-scopes, fips, impersonation, js-adapter, kerberos,
-                       linkedin-oauth, multi-site, par, preview, recovery-codes, scripts,
-                       step-up-authentication, token-exchange, transient-users, update-email,
-                       web-authn.
+                       linkedin-oauth, multi-site, offline-session-preloading, par, preview,
+                       recovery-codes, scripts, step-up-authentication, token-exchange,
+                       transient-users, update-email, web-authn.
 
 Config:
 

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testImportHelpAll.unix.approved.txt

@@ -61,17 +61,18 @@ Feature:
                        admin2[:v1], authorization[:v1], ciba[:v1], client-policies[:v1],
                        client-secret-rotation[:v1], device-flow[:v1], docker[:v1], dpop[:v1],
                        dynamic-scopes[:v1], fips[:v1], impersonation[:v1], js-adapter[:v1], kerberos
-                       [:v1], linkedin-oauth[:v1], multi-site[:v1], par[:v1], preview,
-                       recovery-codes[:v1], scripts[:v1], step-up-authentication[:v1],
-                       token-exchange[:v1], transient-users[:v1], update-email[:v1], web-authn[:v1].
+                       [:v1], linkedin-oauth[:v1], multi-site[:v1], offline-session-preloading[:
+                       v1], par[:v1], preview, recovery-codes[:v1], scripts[:v1],
+                       step-up-authentication[:v1], token-exchange[:v1], transient-users[:v1],
+                       update-email[:v1], web-authn[:v1].
 --features-disabled <feature>
                      Disables a set of one or more features. Possible values are: account-api,
                        account2, account3, admin-api, admin-fine-grained-authz, admin2,
                        authorization, ciba, client-policies, client-secret-rotation, device-flow,
                        docker, dpop, dynamic-scopes, fips, impersonation, js-adapter, kerberos,
-                       linkedin-oauth, multi-site, par, preview, recovery-codes, scripts,
-                       step-up-authentication, token-exchange, transient-users, update-email,
-                       web-authn.
+                       linkedin-oauth, multi-site, offline-session-preloading, par, preview,
+                       recovery-codes, scripts, step-up-authentication, token-exchange,
+                       transient-users, update-email, web-authn.
 
 Config:
 

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testStartDevHelp.unix.approved.txt

@@ -77,17 +77,18 @@ Feature:
                        admin2[:v1], authorization[:v1], ciba[:v1], client-policies[:v1],
                        client-secret-rotation[:v1], device-flow[:v1], docker[:v1], dpop[:v1],
                        dynamic-scopes[:v1], fips[:v1], impersonation[:v1], js-adapter[:v1], kerberos
-                       [:v1], linkedin-oauth[:v1], multi-site[:v1], par[:v1], preview,
-                       recovery-codes[:v1], scripts[:v1], step-up-authentication[:v1],
-                       token-exchange[:v1], transient-users[:v1], update-email[:v1], web-authn[:v1].
+                       [:v1], linkedin-oauth[:v1], multi-site[:v1], offline-session-preloading[:
+                       v1], par[:v1], preview, recovery-codes[:v1], scripts[:v1],
+                       step-up-authentication[:v1], token-exchange[:v1], transient-users[:v1],
+                       update-email[:v1], web-authn[:v1].
 --features-disabled <feature>
                      Disables a set of one or more features. Possible values are: account-api,
                        account2, account3, admin-api, admin-fine-grained-authz, admin2,
                        authorization, ciba, client-policies, client-secret-rotation, device-flow,
                        docker, dpop, dynamic-scopes, fips, impersonation, js-adapter, kerberos,
-                       linkedin-oauth, multi-site, par, preview, recovery-codes, scripts,
-                       step-up-authentication, token-exchange, transient-users, update-email,
-                       web-authn.
+                       linkedin-oauth, multi-site, offline-session-preloading, par, preview,
+                       recovery-codes, scripts, step-up-authentication, token-exchange,
+                       transient-users, update-email, web-authn.
 
 Hostname: