Use event-queue as outbox for CAP (#107)

* wip * wip * wip * wip * wip * wip * wip * wip * fix integration test * wip * add unboxed * changelog and version bump * wip * wip * docu * wip * wip * wip * wip * wip * wip * more tests * allow usage of config.parallelEventProcessing * further improvements * wip * wip * wip * doku * doku * doku * component name should start with slash * adjust tests --------- Co-authored-by: Max Gruenfelder <[email protected]>
cap-js-community · Jan 22, 2024 · afc2fcb · afc2fcb
1 parent 7787c62
commit afc2fcb
Show file tree

Hide file tree

Showing 40 changed files with 947 additions and 110 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,13 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## v1.1.0 - 2024-01-22
+
+### Added
+
+- enable event-queue to work as CAP outbox. The flag `useAsCAPOutbox` replaces the CAP implementation by
+  the event-queue.
+
 ## v1.0.3 - 2024-01-08
 
 ### Fixed
@@ -22,7 +29,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ### Added
 
-- introduced `thresholdLoggingEventProcessing` config variable to adjust logging threshold for event processing [documentation](https://cap-js-community.github.io/event-queue/setup/#initialization-parameters)
+- introduced `thresholdLoggingEventProcessing` config variable to adjust logging threshold for event
+  processing [documentation](https://cap-js-community.github.io/event-queue/setup/#initialization-parameters)
 
 ## v1.0.0 - 2023-12-20
 

diff --git a/docs/implement-event/index.md b/docs/implement-event/index.md
@@ -1,7 +1,7 @@
 ---
 layout: default
 title: Implement Event
-nav_order: 7
+nav_order: 8
 ---
 
 <!-- prettier-ignore-start -->

diff --git a/docs/load-balancing/index.md b/docs/load-balancing/index.md
@@ -1,7 +1,7 @@
 ---
 layout: default
 title: Load Balancing
-nav_order: 5
+nav_order: 6
 ---
 
 <!-- prettier-ignore-start -->

diff --git a/docs/publish-event/index.md b/docs/publish-event/index.md
@@ -1,7 +1,7 @@
 ---
 layout: default
 title: Publishing of Events
-nav_order: 6
+nav_order: 7
 ---
 
 <!-- prettier-ignore-start -->

diff --git a/docs/setup/index.md b/docs/setup/index.md
@@ -66,3 +66,4 @@ The table includes the parameter name, a description of its purpose, and the def
 | skipCsnCheck                         | Whether or not to skip the CSN check. Only relevant if custom tables are supplied.                                                                                                | false                |
 | updatePeriodicEvents                 | Whether or not to update periodic events.                                                                                                                                         | true                 |
 | thresholdLoggingEventProcessing [ms] | Threshold after how many milliseconds the processing of a event or periodic event is logged for observability.                                                                    | 50                   |
+| useAsCAPOutbox                       | Uses the event-queue as the [outbox](https://cap.cloud.sap/docs/node.js/outbox) of CAP. Outbox called are stored and processed in the event-queue instead of the outbox of CAP.   | false                |
diff --git a/docs/status-handling/index.md b/docs/status-handling/index.md
@@ -1,7 +1,7 @@
 ---
 layout: default
 title: Event Status Handling
-nav_order: 9
+nav_order: 10
 ---
 
 <!-- prettier-ignore-start -->

diff --git a/docs/transaction-handling/index.md b/docs/transaction-handling/index.md
@@ -1,7 +1,7 @@
 ---
 layout: default
 title: Transactional Handling
-nav_order: 8
+nav_order: 9
 ---
 
 <!-- prettier-ignore-start -->

diff --git a/docs/use-as-cap-outbox/index.md b/docs/use-as-cap-outbox/index.md
@@ -0,0 +1,73 @@
+---
+layout: default
+title: Use as CAP Outbox
+nav_order: 5
+---
+
+<!-- prettier-ignore-start -->
+
+{: .no_toc}
+
+# Use as CAP Outbox
+
+<!-- prettier-ignore -->
+- TOC
+{: toc}
+<!-- prettier-ignore-end -->
+
+The event-queue can be used to replace the CAP outbox solution to achieve a unified and streamlined architecture for
+asynchronous processing. If this feature is activated, the event-queue replaces the outbox implementation of CAP with
+its own implementation during the bootstrap process. This allows leveraging the features of the event-queue, such as
+transaction modes, load balancing, and others, with outboxed CDS services.
+
+# How to enable the event-queue as outbox mechanism for CAP
+
+The initialization parameter `useAsCAPOutbox` enables the event-queue to act as a CAP outbox. To set this parameter,
+refer to the [setup](/event-queue/setup/#initialization-parameters) part of the documentation. This is the only
+configuration needed to enable the event-queue as a CAP outbox.
+
+# How to Configure an Outboxed Service
+
+Services can be outboxed without any additional configuration. In this scenario, the service is outboxed using the
+default parameters of the CAP outbox and the event-queue. Currently, the CAP outbox implementation supports the
+following parameters, which are mapped to the corresponding configuration parameters of the event-queue:
+
+| CAP outbox  | event-queue             |
+| ----------- | ----------------------- |
+| chunkSize   | selectMaxChunkSize      |
+| maxAttempts | retryAttempts           |
+| parallel    | parallelEventProcessing |
+
+The `parallel` parameter is treated specially. The CAP outbox supports `true` or `false` as values for this parameter.
+Since the event-queue allows specifying concurrency with a number, `parallel=true` is mapped
+to `parallelEventProcessing=5`, and `parallel=false` is mapped to `parallelEventProcessing=1`. For full flexibility, the
+configuration prioritizes the `parallelEventProcessing` parameter over `parallel`.
+
+Parameters are managed via the `cds.require` section, not through the config yml file as with other events. For details
+on maintaining the `cds.requires` section, refer to
+the [CAP documentation](https://cap.cloud.sap/docs/node.js/core-services#required-services). Below is an example of
+using the event-queue to outbox the `@cap-js/audit-logging` service:
+
+```json
+{
+  "cds": {
+    "requires": {
+      "audit-log": {
+        "outbox": {
+          "kind": "persistent-outbox",
+          "transactionMode": "alwaysRollback",
+          "maxAttempts": 5,
+          "checkForNextChunk": true,
+          "parallelEventProcessing": 5
+        }
+      }
+    }
+  }
+}
+```
+
+The parameters in the outbox section of a service are passed as configuration to the event-queue.
+The `persistent-outbox` kind allows the event-queue to persist events instead of executing them in memory, mirroring the
+behavior of the [CAP outbox](https://cap.cloud.sap/docs/node.js/outbox). The
+parameters `transactionMode`, `checkForNextChunk`, and `parallelEventProcessing` are
+exclusive to the event-queue.
diff --git a/example-cap-server/package.json b/example-cap-server/package.json
@@ -8,10 +8,13 @@
     "copy-library": "npx shx rm -rf node_modules/@cap-js-community/event-queue && npx shx mkdir -p node_modules/@cap-js-community/event-queue && npx shx cp -R ../package.json ../index.cds ../cds-plugin.js ../src ../db node_modules/@cap-js-community/event-queue"
   },
   "dependencies": {
-    "@sap/cds": "7.5.0",
-    "@sap/cds-mtxs": "1.14.0",
     "@cap-js-community/event-queue": "file:..",
-    "express": "^4.18.2"
+    "@cap-js/audit-logging": "^0.5.2",
+    "@sap/cds": "^7.5.0",
+    "@sap/cds-mtxs": "1.14.0",
+    "express": "^4.18.2",
+    "redis": "^4.6.12",
+    "verror": "^1.10.1"
   },
   "devDependencies": {
     "@cap-js/sqlite": "1.4.0",
@@ -27,6 +30,7 @@
       "runInterval": 30000
     },
     "requires": {
+      "outbox": true,
       "multitenancy": true,
       "cds.xt.ModelProviderService": true,
       "cds.xt.SaasProvisioningService": true,

diff --git a/jest.setupAfterEnv.js b/jest.setupAfterEnv.js
@@ -1,5 +1,7 @@
 "use strict";
 
+process.env.CDS_CONFIG = '{ "requires": { "outbox": "persistent-outbox" } }';
+
 // turn off regular and error logging;
 jest.spyOn(console, "log").mockImplementation();
 jest.spyOn(console, "info").mockImplementation();

diff --git a/package-lock.json b/package-lock.json
diff --git a/package.json b/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cap-js-community/event-queue",
-  "version": "1.0.3",
+  "version": "1.1.0",
   "description": "An event queue that enables secure transactional processing of asynchronous events, featuring instant event processing with Redis Pub/Sub and load distribution across all application instances.",
   "main": "src/index.js",
   "files": [

diff --git a/src/EventQueueProcessorBase.js b/src/EventQueueProcessorBase.js
@@ -12,7 +12,7 @@ const eventConfig = require("./config");
 const PerformanceTracer = require("./shared/PerformanceTracer");
 
 const IMPLEMENT_ERROR_MESSAGE = "needs to be reimplemented";
-const COMPONENT_NAME = "eventQueue/EventQueueProcessorBase";
+const COMPONENT_NAME = "/eventQueue/EventQueueProcessorBase";
 
 const DEFAULT_RETRY_ATTEMPTS = 3;
 const DEFAULT_PARALLEL_EVENT_PROCESSING = 1;
@@ -869,7 +869,7 @@ class EventQueueProcessorBase {
     return await checkAndUpdatePromise;
   }
 
-  async handleDistributedLock() {
+  async acquireDistributedLock() {
     if (this.concurrentEventProcessing) {
       return true;
     }