diff --git a/content/en/documentation/for-contributors/_index.md b/content/en/documentation/for-contributors/_index.md index 8edfd80..f53047d 100644 --- a/content/en/documentation/for-contributors/_index.md +++ b/content/en/documentation/for-contributors/_index.md @@ -11,23 +11,24 @@ weight: 30 * [1.3 Architectural and coding principles](#13-architectural-and-coding-principles) * [2. The control plane](#2-the-control-plane) * [2.1 Entities](#21-entities) - * [2.2 Programming Primitives](#22-programming-primitives) - * [2.3 Serialization via JSON-LD](#23-serialization-via-json-ld) - * [2.4 Extension model](#24-extension-model) - * [2.5 Dependency injection deep dive](#25-dependency-injection-deep-dive) - * [2.6 Service layers](#26-service-layers) - * [2.7 Policy Monitor](#27-policy-monitor) - * [2.8 Protocol extensions (DSP)](#28-protocol-extensions-dsp) + * [2.2 Policy Monitor](#22-policy-monitor) + * [2.3 Protocol extensions (DSP)](#23-protocol-extensions-dsp) * [3. (Postgre-)SQL persistence](#3-postgre-sql-persistence) * [4. The data plane](#4-the-data-plane) * [4.1 Data plane signaling](#41-data-plane-signaling) * [4.2 Writing a custom data plane extension (sink/source)](#42-writing-a-custom-data-plane-extension-sinksource) * [4.3 Writing a custom data plane (using only DPS)](#43-writing-a-custom-data-plane-using-only-dps) - * [5. Development best practices](#5-development-best-practices) - * [5.1 Writing Unit-, Component-, Integration-, Api-, EndToEnd-Tests](#51-writing-unit--component--integration--api--endtoend-tests) - * [5.1 Coding best practices](#51-coding-best-practices) - * [5.2 Testing best practices](#52-testing-best-practices) - * [5.3 Other best practices](#53-other-best-practices) + * [5. Runtime](#5-runtime) + * [5.1 Serialization via JSON-LD](#51-serialization-via-json-ld) + * [5.2 Programming Primitives](#52-programming-primitives) + * [5.3 Extension model](#53-extension-model) + * [5.4 Dependency injection deep dive](#54-dependency-injection-deep-dive) + * [5.5 Service layers](#55-service-layers) + * [6. Development best practices](#6-development-best-practices) + * [6.1 Writing Unit-, Component-, Integration-, Api-, EndToEnd-Tests](#61-writing-unit--component--integration--api--endtoend-tests) + * [6.1 Coding best practices](#61-coding-best-practices) + * [6.2 Testing best practices](#62-testing-best-practices) + * [6.3 Other best practices](#63-other-best-practices) * [6. Further concepts](#6-further-concepts) * [6.1 Autodoc](#61-autodoc) * [6.2 Adapting the Gradle build](#62-adapting-the-gradle-build) @@ -127,53 +128,14 @@ are located at `extensions/control-plane/api/management-api`. Detailed documentation about entities can be found [here](./control-plane/entities.md) -### 2.2 Programming Primitives - -This chapter describes the fundamental architectural and programming paradigms that are used in EDC. Typically, they -are not related to one single extension or feature area, they are of overarching character. - -Detailed documentation about programming primitives can be found [here](./control-plane/programming-primitives.md) - -### 2.3 Serialization via JSON-LD - -JSON-LD is a JSON-based format for serializing [Linked Data](https://www.w3.org/wiki/LinkedData), and allows adding -specific "context" to the data expressed as JSON format. -It is a [W3C](https://www.w3.org/TR/json-ld/) standard since 2010. - -Detailed information about how JSON-LD is used in EDC can be found [here](./control-plane/json-ld.md) - -### 2.4 Extension model - -One of the principles EDC is built around is _extensibility_. This means that by simply putting a Java module on the -classpath, the code in it will be used to enrich and influence the runtime behaviour of EDC. For instance, contributing -additional data persistence implementations can be achieved this way. This is sometimes also referred to as "plugin". - -Detailed documentation about the EDC extension model can be found [here](./control-plane/extension-model.md) - -### 2.5 Dependency injection deep dive - -In EDC, dependency injection is available to inject services into extension classes (implementors of the -`ServiceExtension` interface). The `ServiceExtensionContext` acts as service registry, and since it's not _quite_ an IoC -container, we'll refer to it simple as the "context" in this chapter. - -Detailed documentation about the EDC dependency injection mechanism can be -found [here](./control-plane/dependency-injection.md) - -### 2.6 Service layers - -Like many other applications and application frameworks, EDC is built upon a vertically oriented set of different layers -that we call "service layers". - -Detailed documentation about the EDC service layers can be found [here](./control-plane/service-layers.md) - -### 2.7 Policy Monitor +### 2.2 Policy Monitor The policy monitor is a component that watches over on-going transfers and ensures that the policies associated with the transfer are still valid. Detailed documentation about the policy monitor can be found [here](./control-plane/policy-monitor.md) -### 2.8 Protocol extensions (DSP) +### 2.3 Protocol extensions (DSP) This chapter describes how EDC abstracts the interaction between connectors in a Dataspace through protocol extensions and introduces the current default implementation which follows the [Dataspace @@ -186,13 +148,13 @@ Detailed documentation about protocol extensions can be found [here](./control-p PostgreSQL is a very popular open-source database and it has a large community and vendor adoption. It is also EDCs data persistence technology of choice. -Every [store](./control-plane/service-layers.md#5-data-persistence) in the EDC, intended to persist state, comes out of +Every [store](runtime/service-layers.md#5-data-persistence) in the EDC, intended to persist state, comes out of the box with two implementations: - in-memory - sql (PostgreSQL dialect) -By default, the [in-memory stores](./control-plane/service-layers.md#51-in-memory-stores) are provided by the dependency +By default, the [in-memory stores](runtime/service-layers.md#51-in-memory-stores) are provided by the dependency injection, the SQL variants can be used by simply adding the relevant extensions (e.g. `asset-index-sql`, `contract-negotiation-store-sql`, ...) to the classpath. @@ -222,17 +184,58 @@ to write a data plane from scratch (without using EDC and [DPF](./data-plane/dat Detailed documentation about writing a custom data plane be found [here](./data-plane/custom-data-plane.md). +## 5. Runtime + +### 5.1 Serialization via JSON-LD + +JSON-LD is a JSON-based format for serializing [Linked Data](https://www.w3.org/wiki/LinkedData), and allows adding +specific "context" to the data expressed as JSON format. +It is a [W3C](https://www.w3.org/TR/json-ld/) standard since 2010. + +Detailed information about how JSON-LD is used in EDC can be found [here](runtime/json-ld.md) + +### 5.2 Programming Primitives + +This chapter describes the fundamental architectural and programming paradigms that are used in EDC. Typically, they +are not related to one single extension or feature area, they are of overarching character. + +Detailed documentation about programming primitives can be found [here](runtime/programming-primitives.md) + + +### 5.3 Extension model + +One of the principles EDC is built around is _extensibility_. This means that by simply putting a Java module on the +classpath, the code in it will be used to enrich and influence the runtime behaviour of EDC. For instance, contributing +additional data persistence implementations can be achieved this way. This is sometimes also referred to as "plugin". + +Detailed documentation about the EDC extension model can be found [here](runtime/extension-model.md) + +### 5.4 Dependency injection deep dive + +In EDC, dependency injection is available to inject services into extension classes (implementors of the +`ServiceExtension` interface). The `ServiceExtensionContext` acts as service registry, and since it's not _quite_ an IoC +container, we'll refer to it simple as the "context" in this chapter. + +Detailed documentation about the EDC dependency injection mechanism can be +found [here](runtime/dependency-injection.md) + +### 5.5 Service layers + +Like many other applications and application frameworks, EDC is built upon a vertically oriented set of different layers +that we call "service layers". + +Detailed documentation about the EDC service layers can be found [here](runtime/service-layers.md) -## 5. Development best practices +## 6. Development best practices -### 5.1 Writing Unit-, Component-, Integration-, Api-, EndToEnd-Tests +### 6.1 Writing Unit-, Component-, Integration-, Api-, EndToEnd-Tests test pyramid... Like any other project, EDC has established a set of recommendations and rules that contributors must adhere to in order to guarantee a smooth collaboration with the project. Note that familiarity with our [formal contribution guidelines](./guidelines) is assumed. There additional recommendations we have compiled that are relevant when deploying and administering EDC instances. -### 5.1 Coding best practices +### 6.1 Coding best practices Code should be written to conform with the EDC [style guide](./guidelines/styleguide). @@ -241,13 +244,13 @@ method X" or "Performing action Z" should be avoided because they pollute the lo Please find detailed information about logging [here](logging.md). -### 5.2 Testing best practices +### 6.2 Testing best practices Every class in the EDC code base should have a test class that verifies the correct functionality of the code. Detailed information about testing can be found [here](./testing.md). -### 5.3 Other best practices +### 6.3 Other best practices Please find general best practices and recommendations [here](./best-practices.md). diff --git a/content/en/documentation/for-contributors/control-plane/entities/_index.md b/content/en/documentation/for-contributors/control-plane/entities/_index.md index 4381e7d..0d9b702 100644 --- a/content/en/documentation/for-contributors/control-plane/entities/_index.md +++ b/content/en/documentation/for-contributors/control-plane/entities/_index.md @@ -640,7 +640,7 @@ any API. Contract negotiations are asynchronous in nature. That means after initiating them, they become (potentially long-running) stateful processes that are advanced by an -internal [state machine](../programming-primitives.md#1-state-machines). +internal [state machine](../../runtime/programming-primitives.md#1-state-machines). The current state of the negotiation can be queried and altered through the management API. Here's a diagram of the state machine applied to contract negotiations: @@ -839,7 +839,7 @@ and construct the list of A `TransferProcess` is a record of the data sharing procedure between a _consumer_ and a _provider_. As they traverse through the system, they transition through several -states ([`TransferProcessStates`](../programming-primitives.md#1-state-machines)). +states ([`TransferProcessStates`](../../runtime/programming-primitives.md#1-state-machines)). Once a contract is [negotiated](#4-contract-negotiations) and an [agreement](#5-contract-agreements) is reached, the consumer connector may send a transfer initiate request to start the transfer. In the course of doing that, both parties diff --git a/content/en/documentation/for-contributors/control-plane/policy-monitor.md b/content/en/documentation/for-contributors/control-plane/policy-monitor.md index 654fb8f..3bffc4e 100644 --- a/content/en/documentation/for-contributors/control-plane/policy-monitor.md +++ b/content/en/documentation/for-contributors/control-plane/policy-monitor.md @@ -8,7 +8,7 @@ Some transfer types, once accepted by the `provider`, never reach the `COMPLETED The policy monitor (`PolicyMonitorManager`) is a component that watches over on-going transfers on the provider side and ensures that the associated policies are still valid. The default implementation of the policy monitor tracks the monitored transfer processes in it's own entity `PolicyMonitorEntry` stored in the `PolicyMonitorStore`. -Once a transfer process transition to the `STARTED` state on the provider side, the policy monitor gets notified through the [eventing system](./service-layers.md#6-events-and-callbacks) of EDC and start tracking transfer process. For each monitored transfer process in the `STARTED` state the policy monitor retrieves the policy associated (through [contract agreement](./entities.md#5-contract-agreements)) and runs the [Policy Engine](./entities.md#22-policy-scopes-and-bindings) using the `policy.monitor` as scope. +Once a transfer process transition to the `STARTED` state on the provider side, the policy monitor gets notified through the [eventing system](../runtime/service-layers.md#6-events-and-callbacks) of EDC and start tracking transfer process. For each monitored transfer process in the `STARTED` state the policy monitor retrieves the policy associated (through [contract agreement](./entities.md#5-contract-agreements)) and runs the [Policy Engine](./entities.md#22-policy-scopes-and-bindings) using the `policy.monitor` as scope. If the policy is no longer valid, the policy monitor marks the transfer process for termination (`TERMINATING`) and stops tracking it. The data plane also gets notified through the [data plane signaling](../contributor-handbook.md#210-data-plane-signaling) protocol about the termination of the transfer process, and if accepted by the data plane, the data transfer terminates as well. diff --git a/content/en/documentation/for-contributors/control-plane/protocol-extensions.md b/content/en/documentation/for-contributors/control-plane/protocol-extensions.md index baf6dc1..9c861d8 100644 --- a/content/en/documentation/for-contributors/control-plane/protocol-extensions.md +++ b/content/en/documentation/for-contributors/control-plane/protocol-extensions.md @@ -52,11 +52,11 @@ and it is registered in the `RemoteMessageDispatcherRegistry`, where it gets ass Internally EDC uses the `RemoteMessageDispatcherRegistry` whenever it needs to deliver a `RemoteMessage` to the counter-party. The `RemoteMessage` then gets routed to the right `RemoteMessageDispatcher` based on the `RemoteMessage#getProtocol` property. -> EDC also uses `RemoteMessageDispatcherRegistry` for non-protocol messages when dispatching [event callbacks](./service-layers.md#63-registering-for-callbacks-webhooks) +> EDC also uses `RemoteMessageDispatcherRegistry` for non-protocol messages when dispatching [event callbacks](../runtime/service-layers.md#63-registering-for-callbacks-webhooks) ### 1.2 Handling incoming messages with protocol services -On the ingress side, protocol implementations should be able to receive messages through the network (e.g. [API Controllers](./service-layers.md#1-api-controllers)), deserialize them into the corresponding `RemoteMessage`s and then dispatching them to the right protocol service. +On the ingress side, protocol implementations should be able to receive messages through the network (e.g. [API Controllers](../runtime/service-layers.md#1-api-controllers)), deserialize them into the corresponding `RemoteMessage`s and then dispatching them to the right protocol service. Protocol services are three: @@ -93,7 +93,7 @@ Handlers map a `RemoteMessage` to an HTTP Request and instruct the `DspHttpRemot ### 2.2 HTTP endpoints -Each `dsp-*-http-api` module exposes its own [API Controllers](./service-layers.md#1-api-controllers) for serving the specification requests. Each request handler transforms the JSON-LD in input, if present, into a `RemoteMessage` and then calls the [protocol service](#12-handling-incoming-messages-with-protocol-services) layer. +Each `dsp-*-http-api` module exposes its own [API Controllers](../runtime/service-layers.md#1-api-controllers) for serving the specification requests. Each request handler transforms the JSON-LD in input, if present, into a `RemoteMessage` and then calls the [protocol service](#12-handling-incoming-messages-with-protocol-services) layer. ### 2.2 `RemoteMessage` transformers diff --git a/content/en/documentation/for-contributors/runtime/_index.md b/content/en/documentation/for-contributors/runtime/_index.md new file mode 100644 index 0000000..f9aa38d --- /dev/null +++ b/content/en/documentation/for-contributors/runtime/_index.md @@ -0,0 +1,4 @@ +--- +title: Runtime +weight: 80 +--- diff --git a/content/en/documentation/for-contributors/control-plane/dependency-injection.md b/content/en/documentation/for-contributors/runtime/dependency-injection.md similarity index 100% rename from content/en/documentation/for-contributors/control-plane/dependency-injection.md rename to content/en/documentation/for-contributors/runtime/dependency-injection.md diff --git a/content/en/documentation/for-contributors/control-plane/extension-model.md b/content/en/documentation/for-contributors/runtime/extension-model.md similarity index 100% rename from content/en/documentation/for-contributors/control-plane/extension-model.md rename to content/en/documentation/for-contributors/runtime/extension-model.md diff --git a/content/en/documentation/for-contributors/control-plane/json-ld.md b/content/en/documentation/for-contributors/runtime/json-ld.md similarity index 98% rename from content/en/documentation/for-contributors/control-plane/json-ld.md rename to content/en/documentation/for-contributors/runtime/json-ld.md index 5c0f242..516be82 100644 --- a/content/en/documentation/for-contributors/control-plane/json-ld.md +++ b/content/en/documentation/for-contributors/runtime/json-ld.md @@ -113,7 +113,7 @@ For playing around JSON-LD and processing algorithm the [playground](https://jso EDC uses JSON-LD as primary serialization format at API layer and at runtime EDC manages the objects in their expanded form, for example when transforming `JsonObject` into EDC entities and and backwards -in [transformers](./programming-primitives.md#2-transformers) or when [validating](#11-json-ld-validation) input +in [transformers](programming-primitives.md#2-transformers) or when [validating](#11-json-ld-validation) input `JsonObject` at API level. > Extensible properties in entities are always stored expanded form. @@ -387,7 +387,7 @@ public class FooValidator { } ``` -and can be registered with the [@Injectable](./dependency-injection.md#2-injecting-a-service) +and can be registered with the [@Injectable](../control-plane/dependency-injection.md#2-injecting-a-service) `JsonObjectValidatorRegistry`: ```java diff --git a/content/en/documentation/for-contributors/control-plane/programming-primitives.md b/content/en/documentation/for-contributors/runtime/programming-primitives.md similarity index 96% rename from content/en/documentation/for-contributors/control-plane/programming-primitives.md rename to content/en/documentation/for-contributors/runtime/programming-primitives.md index bd91d77..0db2538 100644 --- a/content/en/documentation/for-contributors/control-plane/programming-primitives.md +++ b/content/en/documentation/for-contributors/runtime/programming-primitives.md @@ -104,7 +104,7 @@ process and advance its state. ## 2. Transformers EDC uses JSON-LD serialization on API ingress and egress. For information about this can be found [in this -chapter](./json-ld.md), but the TL;DR is that it is necessary because of extensible properties and +chapter](json-ld.md), but the TL;DR is that it is necessary because of extensible properties and namespaces on wire-level DTOs. ### 2.1 Basic Serialization and Deserialization @@ -195,11 +195,11 @@ _Note that this example serves for illustration purposes only!_ Usually, transformation happens in API controllers to deserialize input, process and serialize output, but controllers don't use transformers directly because more than one transformer may be required to correctly deserialize an object. Rather, they have a reference to a `TypeTransformerRegistry` for this. For more information please refer to the [chapter -about service layers](./service-layers.md). +about service layers](service-layers.md). ### 2.3 Reporting transformation errors -Generally speaking, input validation should be performed by [validators](./service-layers.md#2-validators). However, it +Generally speaking, input validation should be performed by [validators](service-layers.md#2-validators). However, it is still possible that an object cannot be serialized/deserialized correctly, for example when a property has has the wrong type, wrong multiplicity, cannot be parsed, unknown property, etc. Those types of errors should be reported to the `TransformerContext`: @@ -221,12 +221,12 @@ private void transformProperties(String key, JsonValue jsonValue, DataPlaneInsta ``` Transformers should report errors to the context instead of throwing exceptions. Please note that basic JSON validation -should be performed by [validators](./service-layers.md#2-validators). +should be performed by [validators](service-layers.md#2-validators). ## 3. Token generation and decorators A token is a datastructure that consists of a header and claims and that is signed with a private key. While EDC -is able to create any type of tokens through [extensions](./extension-model.md), in most use cases JSON Web Tokens (JWT) +is able to create any type of tokens through [extensions](../control-plane/extension-model.md), in most use cases JSON Web Tokens (JWT) are a good option. The `TokenGenerationService` offers a way to generate such a token by passing in a reference to a private key and a set @@ -246,7 +246,7 @@ be opinionated with regards to the token technology. When receiving a token, EDC makes use of the `TokenValidationService` facility to verify and validate the incoming token. Out-of-the-box JWTs are supported, but other token types could be supported through -[extensions](./extension-model.md). This section will be limited to validating JWT tokens. +[extensions](../control-plane/extension-model.md). This section will be limited to validating JWT tokens. Every JWT that is validated by EDC _must_ have a `kid` header indicating the ID of the public key with which the token can be verified. In addition, a `PublicKeyResolver` implementation is required to download the public key. diff --git a/content/en/documentation/for-contributors/control-plane/service-layers.md b/content/en/documentation/for-contributors/runtime/service-layers.md similarity index 100% rename from content/en/documentation/for-contributors/control-plane/service-layers.md rename to content/en/documentation/for-contributors/runtime/service-layers.md