Skip to content

Commit

Permalink
docs: create new runtime section for certain docs
Browse files Browse the repository at this point in the history
  • Loading branch information
atipijasin committed Jan 31, 2025
1 parent a7edb5b commit c05db78
Show file tree
Hide file tree
Showing 10 changed files with 81 additions and 74 deletions.
123 changes: 63 additions & 60 deletions content/en/documentation/for-contributors/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,23 +11,24 @@ weight: 30
* [1.3 Architectural and coding principles](#13-architectural-and-coding-principles)
* [2. The control plane](#2-the-control-plane)
* [2.1 Entities](#21-entities)
* [2.2 Programming Primitives](#22-programming-primitives)
* [2.3 Serialization via JSON-LD](#23-serialization-via-json-ld)
* [2.4 Extension model](#24-extension-model)
* [2.5 Dependency injection deep dive](#25-dependency-injection-deep-dive)
* [2.6 Service layers](#26-service-layers)
* [2.7 Policy Monitor](#27-policy-monitor)
* [2.8 Protocol extensions (DSP)](#28-protocol-extensions-dsp)
* [2.2 Policy Monitor](#22-policy-monitor)
* [2.3 Protocol extensions (DSP)](#23-protocol-extensions-dsp)
* [3. (Postgre-)SQL persistence](#3-postgre-sql-persistence)
* [4. The data plane](#4-the-data-plane)
* [4.1 Data plane signaling](#41-data-plane-signaling)
* [4.2 Writing a custom data plane extension (sink/source)](#42-writing-a-custom-data-plane-extension-sinksource)
* [4.3 Writing a custom data plane (using only DPS)](#43-writing-a-custom-data-plane-using-only-dps)
* [5. Development best practices](#5-development-best-practices)
* [5.1 Writing Unit-, Component-, Integration-, Api-, EndToEnd-Tests](#51-writing-unit--component--integration--api--endtoend-tests)
* [5.1 Coding best practices](#51-coding-best-practices)
* [5.2 Testing best practices](#52-testing-best-practices)
* [5.3 Other best practices](#53-other-best-practices)
* [5. Runtime](#5-runtime)
* [5.1 Serialization via JSON-LD](#51-serialization-via-json-ld)
* [5.2 Programming Primitives](#52-programming-primitives)
* [5.3 Extension model](#53-extension-model)
* [5.4 Dependency injection deep dive](#54-dependency-injection-deep-dive)
* [5.5 Service layers](#55-service-layers)
* [6. Development best practices](#6-development-best-practices)
* [6.1 Writing Unit-, Component-, Integration-, Api-, EndToEnd-Tests](#61-writing-unit--component--integration--api--endtoend-tests)
* [6.1 Coding best practices](#61-coding-best-practices)
* [6.2 Testing best practices](#62-testing-best-practices)
* [6.3 Other best practices](#63-other-best-practices)
* [6. Further concepts](#6-further-concepts)
* [6.1 Autodoc](#61-autodoc)
* [6.2 Adapting the Gradle build](#62-adapting-the-gradle-build)
Expand Down Expand Up @@ -127,53 +128,14 @@ are located at `extensions/control-plane/api/management-api`.

Detailed documentation about entities can be found [here](./control-plane/entities.md)

### 2.2 Programming Primitives

This chapter describes the fundamental architectural and programming paradigms that are used in EDC. Typically, they
are not related to one single extension or feature area, they are of overarching character.

Detailed documentation about programming primitives can be found [here](./control-plane/programming-primitives.md)

### 2.3 Serialization via JSON-LD

JSON-LD is a JSON-based format for serializing [Linked Data](https://www.w3.org/wiki/LinkedData), and allows adding
specific "context" to the data expressed as JSON format.
It is a [W3C](https://www.w3.org/TR/json-ld/) standard since 2010.

Detailed information about how JSON-LD is used in EDC can be found [here](./control-plane/json-ld.md)

### 2.4 Extension model

One of the principles EDC is built around is _extensibility_. This means that by simply putting a Java module on the
classpath, the code in it will be used to enrich and influence the runtime behaviour of EDC. For instance, contributing
additional data persistence implementations can be achieved this way. This is sometimes also referred to as "plugin".

Detailed documentation about the EDC extension model can be found [here](./control-plane/extension-model.md)

### 2.5 Dependency injection deep dive

In EDC, dependency injection is available to inject services into extension classes (implementors of the
`ServiceExtension` interface). The `ServiceExtensionContext` acts as service registry, and since it's not _quite_ an IoC
container, we'll refer to it simple as the "context" in this chapter.

Detailed documentation about the EDC dependency injection mechanism can be
found [here](./control-plane/dependency-injection.md)

### 2.6 Service layers

Like many other applications and application frameworks, EDC is built upon a vertically oriented set of different layers
that we call "service layers".

Detailed documentation about the EDC service layers can be found [here](./control-plane/service-layers.md)

### 2.7 Policy Monitor
### 2.2 Policy Monitor

The policy monitor is a component that watches over on-going transfers and ensures that the policies associated with the
transfer are still valid.

Detailed documentation about the policy monitor can be found [here](./control-plane/policy-monitor.md)

### 2.8 Protocol extensions (DSP)
### 2.3 Protocol extensions (DSP)

This chapter describes how EDC abstracts the interaction between connectors in a Dataspace through protocol extensions
and introduces the current default implementation which follows the [Dataspace
Expand All @@ -186,13 +148,13 @@ Detailed documentation about protocol extensions can be found [here](./control-p
PostgreSQL is a very popular open-source database and it has a large community and vendor adoption. It is also EDCs data
persistence technology of choice.

Every [store](./control-plane/service-layers.md#5-data-persistence) in the EDC, intended to persist state, comes out of
Every [store](runtime/service-layers.md#5-data-persistence) in the EDC, intended to persist state, comes out of
the box with two implementations:

- in-memory
- sql (PostgreSQL dialect)

By default, the [in-memory stores](./control-plane/service-layers.md#51-in-memory-stores) are provided by the dependency
By default, the [in-memory stores](runtime/service-layers.md#51-in-memory-stores) are provided by the dependency
injection, the SQL variants can be used by simply adding the relevant extensions (e.g. `asset-index-sql`,
`contract-negotiation-store-sql`, ...) to the classpath.

Expand Down Expand Up @@ -222,17 +184,58 @@ to write a data plane from scratch (without using EDC and [DPF](./data-plane/dat

Detailed documentation about writing a custom data plane be found [here](./data-plane/custom-data-plane.md).

## 5. Runtime

### 5.1 Serialization via JSON-LD

JSON-LD is a JSON-based format for serializing [Linked Data](https://www.w3.org/wiki/LinkedData), and allows adding
specific "context" to the data expressed as JSON format.
It is a [W3C](https://www.w3.org/TR/json-ld/) standard since 2010.

Detailed information about how JSON-LD is used in EDC can be found [here](runtime/json-ld.md)

### 5.2 Programming Primitives

This chapter describes the fundamental architectural and programming paradigms that are used in EDC. Typically, they
are not related to one single extension or feature area, they are of overarching character.

Detailed documentation about programming primitives can be found [here](runtime/programming-primitives.md)


### 5.3 Extension model

One of the principles EDC is built around is _extensibility_. This means that by simply putting a Java module on the
classpath, the code in it will be used to enrich and influence the runtime behaviour of EDC. For instance, contributing
additional data persistence implementations can be achieved this way. This is sometimes also referred to as "plugin".

Detailed documentation about the EDC extension model can be found [here](runtime/extension-model.md)

### 5.4 Dependency injection deep dive

In EDC, dependency injection is available to inject services into extension classes (implementors of the
`ServiceExtension` interface). The `ServiceExtensionContext` acts as service registry, and since it's not _quite_ an IoC
container, we'll refer to it simple as the "context" in this chapter.

Detailed documentation about the EDC dependency injection mechanism can be
found [here](runtime/dependency-injection.md)

### 5.5 Service layers

Like many other applications and application frameworks, EDC is built upon a vertically oriented set of different layers
that we call "service layers".

Detailed documentation about the EDC service layers can be found [here](runtime/service-layers.md)

## 5. Development best practices
## 6. Development best practices

### 5.1 Writing Unit-, Component-, Integration-, Api-, EndToEnd-Tests
### 6.1 Writing Unit-, Component-, Integration-, Api-, EndToEnd-Tests

test pyramid... Like any other project, EDC has established a set of recommendations and rules that contributors must
adhere to in order to guarantee a smooth collaboration with the project. Note that familiarity with our [formal
contribution guidelines](./guidelines) is assumed. There additional recommendations we have compiled that
are relevant when deploying and administering EDC instances.

### 5.1 Coding best practices
### 6.1 Coding best practices

Code should be written to conform with the EDC [style guide](./guidelines/styleguide).

Expand All @@ -241,13 +244,13 @@ method X" or "Performing action Z" should be avoided because they pollute the lo

Please find detailed information about logging [here](logging.md).

### 5.2 Testing best practices
### 6.2 Testing best practices

Every class in the EDC code base should have a test class that verifies the correct functionality of the code.

Detailed information about testing can be found [here](./testing.md).

### 5.3 Other best practices
### 6.3 Other best practices

Please find general best practices and recommendations [here](./best-practices.md).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -640,7 +640,7 @@ any API.

Contract negotiations are asynchronous in nature. That means after initiating them, they become (potentially
long-running) stateful processes that are advanced by an
internal [state machine](../programming-primitives.md#1-state-machines).
internal [state machine](../../runtime/programming-primitives.md#1-state-machines).
The current state of the negotiation can be queried and altered through the management API.

Here's a diagram of the state machine applied to contract negotiations:
Expand Down Expand Up @@ -839,7 +839,7 @@ and construct the list of

A `TransferProcess` is a record of the data sharing procedure between a _consumer_ and a _provider_. As they traverse
through the system, they transition through several
states ([`TransferProcessStates`](../programming-primitives.md#1-state-machines)).
states ([`TransferProcessStates`](../../runtime/programming-primitives.md#1-state-machines)).

Once a contract is [negotiated](#4-contract-negotiations) and an [agreement](#5-contract-agreements) is reached, the
consumer connector may send a transfer initiate request to start the transfer. In the course of doing that, both parties
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Some transfer types, once accepted by the `provider`, never reach the `COMPLETED

The policy monitor (`PolicyMonitorManager`) is a component that watches over on-going transfers on the provider side and ensures that the associated policies are still valid. The default implementation of the policy monitor tracks the monitored transfer processes in it's own entity `PolicyMonitorEntry` stored in the `PolicyMonitorStore`.

Once a transfer process transition to the `STARTED` state on the provider side, the policy monitor gets notified through the [eventing system](./service-layers.md#6-events-and-callbacks) of EDC and start tracking transfer process. For each monitored transfer process in the `STARTED` state the policy monitor retrieves the policy associated (through [contract agreement](./entities.md#5-contract-agreements)) and runs the [Policy Engine](./entities.md#22-policy-scopes-and-bindings) using the `policy.monitor` as scope.
Once a transfer process transition to the `STARTED` state on the provider side, the policy monitor gets notified through the [eventing system](../runtime/service-layers.md#6-events-and-callbacks) of EDC and start tracking transfer process. For each monitored transfer process in the `STARTED` state the policy monitor retrieves the policy associated (through [contract agreement](./entities.md#5-contract-agreements)) and runs the [Policy Engine](./entities.md#22-policy-scopes-and-bindings) using the `policy.monitor` as scope.
If the policy is no longer valid, the policy monitor marks the transfer process for termination (`TERMINATING`) and stops tracking it.

The data plane also gets notified through the [data plane signaling](../contributor-handbook.md#210-data-plane-signaling) protocol about the termination of the transfer process, and if accepted by the data plane, the data transfer terminates as well.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,11 @@ and it is registered in the `RemoteMessageDispatcherRegistry`, where it gets ass
Internally EDC uses the `RemoteMessageDispatcherRegistry` whenever it needs to deliver a `RemoteMessage` to the counter-party. The `RemoteMessage` then gets routed to the right `RemoteMessageDispatcher` based on the `RemoteMessage#getProtocol` property.


> EDC also uses `RemoteMessageDispatcherRegistry` for non-protocol messages when dispatching [event callbacks](./service-layers.md#63-registering-for-callbacks-webhooks)
> EDC also uses `RemoteMessageDispatcherRegistry` for non-protocol messages when dispatching [event callbacks](../runtime/service-layers.md#63-registering-for-callbacks-webhooks)
### 1.2 Handling incoming messages with protocol services

On the ingress side, protocol implementations should be able to receive messages through the network (e.g. [API Controllers](./service-layers.md#1-api-controllers)), deserialize them into the corresponding `RemoteMessage`s and then dispatching them to the right protocol service.
On the ingress side, protocol implementations should be able to receive messages through the network (e.g. [API Controllers](../runtime/service-layers.md#1-api-controllers)), deserialize them into the corresponding `RemoteMessage`s and then dispatching them to the right protocol service.

Protocol services are three:

Expand Down Expand Up @@ -93,7 +93,7 @@ Handlers map a `RemoteMessage` to an HTTP Request and instruct the `DspHttpRemot

### 2.2 HTTP endpoints

Each `dsp-*-http-api` module exposes its own [API Controllers](./service-layers.md#1-api-controllers) for serving the specification requests. Each request handler transforms the JSON-LD in input, if present, into a `RemoteMessage` and then calls the [protocol service](#12-handling-incoming-messages-with-protocol-services) layer.
Each `dsp-*-http-api` module exposes its own [API Controllers](../runtime/service-layers.md#1-api-controllers) for serving the specification requests. Each request handler transforms the JSON-LD in input, if present, into a `RemoteMessage` and then calls the [protocol service](#12-handling-incoming-messages-with-protocol-services) layer.

### 2.2 `RemoteMessage` transformers

Expand Down
4 changes: 4 additions & 0 deletions content/en/documentation/for-contributors/runtime/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
title: Runtime
weight: 80
---
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ For playing around JSON-LD and processing algorithm the [playground](https://jso

EDC uses JSON-LD as primary serialization format at API layer and at runtime EDC manages the objects in their expanded
form, for example when transforming `JsonObject` into EDC entities and and backwards
in [transformers](./programming-primitives.md#2-transformers) or when [validating](#11-json-ld-validation) input
in [transformers](programming-primitives.md#2-transformers) or when [validating](#11-json-ld-validation) input
`JsonObject` at API level.

> Extensible properties in entities are always stored expanded form.
Expand Down Expand Up @@ -387,7 +387,7 @@ public class FooValidator {
}
```

and can be registered with the [@Injectable](./dependency-injection.md#2-injecting-a-service)
and can be registered with the [@Injectable](../control-plane/dependency-injection.md#2-injecting-a-service)
`JsonObjectValidatorRegistry`:

```java
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ process and advance its state.
## 2. Transformers

EDC uses JSON-LD serialization on API ingress and egress. For information about this can be found [in this
chapter](./json-ld.md), but the TL;DR is that it is necessary because of extensible properties and
chapter](json-ld.md), but the TL;DR is that it is necessary because of extensible properties and
namespaces on wire-level DTOs.

### 2.1 Basic Serialization and Deserialization
Expand Down Expand Up @@ -195,11 +195,11 @@ _Note that this example serves for illustration purposes only!_
Usually, transformation happens in API controllers to deserialize input, process and serialize output, but controllers
don't use transformers directly because more than one transformer may be required to correctly deserialize an object.
Rather, they have a reference to a `TypeTransformerRegistry` for this. For more information please refer to the [chapter
about service layers](./service-layers.md).
about service layers](service-layers.md).

### 2.3 Reporting transformation errors

Generally speaking, input validation should be performed by [validators](./service-layers.md#2-validators). However, it
Generally speaking, input validation should be performed by [validators](service-layers.md#2-validators). However, it
is still possible that an object cannot be serialized/deserialized correctly, for example when a property has has the
wrong type, wrong multiplicity, cannot be parsed, unknown property, etc. Those types of errors should be reported to the
`TransformerContext`:
Expand All @@ -221,12 +221,12 @@ private void transformProperties(String key, JsonValue jsonValue, DataPlaneInsta
```

Transformers should report errors to the context instead of throwing exceptions. Please note that basic JSON validation
should be performed by [validators](./service-layers.md#2-validators).
should be performed by [validators](service-layers.md#2-validators).

## 3. Token generation and decorators

A token is a datastructure that consists of a header and claims and that is signed with a private key. While EDC
is able to create any type of tokens through [extensions](./extension-model.md), in most use cases JSON Web Tokens (JWT)
is able to create any type of tokens through [extensions](../control-plane/extension-model.md), in most use cases JSON Web Tokens (JWT)
are a good option.

The `TokenGenerationService` offers a way to generate such a token by passing in a reference to a private key and a set
Expand All @@ -246,7 +246,7 @@ be opinionated with regards to the token technology.

When receiving a token, EDC makes use of the `TokenValidationService` facility to verify and validate the incoming
token. Out-of-the-box JWTs are supported, but other token types could be supported through
[extensions](./extension-model.md). This section will be limited to validating JWT tokens.
[extensions](../control-plane/extension-model.md). This section will be limited to validating JWT tokens.

Every JWT that is validated by EDC _must_ have a `kid` header indicating the ID of the public key with which the token
can be verified. In addition, a `PublicKeyResolver` implementation is required to download the public key.
Expand Down

0 comments on commit c05db78

Please sign in to comment.