Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: improve transfer samples documentation #162

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions transfer/transfer-00-prerequisites/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@ One connector will be the "consumer" connector while the other will act as the
"provider" connector. More on that in the following chapters.

For the sake of simplicity, the provider and the consumer
will run on the same machine, but in a real world configuration, they will likely be on different
machines.
will run on the same machine, but in a real world configuration, they will likely be on different machines.

This sample will go through:

Expand All @@ -26,7 +25,7 @@ Before we can run a connector, we need to build the JAR file.

Execute this command in project root:

```bash
```shell
./gradlew transfer:transfer-00-prerequisites:connector:build
```

Expand Down Expand Up @@ -56,13 +55,13 @@ This property is used to define the endpoint exposed by the control plane to val

To run the provider, just run the following command

```bash
```shell
java -Dedc.keystore=transfer/transfer-00-prerequisites/resources/certs/cert.pfx -Dedc.keystore.password=123456 -Dedc.vault=transfer/transfer-00-prerequisites/resources/configuration/provider-vault.properties -Dedc.fs.config=transfer/transfer-00-prerequisites/resources/configuration/provider-configuration.properties -jar transfer/transfer-00-prerequisites/connector/build/libs/connector.jar
```

To run the consumer, just run the following command (different terminal)

```bash
```shell
java -Dedc.keystore=transfer/transfer-00-prerequisites/resources/certs/cert.pfx -Dedc.keystore.password=123456 -Dedc.vault=transfer/transfer-00-prerequisites/resources/configuration/consumer-vault.properties -Dedc.fs.config=transfer/transfer-00-prerequisites/resources/configuration/consumer-configuration.properties -jar transfer/transfer-00-prerequisites/connector/build/libs/connector.jar
```

Expand All @@ -85,7 +84,7 @@ request to the management API of the connector.

Open a new terminal and execute:

```bash
```shell
curl -H 'Content-Type: application/json' \
-d @transfer/transfer-00-prerequisites/resources/dataplane/register-data-plane-provider.json \
-X POST "http://localhost:19193/management/v2/dataplanes" -s | jq
Expand Down
56 changes: 25 additions & 31 deletions transfer/transfer-01-negotiation/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# Implement a negotiation workflow between two connectors


Before two connectors can exchange actual data, negotiation has to take place.

The final goal of this example is to showcase the negotiation workflow between two connectors so that
Expand Down Expand Up @@ -41,7 +40,7 @@ of resources offered, through a contract offer, the so-called "catalog".

The following [request](resources/create-asset.json) creates an asset on the provider connector.

```bash
```shell
curl -d @transfer/transfer-01-negotiation/resources/create-asset.json \
-H 'content-type: application/json' http://localhost:19193/management/v2/assets \
-s | jq
Expand All @@ -65,7 +64,7 @@ to keep things simple, we will choose a policy that gives direct access to all t
associated within the contract definitions.
This means that the consumer connector can request any asset from the provider connector.

```bash
```shell
curl -d @transfer/transfer-01-negotiation/resources/create-policy.json \
-H 'content-type: application/json' http://localhost:19193/management/v2/policydefinitions \
-s | jq
Expand All @@ -78,21 +77,18 @@ the asset, on the basis of which a contract agreement can be negotiated. The con
associates policies to a selection of assets to generate the contract offers that will be put in the
catalog. In this case, the selection is empty, so every asset is attached to these policies.

```bash
```shell
curl -d @transfer/transfer-01-negotiation/resources/create-contract-definition.json \
-H 'content-type: application/json' http://localhost:19193/management/v2/contractdefinitions \
-s | jq

```

Sample output:
Part of sample output:

```json
{
...
"@id": "1",
"edc:createdAt": 1674578184023,
...
"edc:createdAt": 1674578184023
}
```

Expand All @@ -103,7 +99,7 @@ all the contract offers available for negotiation. In our case, it will contain
offer, the so-called "catalog". To get the catalog from the consumer side, you can use the following
request:

```bash
```shell
curl -X POST "http://localhost:29193/management/v2/catalog/request" \
-H 'Content-Type: application/json' \
-d @transfer/transfer-01-negotiation/resources/fetch-catalog.json -s | jq
Expand Down Expand Up @@ -181,20 +177,18 @@ looks as follows:
Of course, this is the simplest possible negotiation sequence. Later on, both connectors can also
send counteroffers in addition to just confirming or declining an offer.

```bash
```shell
curl -d @transfer/transfer-01-negotiation/resources/negotiate-contract.json \
-X POST -H 'content-type: application/json' http://localhost:29193/management/v2/contractnegotiations \
-s | jq
```

Sample output:
Part of sample output:

```json
{
...
"@id": "254015f3-5f1e-4a59-9ad9-bf0e42d4819e",
"edc:createdAt": 1685525281848,
...
"edc:createdAt": 1685525281848
}
```

Expand All @@ -207,7 +201,7 @@ state machine. Once both provider and consumer either reach the `confirmed` or t
state, the negotiation is finished. We can now use the UUID to check the current status of the
negotiation using an endpoint on the consumer side.

```bash
```shell
curl -X GET "http://localhost:29193/management/v2/contractnegotiations/<contract negotiation id, returned by the negotiation call>" \
--header 'Content-Type: application/json' \
-s | jq
Expand All @@ -217,21 +211,21 @@ Sample output:

```json
{
"@type": "edc:ContractNegotiationDto",
"@id": "5ca21b82-075b-4682-add8-c26c9a2ced67",
"edc:type": "CONSUMER",
"edc:protocol": "dataspace-protocol-http",
"edc:state": "FINALIZED",
"edc:counterPartyAddress": "http://localhost:19194/protocol",
"edc:callbackAddresses": [],
"edc:contractAgreementId": "0b3150be-feaf-43bc-91e1-90f050de28bd", <---------
"@context": {
"dct": "https://purl.org/dc/terms/",
"edc": "https://w3id.org/edc/v0.0.1/ns/",
"dcat": "https://www.w3.org/ns/dcat/",
"odrl": "http://www.w3.org/ns/odrl/2/",
"dspace": "https://w3id.org/dspace/v0.8/"
}
"@type": "edc:ContractNegotiationDto",
"@id": "5ca21b82-075b-4682-add8-c26c9a2ced67",
"edc:type": "CONSUMER",
"edc:protocol": "dataspace-protocol-http",
"edc:state": "FINALIZED",
"edc:counterPartyAddress": "http://localhost:19194/protocol",
"edc:callbackAddresses": [],
"edc:contractAgreementId": "0b3150be-feaf-43bc-91e1-90f050de28bd",
"@context": {
"dct": "https://purl.org/dc/terms/",
"edc": "https://w3id.org/edc/v0.0.1/ns/",
"dcat": "https://www.w3.org/ns/dcat/",
"odrl": "http://www.w3.org/ns/odrl/2/",
"dspace": "https://w3id.org/dspace/v0.8/"
}
}
```
At this point contract negotiation has been successfully completed.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@
"providerId": "provider",
"protocol": "dataspace-protocol-http",
"offer": {
"offerId": "MQ==:YXNzZXRJZA==:YTc4OGEwYjMtODRlZi00NWYwLTgwOWQtMGZjZTMwMGM3Y2Ey",
"offerId": "{{offerId}}",
"assetId": "assetId",
"policy": {
"@id": "MQ==:YXNzZXRJZA==:YTc4OGEwYjMtODRlZi00NWYwLTgwOWQtMGZjZTMwMGM3Y2Ey",
"@id": "{{offerId}}",
"@type": "Set",
"odrl:permission": [],
"odrl:prohibition": [],
Expand Down
36 changes: 14 additions & 22 deletions transfer/transfer-02-consumer-pull/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ order.
As a pre-requisite, you need to have a logging webserver that runs on port 4000 and logs all the incoming requests, it will
be mandatory to get the EndpointDataReference that will be used to get the data.

```bash
```shell
docker build -t http-request-logger util/http-request-logger
docker run -p 4000:4000 http-request-logger
```
Expand All @@ -39,12 +39,11 @@ In the [request body](resources/start-transfer.json), we need to specify which a
provider connector and where we want the file transferred.
Before executing the request, insert the `contractAgreementId` from the previous chapter. Then run:

```bash
```shell
curl -X POST "http://localhost:29193/management/v2/transferprocesses" \
-H "Content-Type: application/json" \
-d @transfer/transfer-02-consumer-pull/resources/start-transfer.json \
-s | jq

```

> the "HttpProxy" method is used for the consumer pull method, and it means that it will be up to
Expand All @@ -56,14 +55,12 @@ process id) created on the consumer
side, because like the contract negotiation, the data transfer is handled in a state machine and
performed asynchronously.

Sample output:
Part of sample output:

```json
{
...
"@id": "591bb609-1edb-4a6b-babe-50f1eca3e1e9",
"edc:createdAt": 1674078357807,
...
"edc:createdAt": 1674078357807
}
```

Expand All @@ -72,20 +69,17 @@ Sample output:
Due to the nature of the transfer, it will be very fast and most likely already done by the time you
read the UUID.

```bash
curl http://localhost:29193/management/v2/transferprocesses/<transfer process id>
```shell
curl http://localhost:29193/management/v2/transferprocesses/{{transfer-process-id}} -s | jq
```

You should see the Transfer Process in `STARTED` state:
You should see the Transfer Process in `STARTED` state:

```json
{
...
"@id": "591bb609-1edb-4a6b-babe-50f1eca3e1e9",
"edc:state": "STARTED",
...
"edc:state": "STARTED"
}

```

> Note that for the consumer pull scenario the TP will stay in STARTED state after the data has been transferred successfully.
Expand All @@ -101,28 +95,26 @@ to get the data from the provider:
"id": "591bb609-1edb-4a6b-babe-50f1eca3e1e9",
"endpoint": "http://localhost:29291/public/",
"authKey": "Authorization",
"authCode": "eyJhbGciOiJSUzI1NiJ9.eyJkYWQiOiJ7XCJwcm9wZXJ0aWVzXCI6e1wiYXV0aEtleVwiOlwiQXV0aG9yaXphdGlvblwiLFwiYmFzZVVybFwiOlwiaHR0cDpcL1wvbG9jYWxob3N0OjE5MjkxXC9wdWJsaWNcL1wiLFwiYXV0aENvZGVcIjpcImV5SmhiR2NpT2lKU1V6STFOaUo5LmV5SmtZV1FpT2lKN1hDSndjbTl3WlhKMGFXVnpYQ0k2ZTF3aVltRnpaVlZ5YkZ3aU9sd2lhSFIwY0hNNlhDOWNMMnB6YjI1d2JHRmpaV2h2YkdSbGNpNTBlWEJwWTI5a1pTNWpiMjFjTDNWelpYSnpYQ0lzWENKdVlXMWxYQ0k2WENKVVpYTjBJR0Z6YzJWMFhDSXNYQ0owZVhCbFhDSTZYQ0pJZEhSd1JHRjBZVndpZlgwaUxDSmxlSEFpT2pFMk56UTFPRGcwTWprc0ltTnBaQ0k2SWpFNk1XVTBOemc1TldZdE9UQXlOUzAwT1dVeExUazNNV1F0WldJNE5qVmpNemhrTlRRd0luMC5ITFJ6SFBkT2IxTVdWeWdYZi15a0NEMHZkU3NwUXlMclFOelFZckw5eU1tQjBzQThwMHFGYWV0ZjBYZHNHMG1HOFFNNUl5NlFtNVU3QnJFOUwxSE5UMktoaHFJZ1U2d3JuMVhGVUhtOERyb2dSemxuUkRlTU9ZMXowcDB6T2MwNGNDeFJWOEZoemo4UnVRVXVFODYwUzhqbU4wZk5sZHZWNlFpUVFYdy00QmRTQjNGYWJ1TmFUcFh6bDU1QV9SR2hNUGphS2w3RGsycXpJZ0ozMkhIdGIyQzhhZGJCY1pmRk12aEM2anZ2U1FieTRlZXU0OU1hclEydElJVmFRS1B4ajhYVnI3ZFFkYV95MUE4anNpekNjeWxyU3ljRklYRUV3eHh6Rm5XWmczV2htSUxPUFJmTzhna2RtemlnaXRlRjVEcmhnNjZJZzJPR0Eza2dBTUxtc3dcIixcInByb3h5TWV0aG9kXCI6XCJ0cnVlXCIsXCJwcm94eVF1ZXJ5UGFyYW1zXCI6XCJ0cnVlXCIsXCJwcm94eUJvZHlcIjpcInRydWVcIixcInR5cGVcIjpcIkh0dHBEYXRhXCIsXCJwcm94eVBhdGhcIjpcInRydWVcIn19IiwiZXhwIjoxNjc0NTg4NDI5LCJjaWQiOiIxOjFlNDc4OTVmLTkwMjUtNDllMS05NzFkLWViODY1YzM4ZDU0MCJ9.WhbTzERmM75mNMUG2Sh-8ZW6uDQCus_5uJPvGjAX16Ucc-2rDcOhAxrHjR_AAV4zWjKBHxQhYk2o9jD-9OiYb8Urv8vN4WtYFhxJ09A0V2c6lB1ouuPyCA_qKqJEWryTbturht4vf7W72P37ERo_HwlObOuJMq9CS4swA0GBqWupZHAnF-uPIQckaS9vLybJ-gqEhGxSnY4QAZ9-iwSUhkrH8zY2GCDkzAWIPmvtvRhAs9NqVkoUswG-ez1SUw5bKF0hn2OXv_KhfR8VsKKYUbKDQf5Wagk7rumlYbXMPNAEEagI4R0xiwKWVTfwwZPy_pYnHE7b4GQECz3NjhgdIw",
"authCode": "<auth-code>",
"properties": {
"cid": "1:1e47895f-9025-49e1-971d-eb865c38d540"
}
}
```

Once this json is read, use a tool like postman or curl to execute the following query, to read the
data
Once this json is read, use a tool like postman or curl to execute the following query, to read the data

```bash
curl --location --request GET 'http://localhost:29291/public/' --header 'Authorization: <auth code>'
```shell
curl --location --request GET 'http://localhost:29291/public/' --header 'Authorization: {{auth-code}}' -s | jq
```

At the end, and to be sure that you correctly achieved the pull, you can check if the data you get
is the same as the one you can get at https://jsonplaceholder.typicode.com/users


Since we configured the `HttpData` with `proxyPath`, we could also ask for a specific user with:

```bash
curl --location --request GET 'http://localhost:29291/public/1' --header 'Authorization: <auth code>'
```shell
curl --location --request GET 'http://localhost:29291/public/1' --header 'Authorization: {{auth-code}}' -s | jq
```

And the data returned will be the same as in https://jsonplaceholder.typicode.com/users/1
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"@type": "TransferRequestDto",
"connectorId": "provider",
"connectorAddress": "http://localhost:19194/protocol",
"contractId": "<contract agreement id>",
"contractId": "{{contract-agreement-id}}",
"assetId": "assetId",
"protocol": "dataspace-protocol-http",
"dataDestination": {
Expand Down
28 changes: 13 additions & 15 deletions transfer/transfer-03-provider-push/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,47 +16,45 @@ This samples consists of:

The following steps assume your provider and consumer connectors are still up and running and contract
negotiation has taken place successfully. Furthermore, the http server should be up as well.
If not, re-visit the [Prerequisites](../transfer-00-prerequisites/README.md)
, [Negotiation](../transfer-01-negotiation/README.md) and [Consumer Pull](../transfer-02-consumer-pull/README.md) chapters.
If not, re-visit
the [Prerequisites](../transfer-00-prerequisites/README.md), [Negotiation](../transfer-01-negotiation/README.md)
and [Consumer Pull](../transfer-02-consumer-pull/README.md) chapters.

# Run the sample

Running this sample consists of multiple steps, that are executed one by one and following the same
order.
Running this sample consists of multiple steps, that are executed one by one and following the same order.

### 1. Start the transfer

Before executing the request, modify the [request body](resources/start-transfer.json) by inserting the contract agreement ID
from the [Negotiation](../transfer-01-negotiation/README.md) chapter.
Before executing the request, modify the [request body](resources/start-transfer.json) by inserting the contract
agreement ID from the [Negotiation](../transfer-01-negotiation/README.md) chapter.
You can re-use the same asset, policies and contract negotiation from before.

```bash
```shell
curl -X POST "http://localhost:29193/management/v2/transferprocesses" \
-H "Content-Type: application/json" \
-d @transfer/transfer-03-provider-push/resources/start-transfer.json \
-s | jq
```

> keep in mind that, to make a transfer with a provider push method, the dataDestination type should
> be any value different from the "HttpProxy".

Sample output:
Part of sample output:

```json
{
...
"@id": "591bb609-1edb-4a6b-babe-50f1eca3e1e9",
"edc:createdAt": 1674078357807,
...
"edc:createdAt": 1674078357807
}
```

### 2. Check the transfer status

Due to the nature of the transfer, it will be very fast and most likely already done by the time you
read the UUID.
Due to the nature of the transfer, it will be very fast and most likely already done by the time you read the UUID.

```bash
curl http://localhost:29193/management/v2/transferprocesses/<transfer process id>
```shell
curl http://localhost:29193/management/v2/transferprocesses/{{transfer-process-id}} -s | jq
```

Notice the transfer COMPLETED state
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"@type": "TransferRequestDto",
"connectorId": "provider",
"connectorAddress": "http://localhost:19194/protocol",
"contractId": "<contract agreement id>",
"contractId": "{{contract-agreement-id}}",
"assetId": "assetId",
"protocol": "dataspace-protocol-http",
"dataDestination": {
Expand Down
Loading