DataSink misunderstanding?

[klumbe]

Hi,

when reading the official documentation,
I was under the impression, that a DataSink could be used to store data received by the consumers data plane.
When diving deeper into it, in particular when dealing with the transfer requests, it rather looks like a DataSink is used on the provider end to transfer the data to a specific target.
Or maybe both cases are possible?

Can someone point me into the right direction, possibly linking documentation that makes this a bit clearer?

In particular, I am interested in transferring (http/s) the data solely through the data planes on both the providers and the consumers end (data plane to data plane).
How could I achieve that the data-plane then stores it at the desired location?

I am particularly NOT interested in getting any final solution, I just want to know where my extension should be seated.

[ndr-brt]

Hi-level docs about data plane:
https://eclipse-edc.github.io/documentation/for-adopters/data-plane/

In depth documentation:
https://eclipse-edc.github.io/documentation/for-contributors/data-plane/

If there are things that are not clear or you feel like they could described better, feel free to open an issue on the documentation repository

[klumbe]

Hi @jimmarino,

my reference to the sink was meant to be on the consumer end, in particular in the push-scenario.
I thought it would be a good idea to have a reference to the sink the TransferRequest such that when it is created, the consumer would know where to pass the data, regardless of the chosen transfer type.

But I've switched to your suggestion to use a custom public HTTP API to receive the pushed data.
So it is an extension registering a Jersey controller offering a POST request on a dedicated port.

What you say sounds contrary to the statement in the documentation you have linked:

When the control plane signals to the data plane to start a client pull transfer process, the data plane returns a DataAddress. This is only true for consumer-pull transfers - provider push transfers do not return a DataAddress

So my question would be:
If that DataAddress and the token are generated in the consumer control plane in the PUSH-scenario, how could I access it from the custom API?

My previous question around selecting the proper data plane - which may have caused the confusion - is related to the selection process in the consumers control plane: The doc states that the DataFlowStartMessage will be send to the appropriate data plane.
So I am wondering how the control plane could do this decision if the request itself contains just the baseURL of the dataDestination.
In my case, I will keep it as simple as possible and will just have a single data plane where the public API will be running in.
But theoretically, that custom API could be anything, like a dedicated web-server and in such a scenario, I would have to exchange the token, i.e. through a database.

Example of a TransferRequest sent to the consumer:

{
    "@context": {
        "@vocab": "https://w3id.org/edc/v0.0.1/ns/"
    },
    "@type": "TransferRequest",
    "dataDestination": {
        "type": "HttpData",
        "baseUrl": "http://example.org:8080/public/custom/store/<id>}"
    },
    "protocol": "dataspace-protocol-http",
    "contractId": "...",
    "connectorId": "<provider-id>",
    "counterPartyAddress":  "<provider-dsp-url>/api/dsp",
    "transferType": "HttpData-PUSH"
}

For my scenario, the only thing of importance is that a token is generated, sent to the data provider and the token can be retrieved in the custom API to validate its correctness.
Since the doc actually states, that it is not generated in the PUSH scenario, I would guess that I would have to subscribe to state-changes in the consumers control plane and either generate or destroy a token. In the generate case, it should be part of DSP request forwarded to the provider and also in a DataFlowStartMessage shared with the data plane.
If I have to do this manually, I would need to know how to influence the DSP request and which particular fields are used.

Again, many thanks for your answer. I hope I have made it a bit more clear what I am actually missing.
Let me know if you need any further input.

[jimmarino]

Please look at how the TransferProcessManager and Provisioner (particularly the provision method) work on the consumer side where a token can be created, which is then passed to the provider in the DataAddress that is part of the transfer process request.

[klumbe]

@jimmarino: Thanks for that input. I will check these and will give some feedback as soon as I went through it.

Earlier today, I did some tcpdump and checked the messages flowing through DSP. The dataAddress was basically a copy of the dataDestination field in the TransferRequest, not containing any token.
In the [DataAddress]{https://github.com/eclipse-edc/Connector/blob/main/spi/common/core-spi/src/main/java/org/eclipse/edc/spi/types/domain/DataAddress.java} class, there is a field called secret. Would it be the one to populate for the token?

[klumbe]

I guess that the missing else branch in the HttpProvisionerExtension would be what I am actually searching for:

            if (configuration.getProvisionerType() == ProvisionerConfiguration.ProvisionerType.PROVIDER) {
                var generator = new HttpProviderResourceDefinitionGenerator(configuration.getDataAddressType());
                manifestGenerator.registerGenerator(generator);
                monitor.debug(format("Registering provider provisioner: %s [%s]", configuration.getName(), configuration.getEndpoint().toString()));
            } else {
                monitor.warning(format("Client-side provisioning not yet supported by the %s. Skipping configuration for %s", name(), configuration.getName()));
            }

The class AbstractHttpResourceDefinition gives the hint that the dataAddressType is used to determine the correct Provisioner (not too surprising).
Would this mean that I have to define a custom dataAddressType and build an Extension like the HttpProvisionerExtension to be able to fire the custom Provisioner?

Is it related to the dataDestination.type in the TransferRequest?
I am wondering if I could still use the provided HttpData-PUSH together with a custom type such that it would be clear that this processing is only required on the consumer end and only dataDestination is really that custom API.

[klumbe]

Hi @jimmarino,

as far as I understood, the HttpProvisionerExtension is handling provisioning requests against a HTTP callback URL, so that is not necessarily required here and I should build a custom Provisioner.

I tried that and on debugging, I found out that the process.getResourcesToProvision() method returns an empty list.
I couldn't find out all the details about how the Provisioner actually works, but I guess I have to define the resources requiring provisioning somewhere.

Scanning the code, this may be related to the provision.manifest.verify briefly mentioned as a policy scope in the old Developer's Handbook.
Does it mean a dedicated policy definition is required for running the provisioning, or is it just for evaluation during the provisioning itself?
Can't find it in the contract definition api, so I guess this is just outdated information.

Could someone give me an example how it should look like or point me to where I can find documentation on how to use it properly?

[jimmarino]

A DataAddress is just a pointer to data (i.e. an endpoint where it can be retrieved). There are internal data addresses associated with assets the connector uses to retrieve and send data associated with an asset. There are also external DataAddresses that can be sent to a counter-party to push or pull data.

The HTTP Provisioner provides an example of how to write a custom provisioner. The code is also well-documented and explains the relationship between provisioners and resource generators. You can also take a look at S3BucketProvisioner to see how to generate and send access tokens.