How to handle a data transfer from/to ASW S3?

[lucian-torje-siemens]

Hello,
I have the following scenario:

• provider (provides a file at a given http endpoint) - can be any EDC implementation (it should have at minimum dataplane support e.g. http to read the asset)
• consumer (provides a destination location where to upload the provider's file; needs to be provisioned) - enhanced EDC code (support custom provisioning for the asset meaning it can generate presigned url for upload - in the future it may need to support multipart file upload/download to/from AWS)
• asset - source as http endpoint providing octet-stream data, destination AWS S3 location that needs to be provisioned

Problem - how to send the data to the presigned url for upload

Possible solutions and their

1. upload from provider - the question is how to create the endpoint url inside the consumer and give it to the provider before the provider checks it inside the http dataplane factory?
2. upload from consumer - the question is how to mark is as done so that the provider does not have to do anything ?
3. what if both provider and consumer need to provision (e.g. send file from Company X's AWS S3 to Company Y's AWS S3)

[jimmarino]

Either the consumer needs to furnish data destination address for the provider to write the data to, or the consumer must pull the data from an address controlled by the provider. It's not clear to me which scenario you want to enable.

[lucian-torje-siemens]

I've already investigated that part - please check if the way I've checked is correct:

• the provisioner on consumer part handles the presigned upload url
• if the type of the transfer is HttpData -> this does not work because this implementation uses POST and appends the part name to the endpoint (which in my case is an S3 upload presigned url)
• if the type is S3Bucket -> I do not have the AWS credentials to perform the operation
• if the type is custom -> it will do the trick on consumer side (has access to the source location of the data and has the knowledge to push the data to the destination) but the provider side will fail because it does not support my custom dataplane implementation
• is there a way to notify the provider that the consumer will handle the all flow and the provider does not need to do a thing, since the provider is calling the consumer anyway to do the transfer?

[jimmarino]

I'm having difficulty understanding this flow. Do you have sequence diagrams or something similar that outlines this?

[lucian-torje-siemens]

[lucian-torje-siemens]

Related question

What should happen in case the consumer and provider sign a contract and the asset/s are incompatible (cannot be transferred - like in my case custon type)?
Does a contract with untransferable asset/s bring any value?

(I would expect the contract negotiation to fail as to signal the incompatibility between provider and consumer for the specific asset.)

[lucian-torje-siemens]

Hi Jim,
Do you think that using HttpProxy type would help - if I got it right this would allow me to proxy the provider data to the consumer side and process it there (sink it to S3) ?