Consider making OpenTelemetry Bridge implementation agnostic

[Exidex]

Currently micrometer-tracing-bridge-otel depends on OpenTelemetry API as well as its implementation OpenTelemetry SKD. Also currently Spring Boot has Autoconfiguration of OpenTelemetry SDK when it is present.

There are cases where you want to use OpenTelemetry API with different implementation, in our case, one provided by DataDog Java Agent. Having multiple implementations on classpath is problematic for various reasons. The only 2 ways to disable auto configuration is by using spring.autoconfigure.exclude (which can't be used if you define configuration in library) or by removing OpenTelemetry SDK dependency completely. Thankfully second way currently works but feels hacky and is prone to breakage.

implementation("io.micrometer:micrometer-tracing-bridge-otel") {
    exclude(group = "io.opentelemetry", module = "opentelemetry-sdk-common")
    exclude(group = "io.opentelemetry", module = "opentelemetry-sdk")
    exclude(group = "io.opentelemetry", module = "opentelemetry-sdk-trace")
}

Therefore my request is to consider splitting SDK related code into a separate package

[jonatan-ivanov]

Doesn't the Datadog Agent use the OTel SDK too? IIRC, there is no separate OTel SDK by Datadog, the agent uses the same (shadowed? or already available) SDK. Can you confirm this?

What happens when you want to use the OTel API+SDK (no Spring, no Micrometer Tracing) in your codebase (compile time + runtime), will the Datadog agent bring in another version of the OTel SDK? I think it should not.

If my assumptions are correct (the Datadog agent uses the same OTel SDK), I think you might have a split-brain problem introduced by the Datadog agent and moving SDK-related code out of the OTel bridge or disabling auto-configuration won't solve it. Here’s what I think is happening:
The datadog agent creates a global OTel Tracer instance and it uses that to create spans. I think this global instance is also available to the OTel API (you need to use it in the "global-way" for this though).
Using globals are not very fortunate in Java so Spring Boot avoids them as much as it can. It creates a non-global instance of the OTel Tracer and it passes that to Micrometer Tracing.
So out of the box when Datadog creates a span (using the global Tracer instance) that will not be available for Micrometer Tracing/Spring (using a non-global Tracer) and vice-versa. That's where you have the split brain: you have two Tracers. You can verify this with a debugger, check which Tracer is used when you use the OTel API and which Tracer is used when you use the Observation API/Micrometer Tracing/Spring (you should find it in DefaultTracingObservationHandler).

How to fix it:
If you check Spring Boot’s auto-configuration, it lets you define your own OTel Tracer. Theoretically, if you could tell the agent which instance it should use, and pass the instance that Spring Boot creates, the problem should go away. Unfortunately, as far as I know, this is not possible today simply because the agent is not able to accept a Tracer from the user. (This feature was requested by users multiple times but it was not implemented).
You can try to do the reverse of this: get the Tracer instance from the agent and ask spring to use it (you need to create a @Bean from it). I'm not sure how to do this and if the agent has a getter for the Tracer. In addition, you might also want to redo all the configuration that Spring Boot does for the Tracer (moving target).
I think the simplest might be getting rid of the agent and use the Tracer instance that Spring Boot crates for you through Micrometer Tracing and/or the Observation API.

Does this help?

[Exidex]

Doesn't the Datadog Agent use the OTel SDK too?

Doesn't seem to be the case. There is only one usage of io.opentelemetry.sdk in code and it is not relevant. See https://github.com/DataDog/dd-trace-java/tree/master/dd-java-agent/agent-otel/otel-shim/src/main/java/datadog/opentelemetry/shim/trace for tracer implementaition

About the rest, that is a lot and will take me some time to unpack. I will report back if there is anything relevant for my use case

[jonatan-ivanov]

Doesn't seem to be the case. There is only one usage of io.opentelemetry.sdk in code and it is not relevant.

You might be looking at the wrong thing, the OTel SDK should be used through the OTel API, try to search for io.opentelemetry.api instead, I think it's unlikely that Datadog reimplemented everything but I did not look deeper into this.

[Exidex]

the OTel SDK should be used through the OTel API

This doesn't make sense, of course API will be used because they implement it

I think it's unlikely that Datadog reimplemented everything

Why? datadog existed way before opentelemetry, they just need a wrapper around their api and this is what I linked in the previous comment