chore(llmobs): automatically set span links with decorators

[lievan]

Re-opening #12043 due to conflicts with main after 3.x-staging was merged in

Decorators set span links by tracking common objects passed as inputs & outputs of functions.

This functionality will be gated behind the environment variable _DD_LLMOBS_AUTO_SPAN_LINKING_ENABLED

We maintain a dictionary in the LLMObs service that remembers which objects are used as the input/output for a spans generated by LLM Obs decorators. This is how we record the from direction of span links.

When objects are encountered again as the input/output for another span, we now know to set the to direction for a span link.

In my opinion, this does not need to be gated behind a feature flag since it's a read-only on app data. A follow up PR will mutate data actually used in the user app for enhanced span link inferencing, and the features introduced then should be gated by a flag.

Implementation notes:

• Objects are remembered by generating a string object id through the type + memory location of the object.
• We ignore "input" -> "output" edges. This is not a valid edge.
• Spans can only be linked to other spans belonging to the same trace

Limitations:

• it is very easy for link information to be lost if an object is mutated or used to create another object. A follow up PR will implement a best-effort attempt for objects to inherit link info from other objects
• doesn't work for distributed tracing scenarios

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

ddtrace/llmobs/_llmobs.py                                               @DataDog/ml-observability
ddtrace/llmobs/_utils.py                                                @DataDog/ml-observability
ddtrace/llmobs/decorators.py                                            @DataDog/ml-observability
ddtrace/settings/_config.py                                             @DataDog/apm-core-python
tests/llmobs/_utils.py                                                  @DataDog/ml-observability
tests/llmobs/test_llmobs_decorators.py                                  @DataDog/ml-observability
tests/telemetry/test_writer.py                                          @DataDog/apm-python
tests/utils.py                                                          @DataDog/python-guild

[datadog-dd-trace-py-rkomorn]

Datadog Report

Branch report: evan.li/span-link-again
Commit report: e677948
Test service: dd-trace-py

✅ 0 Failed, 869 Passed, 445 Skipped, 13m 0.5s Total duration (15m 53.38s time saved)

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-02-13 14:54:37

Comparing candidate commit e677948 in PR branch evan.li/span-link-again with baseline commit 47f3bb2 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 394 metrics, 2 unstable metrics.