ddtrace/tracer: add integration tag to spans_started/finished

[hannahkm]

What does this PR do?

Add a integration tag to the existing datadog.tracer.spans_started and datadog.tracer.spans_finished metrics. The value of the tag will be the name of the component from which the span was started. For example, for a contrib, it will be the name of the contrib package (chi, net/http, etc). For spans that were created manually, the tag will say manual.

Motivation

We want to know, in addition to when a span is started, where the span originated from. This could be a contrib or a manual implementation.

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.
• For internal contributors, a matching PR should be created to the v2-dev branch and reviewed by @DataDog/apm-go.

Unsure? Have a question? Request a review!

[datadog-datadog-prod-us1]

Datadog Report

Branch report: apm-rd/span-source-health-metric
Commit report: ddbc079
Test service: dd-trace-go

✅ 0 Failed, 5199 Passed, 72 Skipped, 2m 55.39s Total Time

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-01-17 16:08:31

Comparing candidate commit 6899d90 in PR branch apm-rd/span-source-health-metric with baseline commit 382c311 in branch main.

Found 1 performance improvements and 5 performance regressions! Performance is the same for 53 metrics, 0 unstable metrics.

scenario:BenchmarkInjectW3C-24

• 🟩 execution_time [-133.484ns; -112.916ns] or [-3.243%; -2.743%]

scenario:BenchmarkSetTagStringPtr-24

• 🟥 execution_time [+3.976ns; +6.764ns] or [+2.325%; +3.956%]

scenario:BenchmarkSingleSpanRetention/no-rules-24

• 🟥 execution_time [+8.931µs; +9.840µs] or [+3.811%; +4.199%]

scenario:BenchmarkSingleSpanRetention/with-rules/match-all-24

• 🟥 execution_time [+8.712µs; +9.619µs] or [+3.684%; +4.068%]

scenario:BenchmarkSingleSpanRetention/with-rules/match-half-24

• 🟥 execution_time [+8.676µs; +11.137µs] or [+3.668%; +4.709%]

scenario:BenchmarkTracerAddSpans-24

• 🟥 execution_time [+170.625ns; +190.975ns] or [+4.435%; +4.964%]

[mtoffl01]

Ok, so you're reporting spansStarted/spansFinished on span.Start/span.Finished if the integration is not empty, and leaving the chunk reporting to any spans that are manual... I understand why you did this but not totally sure about the approach.

span.Start and span.Stop are typically called quite frequently, so if a majority of the spans are from automatic integrations, this will be very noisy (and defeats the purpose of reporting the metrics at a specified interval, to reduce noise)

One alternative idea:
Change the way we track spansStarted and spansFinished to be some kind of counter map that includes the integration name, e.g. map[string]uint32 where the key is the integration name and the value is the count of spans started/finished that integration name. Then, in this goroutine, we'll have to iterate over the map and report the spans started/finished per integration
(or some other idea I haven't thought of?)

[hannahkm]

@mtoffl01 Good points! A map would probably work better; I was hesitant at first since I didn't want to change too much of what already exists, but knowing that these metrics are pretty old... I'm more down to change it up now.

[darccio]

@hannahkm I'm approving this but we should investigate why the benchmarks report the increased allocations.

[mtoffl01]

Overall, I definitely have some concerns 🤔 Maybe you can write some additional tests to provide peace of mind....

1. Tests designed to try and make the system fail -- what happens when you have multiple goroutines access a start span / finish span method, can we prove that we've protected against a race condition?
2. Maybe you want to write dedicated benchmarks to show how much performance is impacted

[mtoffl01]

It lgtm, BUT I would recommend we have more complex tests for SpansStarted and SpansFinished; like, generating multiple spans of different integrations and checking the metrics reported (rather than just 1 span).