Phase one logger optimisations

[stevejgordon]

This PR dramatically reduces the overhead of scoped logging, even for scenarios where the configured log level verbosity was low (e.g., Warning). The most critical improvement is caching scoped loggers within Spans, Traces, and Errors. We lose convenience in no longer having the span ID in the scope, but it's generally available from the Span argument.

Previously, a new ScopedLogger was created for every span and transaction, and a new ConditionalWeakTable was also allocated. Due to the use of Finalisers on the CWT, this had an even more significant GC impact. We now maintain a cache of scoped loggers (of which there are relatively few).

The ID creation for the Spans has been switched to ActivitySpanId.CreateRandom().ToString() from the MS diagnostics library, as this has been more heavily optimised than our original version and achieves the same goal.

In Tracer.End(), we used an Assertion with a large interpolated string. This was being interpolated for each transaction as the Assertion path was enabled. However, the condition we assert is generally not met. Therefore, we now check this manually to only create the message if we are in an invalid state.

I've also switched two structs to record structs, as these were used as keys in dictionaries and ended up being boxed due to a lack of IEquality implementations.

These are phase one of optimisations for the logging implementation and address the general production scenario of using a less verbose log level.

Profiling

Profiling was achieved using a .NET 9 ASP.NET Core minimal API project with 20 concurrent connections and 100,000 requests.

Before Memory Profile

After Memory Profile

Before Top Retained

After Top Retained

Closes #2500

[gregkalapos]

Overall I totally agree with the direction. 👍

Below one thing about replacing the ?. notation to check log levels. Less relevant from the allocation's perspective, but I think it'd be still worth keeping that.

[ArminShoeibi]

There is a gotcha here, we should use lock like dotnet framework for AddOrUpdate

apm-agent-dotnet/src/Elastic.Apm/Logging/IApmLoggingExtensions.cs

Line 151 in 3386ba6

Formatters.AddOrUpdate(message, formatter);

for more details:

https://learn.microsoft.com/en-us/dotnet/api/system.collections.concurrent.concurrentdictionary-2.addorupdate?view=net-9.0

[stevejgordon]

@ArminShoeibi Formatters is ConditionalWeakTable<string, LogValuesFormatter>. I need to review that, but I think it's okay here.

[ArminShoeibi]

@ArminShoeibi Formatters is ConditionalWeakTable<string, LogValuesFormatter>. I need to review that, but I think it's okay here.

My bad, it has a lock itself

https://github.com/dotnet/runtime/blob/1d1bf92fcf43aa6981804dc53c5174445069c9e4/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/ConditionalWeakTable.cs#L126

[stevejgordon]

No worries. It's hard to keep track!

[gregkalapos]

Thanks for the updates.

lgtm