Concurrent DownstreamApi calls fail after a while with: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.

[Rans4ckeR]

Microsoft.Identity.Web Library

Microsoft.Identity.Web.DownstreamApi

Microsoft.Identity.Web version

3.6.2

Web app

Not Applicable

Web API

Not Applicable

Token cache serialization

In-memory caches

Description

We are using a .NET 9 Microsoft.NET.Sdk.Worker daemon application that continuously and concurrently calls IDownstreamApi.PostForAppAsync() to process individual units of work. This is a high throughput real-time processing engine that uses 30 concurrent threads all using the same IDownstreamApi instance.
This works very well with all Microsoft.Identity.Web versions so far, until updating to any 3.6.x version.
It can take a few hours but eventually every call to the IDownstreamApi will fail until we restart the daemon application.

Could this be related to #3202?

Reproduction steps

1. Spin up concurrent Tasks that will call the IDownstreamApi in an infinite loop
2. Let it run for some hours

Error message

Exception.GetType: System.InvalidOperationException
Exception.Message: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
Exception.Source: System.Private.CoreLib
Exception.TargetSite: Void ThrowInvalidOperationException_ConcurrentOperationsNotSupported()
Exception.StackTrace: 
at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior) 

at Microsoft.Identity.Client.Internal.Requests.AuthenticationRequestParameters..ctor(IServiceBundle serviceBundle, ITokenCacheInternal tokenCache, AcquireTokenCommonParameters commonParameters, RequestContext requestContext, Authority initialAuthority, String homeAccountId) 
at Microsoft.Identity.Client.ApplicationBase.CreateRequestParametersAsync(AcquireTokenCommonParameters commonParameters, RequestContext requestContext, ITokenCacheInternal cache) 
at Microsoft.Identity.Client.ConfidentialClientApplication.CreateRequestParametersAsync(AcquireTokenCommonParameters commonParameters, RequestContext requestContext, ITokenCacheInternal cache) 
at Microsoft.Identity.Client.ApiConfig.Executors.ConfidentialClientExecutor.ExecuteAsync(AcquireTokenCommonParameters commonParameters, AcquireTokenForClientParameters clientParameters, CancellationToken cancellationToken) 

at Microsoft.Identity.Web.TokenAcquisition.GetAuthenticationResultForAppAsync(String scope, String authenticationScheme, String tenant, TokenAcquisitionOptions tokenAcquisitionOptions) 
at Microsoft.Identity.Web.DefaultAuthorizationHeaderProvider.CreateAuthorizationHeaderAsync(IEnumerable`1 scopes, AuthorizationHeaderProviderOptions downstreamApiOptions, ClaimsPrincipal claimsPrincipal, CancellationToken cancellationToken) 
at Microsoft.Identity.Web.DownstreamApi.UpdateRequestAsync(HttpRequestMessage httpRequestMessage, HttpContent content, DownstreamApiOptions effectiveOptions, Boolean appToken, ClaimsPrincipal user, CancellationToken cancellationToken) 
at Microsoft.Identity.Web.DownstreamApi.CallApiInternalAsync(String serviceName, DownstreamApiOptions effectiveOptions, Boolean appToken, HttpContent content, ClaimsPrincipal user, CancellationToken cancellationToken) 
at Microsoft.Identity.Web.DownstreamApi.PostForAppAsync[TInput,TOutput](String serviceName, TInput input, Action`1 downstreamApiOptionsOverride, CancellationToken cancellationToken)

Id Web logs

No response

Relevant code snippets

using System.Net;
using Microsoft.Identity.Abstractions;
using Microsoft.Identity.Web;
using Microsoft.Identity.Web.TokenCacheProviders.InMemory;

HostApplicationBuilder builder = Host.CreateApplicationBuilder(args);

_ = builder.Services
    .AddWindowsService()
    .AddHostedService<XXXBackgroundService>()
    .AddTokenAcquisition(true)
    .Configure<MicrosoftIdentityApplicationOptions>(builder.Configuration.GetRequiredSection("AzureAd"))
    .AddInMemoryTokenCaches()
    .ConfigureHttpClientDefaults(static httpClientBuilder => httpClientBuilder
        .ConfigureHttpClient(static httpClient => httpClient.DefaultVersionPolicy = HttpVersionPolicy.RequestVersionOrHigher)
        .ConfigurePrimaryHttpMessageHandler(static _ => new SocketsHttpHandler
        {
            AutomaticDecompression = DecompressionMethods.All,
            PooledConnectionLifetime = TimeSpan.FromMinutes(15),
            PreAuthenticate = true
        })
        .SetHandlerLifetime(Timeout.InfiniteTimeSpan))
    .AddDownstreamApi(nameof(DownstreamApiOptions), builder.Configuration.GetRequiredSection(nameof(DownstreamApiOptions)));

await builder.Build().RunAsync(CancellationToken.None).ConfigureAwait(ConfigureAwaitOptions.None);

  "DownstreamApiOptions": {
    "BaseUrl": "https://xxx/",
    "RequestAppToken": true,
    "Scopes": [ "api://xxx/.default" ]
  },
  "AzureAd": {
    "Instance": "https://login.microsoftonline.com/",
    "TenantId": "xxx",
    "ClientId": "xxx",
    "ClientCredentials": [
      {
        "SourceType": "StoreWithDistinguishedName",
        "CertificateStorePath": "LocalMachine/My",
        "CertificateDistinguishedName": "xxx"
      }
    ]
  }

using Microsoft.Identity.Abstractions;

namespace XXX;

internal sealed class XXX(IDownstreamApi downstreamApi) : IXXX
{
    private readonly IDownstreamApi downstreamApi = downstreamApi;

    public Task<IEnumerable<long>?> XXXAsync(CancellationToken cancellationToken)
        => downstreamApi.PostForAppAsync<int[], IEnumerable<long>>(nameof(DownstreamApiOptions), xxx, static options => options.RelativePath = "xxx", cancellationToken);
}

Regression

3.5.0

Expected behavior

Don't trigger ThrowInvalidOperationException_ConcurrentOperationsNotSupported() inside System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior).

[msbw2]

In the constructor for AuthenticationRequestParameters in MSAL a reference to ExtraQueryParameters on the config in the service bundle is kept and mutated: https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/blob/79de40fe135b1882f086bdc6c5958379638ef1cf/src/client/Microsoft.Identity.Client/Internal/Requests/AuthenticationRequestParameters.cs#L49-L59

This seems unintentional and this should probably instead create a copy of the dictionary to mutate.

Between 3.5.0 and 3.6.0 new query parameters were added in this commit (@bgavrilMS ): bb7c3e8

microsoft-identity-web/src/Microsoft.Identity.Web.DownstreamApi/DownstreamApi.cs

Lines 539 to 560 in d552f4a

	internal /* for test */ static Dictionary<string, string> CallerSDKDetails { get; } = new()
	{
	{ "caller-sdk-id", "IdWeb_1" },
	{ "caller-sdk-ver", IdHelper.GetIdWebVersion() }
	};
	private static void AddCallerSDKTelemetry(DownstreamApiOptions effectiveOptions)
	{
	if (effectiveOptions.AcquireTokenOptions.ExtraQueryParameters == null)
	{
	effectiveOptions.AcquireTokenOptions.ExtraQueryParameters = CallerSDKDetails;
	}
	else
	{
	effectiveOptions.AcquireTokenOptions.ExtraQueryParameters["caller-sdk-id"] =
	CallerSDKDetails["caller-sdk-id"];
	effectiveOptions.AcquireTokenOptions.ExtraQueryParameters["caller-sdk-ver"] =
	CallerSDKDetails["caller-sdk-ver"];
	}
	}
	}
	}

It looks like this may be propagating all the way down. Most likely before ExtraQueryParameters was null and thus the dictionary was not mutated, but with this recent change the mutation now takes place.

[bgavrilMS]

Thanks for the investigation @msbw2 and for the bug report @Rans4ckeR, looking into it.

[bgavrilMS]

I can repro this in Microsoft.Identity.Client by running in a tight loop.

[bgavrilMS]

We will fix this in MSAL AzureAD/microsoft-authentication-library-for-dotnet#5108

[bgavrilMS]

Fix is merged and will be available in the next MSAL release, 4.68.0, ETA Feb 10