Possible parallel workload resolver use

[Forgind]

Product

dotnet CLI (dotnet new)

Describe The Bug

It seems like there may be some parallel calls to the same workload resolver after it's initialized but before any (serialized) access calls (version or manifests), and it means that we follow the lazy process for computing our manifests multiple times, leading to errors. This should not happen in the first place because the workload resolver should only ever be used from one thread.

One example of this was dotnet/installer#19222

A partial solution is dotnet/sdk#40485, but that doesn't get at the root cause of this instance. We currently believe that root cause is somewhere here.

To Reproduce

See dotnet/installer#19222

dotnet Info

output

Visual Studio Version

No response

Additional context

No response

[dsplaisted]

This started showing up a lot more in validation for .NET SDK 8.0.300. See dotnet/aspire#4145

I was able to debug it, and it looks like the issue is here:

templating/src/Microsoft.TemplateEngine.Edge/Settings/TemplatePackageManager.cs

Lines 351 to 363 in a78c4f5

	Parallel.For(0, allTemplatePackages.Count, async (int index) =>
	{
	try
	{
	var scanResult = await _installScanner.ScanAsync(allTemplatePackages[index].MountPointUri, cancellationToken).ConfigureAwait(false);
	scanResults[index] = scanResult;
	}
	catch (Exception ex)
	{
	_logger.LogWarning(LocalizableStrings.TemplatePackageManager_Error_FailedToScan, allTemplatePackages[index].MountPointUri, ex.Message);
	_logger.LogDebug($"Stack trace: {ex.StackTrace}");
	}
	});

That code is scanning each template package in a separate parallel task. That leads to each template being instantiated in parallel, and when there are workload installation constraints the constraints get created in parallel, thus getting the list of installed workloads in parallel, which results to parallel calls to the workload resolver, which is not thread-safe.

The workload resolver checks to see whether the manifests have been initialized, and if not, then it gets the manifests, adds them to the internal structures that hold them, and then sets the initialized flag to true. Here's that code: https://github.com/dotnet/sdk/blob/4d3cca6fa5437395a0bc984a809c1d5c12c18a26/src/Resolvers/Microsoft.NET.Sdk.WorkloadManifestReader/WorkloadResolver.cs#L87-L95

When multiple threads run this code at once, it's possible for both of them to see that the manifests haven't been initialized, and then for both threads to try to load the manifests, resulting in an error when one thread tries to add a manifest that has already been added by the other thread.

The workload resolver was never intended to be thread-safe, and writing thread-safe code can be difficult. So probably we need to find a way for the template engine to not call the workload resolver code in parallel.

[dsplaisted]

In case they help, here are some stack traces of the code that ends up accessing the workload resolver in parallel:

Microsoft.DotNet.TemplateLocator.dll!Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.LoadManifestsFromProvider(Microsoft.NET.Sdk.WorkloadManifestReader.IWorkloadManifestProvider manifestProvider) Line 196	C#
Microsoft.DotNet.TemplateLocator.dll!Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.InitializeManifests() Line 164	C#
Microsoft.DotNet.TemplateLocator.dll!Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.GetAvailableWorkloadDefinitions() Line 568	C#
System.Linq.dll!System.Linq.Enumerable.SelectEnumerableIterator<(Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadDefinition, Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadManifest), string>.MoveNext()	Unknown
System.Linq.dll!System.Linq.Enumerable.Contains<string>(System.Collections.Generic.IEnumerable<string> source, string value, System.Collections.Generic.IEqualityComparer<string> comparer)	Unknown
dotnet.dll!Microsoft.DotNet.Workloads.Workload.VisualStudioWorkloads.GetInstalledWorkloads(Microsoft.NET.Sdk.WorkloadManifestReader.IWorkloadResolver workloadResolver, Microsoft.DotNet.Workloads.Workload.List.InstalledWorkloadsCollection installedWorkloads, Microsoft.NET.Sdk.WorkloadManifestReader.SdkFeatureBand? sdkFeatureBand) Line 59	C#
dotnet.dll!Microsoft.DotNet.Workloads.Workload.List.WorkloadInfoHelper.AddInstalledVsWorkloads(System.Collections.Generic.IEnumerable<Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadId> sdkWorkloadIds) Line 66	C#
dotnet.dll!Microsoft.DotNet.Workloads.Workload.List.WorkloadInfoHelper.InstalledAndExtendedWorkloads.get() Line 38	C#
dotnet.dll!Microsoft.DotNet.Tools.New.WorkloadsInfoProvider.GetInstalledWorkloadsAsync(System.Threading.CancellationToken cancellationToken) Line 27	C#
Microsoft.TemplateEngine.Edge.dll!Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory.WorkloadConstraint.ExtractWorkloadInfoAsync(System.Collections.Generic.IEnumerable<Microsoft.TemplateEngine.Abstractions.Components.IWorkloadsInfoProvider> workloadsInfoProviders, Microsoft.Extensions.Logging.ILogger logger, System.Threading.CancellationToken token) Line 105	C#
Microsoft.TemplateEngine.Edge.dll!Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory.WorkloadConstraint.CreateAsync(Microsoft.TemplateEngine.Abstractions.IEngineEnvironmentSettings environmentSettings, Microsoft.TemplateEngine.Abstractions.Constraints.ITemplateConstraintFactory factory, System.Threading.CancellationToken cancellationToken) Line 52	C#
Microsoft.TemplateEngine.Edge.dll!Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory.Microsoft.TemplateEngine.Abstractions.Constraints.ITemplateConstraintFactory.CreateTemplateConstraintAsync(Microsoft.TemplateEngine.Abstractions.IEngineEnvironmentSettings environmentSettings, System.Threading.CancellationToken cancellationToken) Line 27	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.Task<Microsoft.TemplateEngine.Abstractions.Constraints.ITemplateConstraint>>.InnerInvoke() Line 209	C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread threadPoolThread, System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 192	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot, System.Threading.Thread threadPoolThread) Line 2194	C#
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 690	C#
System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 1328	C#

Microsoft.TemplateEngine.Edge.dll!Microsoft.TemplateEngine.Edge.TemplateConstraintManager.TemplateConstraintManager(Microsoft.TemplateEngine.Abstractions.IEngineEnvironmentSettings engineEnvironmentSettings) Line 32	C#
Microsoft.TemplateEngine.Cli.dll!Microsoft.TemplateEngine.Cli.TemplatePackageCoordinator.TemplatePackageCoordinator(Microsoft.TemplateEngine.Abstractions.IEngineEnvironmentSettings environmentSettings, Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager templatePackageManager)	Unknown
Microsoft.TemplateEngine.Cli.dll!Microsoft.TemplateEngine.Cli.Commands.TemplateCommand.InvokeAsync(System.CommandLine.ParseResult parseResult, System.Threading.CancellationToken cancellationToken)	Unknown
Microsoft.TemplateEngine.Cli.dll!Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.HandleTemplateInstantiationAsync(Microsoft.TemplateEngine.Cli.Commands.InstantiateCommandArgs args, Microsoft.TemplateEngine.Abstractions.IEngineEnvironmentSettings environmentSettings, Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager templatePackageManager, Microsoft.TemplateEngine.Cli.TemplateGroup templateGroup, System.Threading.CancellationToken cancellationToken)	Unknown
Microsoft.TemplateEngine.Cli.dll!Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.ExecuteIntAsync(Microsoft.TemplateEngine.Cli.Commands.InstantiateCommandArgs instantiateArgs, Microsoft.TemplateEngine.Abstractions.IEngineEnvironmentSettings environmentSettings, Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager templatePackageManager, System.CommandLine.ParseResult parseResult, System.Threading.CancellationToken cancellationToken)	Unknown
[Resuming Async Method]	
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<Microsoft.TemplateEngine.Cli.NewCommandStatus>.AsyncStateMachineBox<Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.<ExecuteIntAsync>d__17>.ExecutionContextCallback(object s) Line 131	C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 138	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<Microsoft.TemplateEngine.Cli.NewCommandStatus>.AsyncStateMachineBox<Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.<ExecuteIntAsync>d__17>.MoveNext(System.Threading.Thread threadPoolThread) Line 163	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<Microsoft.TemplateEngine.Cli.NewCommandStatus>.AsyncStateMachineBox<Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.<ExecuteIntAsync>d__17>.MoveNext() Line 146	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 140	C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 162	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2850	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 171	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.SetExistingTaskResult(System.Threading.Tasks.Task<System.__Canon> task, System.__Canon result) Line 389	C#
[Completed] Microsoft.TemplateEngine.Cli.dll!Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.GetTemplateGroupsAsync(Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager templatePackageManager, Microsoft.TemplateEngine.Cli.HostSpecificDataLoader hostSpecificDataLoader, System.Threading.CancellationToken cancellationToken)	Unknown
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.<GetTemplateGroupsAsync>d__11>.ExecutionContextCallback(object s) Line 131	C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 138	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Collections.Generic.IEnumerable<Microsoft.TemplateEngine.Cli.TemplateGroup>>.AsyncStateMachineBox<Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.<GetTemplateGroupsAsync>d__11>.MoveNext(System.Threading.Thread threadPoolThread) Line 163	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.<GetTemplateGroupsAsync>d__11>.MoveNext() Line 146	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 140	C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 162	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2850	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 171	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.SetExistingTaskResult(System.Threading.Tasks.Task<System.__Canon> task, System.__Canon result) Line 389	C#
[Completed] Microsoft.TemplateEngine.Edge.dll!Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.GetTemplatesAsync(System.Threading.CancellationToken cancellationToken) Line 163	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<GetTemplatesAsync>d__16>.ExecutionContextCallback(object s) Line 131	C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 138	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Collections.Generic.IReadOnlyList<Microsoft.TemplateEngine.Abstractions.ITemplateInfo>>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<GetTemplatesAsync>d__16>.MoveNext(System.Threading.Thread threadPoolThread) Line 163	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<GetTemplatesAsync>d__16>.MoveNext() Line 146	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 140	C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 162	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2850	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 171	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.SetExistingTaskResult(System.Threading.Tasks.Task<System.__Canon> task, System.__Canon result) Line 389	C#
[Completed] Microsoft.TemplateEngine.Edge.dll!Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.UpdateTemplateCacheAsync(bool needsRebuild, System.Threading.CancellationToken cancellationToken) Line 383	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<UpdateTemplateCacheAsync>d__23>.ExecutionContextCallback(object s) Line 131	C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 138	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<Microsoft.TemplateEngine.Edge.Settings.TemplateCache>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<UpdateTemplateCacheAsync>d__23>.MoveNext(System.Threading.Thread threadPoolThread) Line 163	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<UpdateTemplateCacheAsync>d__23>.MoveNext() Line 146	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 140	C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 162	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2850	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 171	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.SetExistingTaskResult(System.Threading.Tasks.Task<System.__Canon> task, System.__Canon result) Line 389	C#
[Completed] Microsoft.TemplateEngine.Edge.dll!Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.GetTemplatePackagesAsync(bool force, System.Threading.CancellationToken cancellationToken) Line 121	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<GetTemplatePackagesAsync>d__13>.ExecutionContextCallback(object s) Line 131	C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 138	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Collections.Generic.IReadOnlyList<Microsoft.TemplateEngine.Abstractions.TemplatePackage.ITemplatePackage>>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<GetTemplatePackagesAsync>d__13>.MoveNext(System.Threading.Thread threadPoolThread) Line 163	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<GetTemplatePackagesAsync>d__13>.MoveNext() Line 146	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 140	C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 162	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2850	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 171	C#
System.Private.CoreLib.dll!System.Threading.Tasks.UnwrapPromise<System.Collections.Generic.IReadOnlyList<Microsoft.TemplateEngine.Abstractions.TemplatePackage.ITemplatePackage>>.TrySetFromTask(System.Threading.Tasks.Task task, bool lookForOce) Line 122	C#
System.Private.CoreLib.dll!System.Threading.Tasks.UnwrapPromise<System.__Canon>.ProcessInnerTask(System.Threading.Tasks.Task task) Line 134	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2741	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishSlow(bool userDelegateExecute) Line 1971	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot, System.Threading.Thread threadPoolThread) Line 2205	C#
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 690	C#
System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 1328	C#
[Async Call Stack]	
[Async] Microsoft.TemplateEngine.Cli.dll!Microsoft.TemplateEngine.Cli.Commands.BaseCommand<Microsoft.TemplateEngine.Cli.Commands.NewCommandArgs>.CommandAction.InvokeAsync(System.CommandLine.ParseResult parseResult, System.Threading.CancellationToken cancellationToken)	Unknown

[dsplaisted]

Because it's buried in the linked issue, here's what the failure would actually look like when you tried to create a project using a template that used the workload constraint:

Failed to instatiate template '.NET Aspire Application', the following constraints are not met:
  workload: The constraint 'workload' failed to initialize: Manifest provider Microsoft.NET.Sdk.WorkloadManifestReader.SdkDirectoryWorkloadManifestProvider returned a duplicate manifest ID 'microsoft.net.sdk.android' [/vmr/artifacts/obj/extracted-dotnet-sdk/sdk-manifests/9.0.100-preview.1/microsoft.net.sdk.android/34.99.0-preview.1.151/WorkloadManifest.json] that conflicts with existing manifest [/vmr/artifacts/obj/extracted-dotnet-sdk/sdk-manifests/9.0.100-preview.1/microsoft.net.sdk.android/34.99.0-preview.1.151/WorkloadManifest.json]

[rmarinho]

We are hitting this also when executing some of our template tests on dotnet/maui

https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=10432032&view=ms.vss-test-web.build-test-results-tab&runId=114116150&resultId=100022&paneView=attachments

Standard_Console_Output-3.log