Add hooks to debug OpenSSL memory allocations

[rzikm]

This ressurects #101626. CC: @wfurt.

Changes since his PR:

• Moved some code around, the managed part now lives in Interop.Crypto.
• Moved tracking to native code and exposed to C# via foreach-callback

Open questions:

• Should the functionality (or some part of it) be under #if DEBUG?

We had several cases when users complained about large memory use. For than native it is quite difficult to figure out where the memory goes. This PR aims to make that somewhat easier.

OpenSSL provides hooks for memory function so this PR adds switch to optimally hook into that.
The only one caveat that the CRYPTO_set_mem_functions works only if called before any allocations e.g. it needs to be done very early in the process. So I end up putting into initialization process.

The simple use pattern is something like

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net.Http;
using System.Net.Security;
using System.Reflection;
using System.Runtime.InteropServices;
// Environment variable needs to be set before launching the process, as otherwise the native layer will
// not see the environment variable
// Environment.SetEnvironmentVariable("DOTNET_SYSTEM_NET_SECURITY_OPENSSL_MEMORY_DEBUG", "1");
//
// Once enabled, the functionality can be accessed by following methods on the Interop.Crypto class:
// - GetOpenSslAllocatedMemory - Gets the total amount of memory allocated by OpenSSL
// - GetOpenSslAllocationCount - Gets the number of allocations made by OpenSSL
// - EnableTracking/DisableTracking - toggles tracking of individual (outstanding) allocations via internal dictionary.
// - ForEachTrackedAllocation - Accepts an Action<IntPtr, int, IntPtr, int> callback and calls it for each allocation performed since the last EnableTracking call. The order of allocations is not guaranteed, the functions holds an inner lock which prevents concurrent memory operations. The parameters passed to the callback are:
//   - IntPtr to the memory allocated
//   - size of the allocation
//   - IntPtr to the filename where the allocation was made
//   - line number of the allocation

var ci = typeof(SslStream).Assembly.GetTypes().First(t => t.Name == "Crypto");
ci.InvokeMember("EnableTracking", BindingFlags.InvokeMethod | BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Static, null, null, null);

HttpClient client = new HttpClient();
await client.GetAsync("https://www.google.com");

using var process = Process.GetCurrentProcess();
Console.WriteLine($"Bytes known to GC [{GC.GetTotalMemory(false)}], process working set [{process.WorkingSet64}]");
Console.WriteLine("OpenSSL memory {0}", ci.InvokeMember("GetOpenSslAllocatedMemory", BindingFlags.InvokeMethod | BindingFlags.Public | BindingFlags.Static, null, null, null));
Console.WriteLine("OpenSSL allocations {0}", ci.InvokeMember("GetOpenSslAllocationCount", BindingFlags.InvokeMethod | BindingFlags.Public | BindingFlags.Static, null, null, null));

Dictionary<(IntPtr file, int line), int> allAllocations = new Dictionary<(IntPtr file, int line), int>();

Action<IntPtr, int, IntPtr, int> callback = (ptr, size, namePtr, line) =>
{
    CollectionsMarshal.GetValueRefOrAddDefault(allAllocations, (namePtr, line), out _) += size;
};
ci.InvokeMember("ForEachTrackedAllocation", BindingFlags.InvokeMethod | BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Static | BindingFlags.Instance, null, null, new object[] { callback });

System.Console.WriteLine("Total allocated OpenSSL memory by location");
foreach (var ((filenameptr, line), total) in allAllocations.OrderByDescending(kvp => kvp.Value))
{
    string filename = Marshal.PtrToStringUTF8(filenameptr);
    Console.WriteLine($"{total:N0} B from {filename}:{line}");
}

Access through Reflection should be OK since this is only last resort debug hook e.g. it does not need stable API and convenient access.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[Copilot]

Copilot reviewed 2 out of 8 changed files in this pull request and generated 1 comment.

Files not reviewed (6)

• src/native/libs/System.Security.Cryptography.Native/apibridge_30.h: Language not supported
• src/native/libs/System.Security.Cryptography.Native/entrypoints.c: Language not supported
• src/native/libs/System.Security.Cryptography.Native/openssl.c: Language not supported
• src/native/libs/System.Security.Cryptography.Native/openssl.h: Language not supported
• src/native/libs/System.Security.Cryptography.Native/opensslshim.h: Language not supported
• src/native/libs/System.Security.Cryptography.Native/pal_ssl.c: Language not supported

[rzikm]

I moved the tracking to native code. And it seems stable during my tests.

I think we are ready for another round of review

[bartonjs]

What of this is important? It seems like it's mainly testing the P/Invokes, which suggests it should be deleted.

[rzikm]

The methods otherwise don't have a caller, meaning that the compiler will just eliminate them and they are not reachable via reflection.

There might be better way to prevent that from happening, I am open to suggestions

[jkotas]

The right way to do this is to list these methods in ILLink.Descriptors.LibraryBuild.xml. For example:

https://github.com/dotnet/runtime/blob/main/src/libraries/System.Text.Json/src/ILLink/ILLink.Descriptors.LibraryBuild.xml

[bartonjs]

I'm not sure that what you've done to work around the OSSL 1.0 version of this function will work in a non-portable build from 1.1.

I believe it will try to call the non-existent function CRYPTO_set_mem_functions11 and fail to link.

But maybe I was more clever with RENAMED_FUNCTION that I'm giving myself credit for.

[wfurt]

I do feel we do not need to worry about 1.0 anymore .... as long as it does not break the build.

[vcsjones]

I do feel we do not need to worry about 1.0 anymore .... as long as it does not break the build.

Official builds are currently done on Ubuntu 16.04 with 1.0.2. Jeremy and I have had discussions on bumping to 1.1, but at the moment I think it needs to build against 1.0.2 headers.