NATS generates incorrect signature when negotiating TLS using a P384/SHA384 certificate from the windows cert store

[andrew-strachan-ggl]

Observed behavior

nats-server configured with a P384/SHA384 certificate and a NATS.Net client attempting to connect throws an exception on the client.

In certstore_windows.go, when winSignECDSA is called with a 32 byte digest, winNCryptSignHash returns a 96 byte signature.

However, winPackECDSASigValue uses the length of the digest (32 bytes) as the size of sigR and sigS, meaning not all of the signature generated by winNCryptSignHash is used in the generation of the signature sent as part of the TLS negotiation.

This causes a cannot contact local security authority exception to be thrown when the client attempts to continue the TLS handshake.

When using the nats cli, the digest passed in is 48 bytes long, so the issue doesn't arise.

Expected behavior

The TLS negotiation completes successfully

Server and client version

nats-server: v2.11.0-dev
NATS.Net 2.5.5

Host environment

Windows 10 Enterprise
22H2
19045.5371

Steps to reproduce

Create the certificate

openssl req -x509 -days 365 -newkey ec -pkeyopt ec_paramgen_curve:secp384r1 -sha384 -subj "/CN=nats-server" --addext "subjectAltName=IP:127.0.0.1,DNS:localhost" -nodes -out nats.pem -keyout nats.key -outform PEM
openssl pkcs12 -inkey nats.key -in nats.pem -export -out nats.pfx

Import the certificate into the windows cert store, then configure nats-server to use that cert:
e.g. nats.conf

{
  "host": "localhost",
  "port": 4222,
  "tls": {
    "cert_store": "windowscurrentuser",
    "cert_match_by": "thumbprint",
    "cert_match": "<certificate SHA1 thumbprint>",    
    "verify": false
  }
}

Start nats:

nats-server -c nats.conf

Using a minimal C# program:

var client = new NATS.Net.NatsClient("nats://127.0.0.1:4222");

await client.ConnectAsync();

The connect line throws an exception:

NATS.Client.Core.NatsException
  HResult=0x80131500
  Message=can not start to connect nats server: nats://127.0.0.1:4222
  Source=NATS.Client.Core
  StackTrace:
   at NATS.Client.Core.NatsConnection.<InitialConnectAsync>d__88.MoveNext()
   at NATS.Client.Core.NatsConnection.<ConnectAsync>d__77.MoveNext()
   at Program.<<Main>$>d__0.MoveNext() in c:\temp\natsrepro\TestClient\Program.cs:line 3
   at Program.<Main>(String[] args)

Inner Exception 1:
NatsException: TLS authentication failed

Inner Exception 2:
AuthenticationException: Authentication failed, see inner exception.

Inner Exception 3:
Win32Exception: The Local Security Authority cannot be contacted