"Could not locate PKCS 8 private keys" Error Running in Container in k3s

[kbjr]

Hi,

I have recently started trying to get the JMAP server running in my k3s homelab cluster, and have run into an issue pretty early on; I assume I'm just missing something easy, but don't have much to go on at the moment.

I'm using the stalwartlabs/jmap-server:latest image from docker, and the only log output I get from the server is this:

2022-09-29T00:59:09.742361Z INFO stalwart_jmap::lmtp::listener Starting LMTP service at 0.0.0.0:11200...
2022-09-29T00:59:09.742712Z ERROR stalwart_jmap::cluster::rpc::tls Could not locate PKCS 8 private keys.

This seems to indicate that something is wrong with the private key associated to the TLS certificate(s). Specifically, it first seemed like (from the ::cluster::rpc::tls part of the message) that the error is related to the clustering RPC config (as described here https://stalw.art/jmap/cluster/rpc/). That error messages appears to come from this load_tls_server_config function:

jmap-server/src/cluster/rpc/tls.rs

Lines 75 to 85 in 87cc39c

	let mut keys: Vec<PrivateKey> = pkcs8_private_keys(key_file)
	.failed_to("parse jey file")
	.into_iter()
	.map(PrivateKey)
	.collect();
	// exit if no keys could be parsed
	if keys.is_empty() {
	error!("Could not locate PKCS 8 private keys.");
	std::process::exit(1);
	}

Unfortunately, it seems like that function is actually also used by the JMAP server:

jmap-server/src/server/http.rs

Lines 243 to 253 in 9edcb2e

	let tls_config = if let Some(cert_path) = settings.get("jmap-cert-path") {
	load_tls_server_config(
	&cert_path,
	&settings
	.get("jmap-key-path")
	.failed_to("load TLS config, missing 'jmap-key-path' argument."),
	)
	.into()
	} else {
	None
	};

and the LMTP server:

jmap-server/src/lmtp/listener.rs

Lines 83 to 93 in 87cc39c

	let tls_acceptor = if let (Some(cert_path), Some(key_path)) = (
	settings.get("lmtp-cert-path"),
	settings.get("lmtp-key-path"),
	) {
	Arc::new(TlsAcceptor::from(Arc::new(load_tls_server_config(
	&cert_path, &key_path,
	))))
	.into()
	} else {
	None
	};

so its a bit difficult to say which one is actually the source of the error.

I'm also fairly sure the certificate files are correctly being mounted into the container; I'm just using a regular volume mount to mount a kubernetes secret into a directory and pointing to that location with environment variables.

volumeMounts:
- name: stalwart-jmap-certs
  mountPath: /usr/local/stalwart-jmap/certs
  readOnly: true

JMAP_KEY_PATH=/usr/local/stalwart-jmap/certs/tls.key
JMAP_CERT_PATH=/usr/local/stalwart-jmap/certs/tls.crt

If I change any part of this such that these paths no longer match, I (expectedly) get different log output telling me the file couldn't be found:

2022-09-29T01:08:50.630810Z  INFO stalwart_jmap::lmtp::listener: Starting LMTP service at 0.0.0.0:11200...
Failed to open certificate file: No such file or directory (os error 2)

I had been assuming that the server causing the problem is the LMTP server, based on the first log line, but the error seems more attached to the JMAP_* config variables and I get the error even when the LMTP_* variables are not provided at all.

Based on the context around where the error is logged from, it looks like this error is specifically for if the file is found, but the key file fails to be parsed; The certificate / key themselves are being created by cert-manager, and I've looked at the key file itself and it looks like I would expect of a key file, i.e.

-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----

I also tried passing LOG_LEVEL=trace to get more info, but didn't get any additional log output.

So, any thoughts as to what I might be missing?

[mdecimus]

Hi,

Just to help us debug the problem, could you try using self-signed certificates on all services?

openssl req -x509 -nodes -days 1825 -newkey rsa:4096 -subj '/CN=localhost' -keyout ${CFG_PATH}/private/jmap.key -out ${CFG_PATH}/certs/jmap.crt
openssl req -x509 -nodes -days 1825 -newkey rsa:4096 -subj '/CN=localhost' -keyout ${CFG_PATH}/private/lmtp.key -out ${CFG_PATH}/certs/lmtp.crt
openssl req -x509 -nodes -days 1825 -newkey rsa:4096 -subj '/CN=localhost' -keyout ${CFG_PATH}/private/rpc.key -out ${CFG_PATH}/certs/rpc.crt

I'd like to see if this is a problem related to the certificate files or the mount volumes.

[kbjr]

Well, the self-signed certs do work, so it looks like the key file itself is the issue somehow. I created a new kubernetes secret with the new self-signed files and mounted it in the same way, and the log output looks much healthier now.

So, I guess I'll go try to figure out what the difference is between these keys now

[mdecimus]

The certificates you are trying to use were they issued by Letsencrypt or some other entity? If they are from Letsencrypt I believe you need to use the fullchain.pem certificate file.

[kbjr]

Yeah, they are Letsencrypt.

I just figured it out, though. The root of the problem was that the certs were PKCS#1 encoded instead of PKCS#8.

I thought I had already corrected that, but apparently reconfiguring the cert-manager Certificate resource to change the encoding of an already issued certificate does not actually cause anything to be updated or re-issued; I needed to fully delete those secrets and Certificates and rebuild them.

Thanks for the help.

[Lykos153]

I ran into this problem today, too. Took me a while to figure it out and only now I find this thread. Let's put this somewhere more visible so new users can find it more easily! (Or, better yet, add support for PKCS#1).

For future reference, to fix the issue add

  privateKey:
    encoding: PKCS8

to your Certificate resource under spec.

@kbjr btw it's actually sufficient to delete the Secret after this change. cert-manager will then re-create it from the Certificate with the correct encoding.

[plantroon]

I ran into this problem and do not have certificate manager (ie. whatever is mentioned here is not applicable). But this is a quick dirty fix:

openssl pkcs8 -topk8 -inform PEM -outform PEM -in original_key.pem -out pkcs8_key.pem -nocrypt

You can use it as part of your certificate deployment if you don't have a better way.