Update Caddy docs

[mholt]

Someone in our community noted that the Caddy docs on Stalwart's website were a bit old or unclear/inaccurate.

I am not a Stalwart user but wanted to check if Stalwart does in fact use HTTP? The suggested Caddyfiles proxy HTTP, not raw TCP.

Also, Caddy does support the PROXY protocol as of a while ago: https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#proxy_protocol

For proxying TCP, there's a layer 4 plugin that does this: https://github.com/mholt/caddy-l4

And instead of copying certs with a cron job, Caddy has an eventing system that can be utilized more appropriately. For example: https://github.com/mholt/caddy-events-exec (formal documentation is still forthcoming so it's understandable that this was missed; it's also new).

[mdecimus]

Hi,

Thank you for pointing this out.

Yes, Stalwart uses HTTP for multiple purposes (JMAP, REST API, ACME, MTA-STS, etc) in addition to the traditional email protocols.

The Caddy configuration file in our documentation was a user contribution and I haven't personally tested it. I assumed that the proxy protocol was not supported by Caddy because multiple Stalwart users reported having problems configuring/using the L4 plugin (they couldn't find examples on the Caddy website I believe) and, in addition to this, there were some (probably old?) posts around the internet mentioning that Caddy did not support the proxy protocol.

I will ask around in our Discord community to see if any of our users has the L4 plugin working with Stalwart and can contribute their Caddy configuration file to the Stalwart docs.

Thanks.

[MarcA711]

Hey,
I am currently trying to configure caddy for stalwart. Once I get it up and running I can gladly share my configuration. However, I have two questions and hope that someone can help me:

1. If Caddy handles the secure (encrypted) connections to the client, can Caddy establish insecure (unencrypted) connections to stalwart? For example, the client connects to caddy via https (443) or imaps (993), can caddy then connect unencrypted via http (8080) or imap (143) to stalwart or should an encrypted connection still be established here to avoid security vulnerabilities?
2. If caddy handles all outgoing connections, does stalwart need access to the certificate of mail.example.com? Or can stalwart use a self-signed certificate for encrypted connections to caddy/ no certificate at all if caddy only connects unencrypted?

[mholt]

should an encrypted connection still be established here to avoid security vulnerabilities?

Reverse-proxying to plaintext endpoints is totally normal and acceptable if the network is trusted; i.e. the loopback interface or a private network that you trust/control. A common use case is to terminate TLS for backend apps.

Or can stalwart use a self-signed certificate for encrypted connections to caddy/ no certificate at all if caddy only connects unencrypted?

I can't answer about Stalwart, but Caddy can be configured to accept self-signed certificates to backends. However, again, if the internal network is trusted/private, no cert may be required at all.

[MarcA711]

Thank you really much for answering my questions @mholt!

Then the only remaining question is whether stalwart needs access to the certificate for mail.example.com for another reason than handling TLS. I hope @mdecimus or another stalwart expert can help me here.

[mdecimus]

does stalwart need access to the certificate of mail.example.com?

Yes, the TLS certificates are needed for plain-text connections that are upgraded to TLS with the STARTTLS SMTP or IMAP command.

[MarcA711]

@mdecimus Thank you for you answer. I am working on it and will share my config once everything is running.

However, one other thing that I noticed: The docs for setting up traefik suggest that you mount the docker sock into the traefik container. This is questionable in terms of security. The docs should mention that this configuration is just an insecure example. Or even better, the configuration could be improved. Traefik can be configured using a config file instead of docker labels, which requires no access to the docker sock. As an alternative, there a docker sockets proxies, for example https://github.com/wollomatic/socket-proxy and https://github.com/Tecnativa/docker-socket-proxy.

If you want, I can open a separate issue for easier issue tracking

[mdecimus]

If you want, I can open a separate issue for easier issue tracking

Sure, thank you. Or if you are planning to submit a patch to the documentation, you can include both updated configurations in the same PR.

[MarcA711]

Hey,

@mdecimus I am no traefik user so I have no idea how to fix or improve the configuration myself. But once I submit a PR for caddy I can add a warning for traefik.

So far I didn't use caddy l4 for handling tcp traffic on other ports. Caddy l4 can't handle STARTTLS (afaik), so stalwart will need access to the certificate in any case. It will also break certain stalwart features. For example, if caddy l4 terminates tls traffic on 993 and proxies it to stalwarts 143, stalwart thinks that imap runs on 143 without implicit tls. This results in wrong configuration in autoconfig/autodiscover and wrong entries in "view dns records".
I think using caddy l4 for stalwart is not useful as long as you don't need loadbalancing. Otherwise it is just additional configuration. Or am I missing any advantage?

Below is my Caddyfile. Any feedback is very welcome. It handles http requests and certificates for stalwart. It will also use the API to reload stalwart after a new certificate was obtained. I am currently very busy myself, but I would like submit a PR and explain the configuration a bit. But it will maybe take a couple of weeks until I can do this.

Caddyfile:

{
	email [email protected]
	admin off
	log {
		output file /data/log/error.log
		level ERROR
	}
	events {
		on cert_obtained exec /bin/sh -c <<RENEW_CERT
			[[ {event.data.identifier} == mail.example.com ]] &&
			cp -rf /data/caddy/{event.data.storage_path} /cert/ &&
			wget --header="Accept: application/json" --header="Authorization: Bearer {$STALWART_API}" -qO- http://stalwart:8081/api/reload/
			RENEW_CERT
	}
}

mail.example.com, autodiscover.example.com, autoconfig.example.com, mta-sts.example.com {
	log {
		output file /data/log/access-stalwart.log
		level INFO
		format json
	}
	tls {
		reuse_private_keys
	}
	reverse_proxy http://stalwart:8080 {
		transport http {
			proxy_protocol v2
		}
	}
}

Stalwart config.toml:

certificate.default.cert = "%{file:/cert/mail.example.com/mail.example.com.crt}%"
certificate.default.default = true
certificate.default.private-key = "%{file:/cert/mail.example.com/mail.example.com.key}%"


server.listener.http.bind = "[::]:8080"
server.listener.http.protocol = "http"
server.listener.http.proxy.override = true
server.listener.http.proxy.trusted-networks.0 = "172.10.0.10/32"

server.listener.api.bind = "[::]:8081"
server.listener.api.protocol = "http"

Additional configuration: Pass an API key as environment variable STALWART_API to caddy. Mount a directory to both stalwart and caddy under /cert. Give caddy a static IP like "172.10.0.10".

[MarcA711]

@mdecimus Sorry to bother you with another question, but is there another way to get an API key with just "Refresh system settings" and "Authenticate" permissions set to yes (everything else set to no)? I went through the whole list and selected "no" for every other permission which was kind of tedious.

[mdecimus]

@MarcA711 Please open a Github discussion on the mail-server repository for issues that are not related to fixing the Caddy documentation. Thanks.

[xenadmin]

Hello and good day! I would like to participate in this issue and test and improve the configuration for Caddy.
I also installed stalwart-mail and Caddy via Docker yesterday.

Also, hello @MarcA711 from Germany! Maybe we can connect to make Caddy work via 443 and improve the example in the docs?

In my scenario, I decided that Stalwart can independently manage ports 25, 465, 993, 587 and (8080). Port 443 should run via Caddy because my server already offers other services on this port via port 443.

Caddy and stalwart-mail both use the Let's Encrypt ACME DNS-01 Challenge, so ports 80 and 443 are irrelevant for the certificate.

I can already send and receive emails, but everything that has to do with port 443 doesn't work (I think).

I'm posting my current config and the associated error messages here:

stalwart-mail docker compose YAML

services:
  stalwart-mail:
    container_name: stalwart-mail
    image: stalwartlabs/mail-server:latest
    restart: unless-stopped
    ports:
      # Essential ports
      - "25:25" # SMTP
      - "465:465" # SMTPS
      - "993:993" # IMAPS
      # - "443:443" # HTTPS
      # Non-essential ports
      - "587:587" # Submission
      # - "143:143" # IMAP
      # - "4190:4190" # ManageSieve
      # - "110:110" # POP3
      # - "995:995" # POP3S
      # Setup port
      - "8080:8080" # Setup
    volumes:
      - ./data:/opt/stalwart-mail
    networks:
      - reverse-proxy

networks:
  reverse-proxy:
    external: true

Caddyfile (one of many tries)

mail.{$DOMAIN-MKF}, autoconfig.{$DOMAIN-MKF}, autodiscover.{$DOMAIN-MKF}, mta-sts.{$DOMAIN-MKF} {
	log {
		output file /data/log/access-stalwart.log
		level INFO
		format json
	}
	reverse_proxy stalwart-mail:443 {
		transport http {
			proxy_protocol v2
			tls_insecure_skip_verify
		}
	}
}

Error from Caddy log. IMHO the relevant line is the 502 error:

{
    "level": "error",
    "ts": 1738195661.941174,
    "logger": "http.log.access.log0",
    "msg": "handled request",
    "request": {
        "remote_ip": "1.2.3.4",
        "remote_port": "51485",
        "client_ip": "1.2.3.4",
        "proto": "HTTP/2.0",
        "method": "GET",
        "host": "mail.home.net",
        "uri": "/login",
        "headers": {
            "User-Agent": [
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0"
            ],
            "Cookie": [
                "REDACTED"
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "Te": [
                "trailers"
            ],
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Sec-Fetch-Site": [
                "cross-site"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br, zstd"
            ],
            "Priority": [
                "u=0, i"
            ],
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
            ],
            "Accept-Language": [
                "de,en-US;q=0.7,en;q=0.3"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h2",
            "server_name": "mail.home.net"
        }
    },
    "bytes_read": 0,
    "user_id": "",
    "duration": 0.001701237,
    "size": 0,
    "status": 502,
    "resp_headers": {
        "Server": [
            "Caddy"
        ],
        "Alt-Svc": [
            "h3=\":443\"; ma=2592000"
        ]
    }
}

Error from stalwart-mail log. 172.18.0.2 is the IP from Caddy.

2025-01-30T09:02:39Z DEBUG TLS handshake error (tls.handshake-error) listenerId = "https", localPort = 443, remoteIp = 172.18.0.2, remotePort = 56316, reason = "received corrupt message of type InvalidContentType"

Ps.: Of course, I tested what happens, if I disable Caddy temporary and enable Port 443 in the stalwart-mail docker compose.
The Admin login is reachable immediately.

[MarcA711]

Hey, would be great if we can work on this together.

To your error:
Maybe specify https explicitly by replacing the line:
reverse_proxy stalwart-mail:443 with reverse_proxy https://stalwart-mail:443 in your Caddyfile.

[xenadmin]

Hi! I tried to replace the line, no change. Errros stay the same.
I believe my error is, that I didn't configure anything in stalwart-mail (config.toml) itself, to adjust Port 443 being handle by Caddy.
But I find the documentation and your comment in this regard more confusing than helpful.

EDIT1: I tried adding the following to the config.toml, without success:

server.listener.https.bind = "[::]:443"
server.listener.https.protocol = "http"
server.listener.https.proxy.override = true
server.listener.https.proxy.trusted-networks = "172.18.0.0/16"
server.listener.https.socket.override = false
server.listener.https.tls.implicit = true
server.listener.https.tls.override = false

[MarcA711]

Yes, you need to configure a listener for port 443 in stalwarts config.toml. Otherwise stalwart won't bind to port 443.

I had it working before with https, but as discussed above using https for proxying on the same machine shouldn't bring any advantage. This is why I switched to unencrypted http.

When I used https, I didn't specify these two lines:

server.listener.https.socket.override = false
Server.listener.https.tls.override = false

Does it work if you remove these?

What specific part do you find confusing about the docs or my comment when it comes to binging to ports?

[xenadmin]

So I've tried many more configurations. I know a bit more, but I'm not near the solution.

Does it work if you remove these?

Doesn't make any difference. They get auto-generated, when I save this window:

Further on, I tested the following:

Still doesn't work.

What does work is, when I disable proxy protocol in Caddy. I can immediately reach the admin webinterface through caddy 443.

mail.{$DOMAIN-MKF}, autoconfig.{$DOMAIN-MKF}, autodiscover.{$DOMAIN-MKF}, mta-sts.{$DOMAIN-MKF} {
	log {
		output file /data/log/access-stalwart.log
		level INFO
		format json
	}
	reverse_proxy stalwart-mail:443 {
		transport http {
			# proxy_protocol v2
			tls_insecure_skip_verify
		}
	}
}

But when I do that, the stalwart logs looks like this:

2025-01-30T15:37:56Z DEBUG HTTP request URL (http.request-url) listenerId = "https", localPort = 443, remoteIp = 172.18.0.2, remotePort = 41850, remoteIp = 123.123.123.123, url = "/login"

And now I'm even more confused. How to read that line? There are my Caddy and my public IP. But I've proxy_protocol disabled. I know from @mdecimus in Discord, that it's very important for stalwart to receive the Original Remote IP on Port 443, to make use of all it's features.

What specific part do you find confusing about the docs or my comment when it comes to binging to ports?

Regarding your question:

First: I find it confusing, that the syntax in the docs for proxy protocol look kinda different from what I see in my original local config.toml that gets generated from the webadmin interface.

Docs:

local config.toml:

Which one is to current up-to-date working syntax?

Second: You also post an example about settings in the config.toml. But how are we expected to add those changes? Is it allowed to edit the file directly? Will webadmin overwrite it? Can I mix manual edits and Web-Gui edits? Every software behaves differently about things like that.

Third: I think it's maybe unnecessary complicated to link Stalwart with the cert from Caddy?
I use the Let's Encrypt DNS-01 challenge in both products, so they can both create certs independently of each other, without incoming traffic. So they both own a valid cert for mail.home.net, but of course, with different private keys.
PS.: I hope that's not part of my issue..

Conclusion: As soon as I enable proxy protocol, I have the following error in my stalwart log:

2025-01-30T15:59:41Z DEBUG TLS handshake error (tls.handshake-error) listenerId = "https", localPort = 443, remoteIp = 172.18.0.2, remotePort = 58854, reason = "received corrupt message of type InvalidContentType"

Notice, that the public IP is gone now?!
And the 502 (http?) error is back in the caddy log.

[MarcA711]

Hey,

I am not sure. Could you post the access-stalwart.log from caddy? Did you try using http instead of https (I mean for proxying stalwart to caddy, not for public connections)?

Which one is to current up-to-date working syntax?

I didn't know that your syntax even works. I always used cidr range and specified it in a list rather than a single string with an IP.

Second: You also post an example about settings in the config.toml. But how are we expected to add those changes? Is it allowed to edit the file directly? Will webadmin overwrite it? Can I mix manual edits and Web-Gui edits? Every software behaves differently about things like that.

I usually edit the file directly, but I also use the webadmin sometimes. Stalwart once reformatted my config without changing any values. So I think mixing is allowed and it is fine to add the lines that I posted above to the config.toml directly.

Third: I think it's maybe unnecessary complicated to link Stalwart with the cert from Caddy?
I use the Let's Encrypt DNS-01 challenge in both products, so they can both create certs independently of each other, without incoming traffic. So they both own a valid cert for mail.home.net, but of course, with different private keys.

I guess this depends on what one thinks is more complicated. Configuring DNS-01 in both services or configuring caddy to update the cert for stalwart. The certificate renewal of caddy is very robust and can fallback to different services if let's encrypt doesn't work.

Moreover, I don't think that using two certificates for one fqdn is good practice. It might cause issues in combination with dane. But I am not sure.