Configure ssh client not to store host keys for floating hostnames #716
Conversation
Some of our hosts are redundant with keepalived in a primary/backup pattern, and we sometimes want to use them as jump hosts with ssh. Usually ssh will proxy through the primary host, but in those rare moments where the backup is claiming the IP address, we still want to connect, even though the backup host has different ssh host keys. Even when /etc/ssh/ssh_known_hosts is managed, the ssh client will also store keys in the user's known_hosts file, resulting in the possibility of failed ssh connections while e.g. in the middle of a reboot. We can't configure ssh not to save the host keys, but we can configure it to write them to /dev/null. We also set the log level to ERROR because otherwise every single connection will yell that this is the first time it's seen this server and it's saving the host keys in /dev/null.
|
This seems to defeat the point of having ssh host keys at all for these hosts and at least theoretically introduces a new risk. Was there any review/discussion of this? Maybe it would it be preferable to configure both of the redundant hosts with the same host key -- it seems like there must be some best practice for this situation? |
|
That is another option! But FWIW this shouldn't open a new risk, because the only hosts this impacts have their /etc/ssh/ssh_known_hosts populated by puppet, so having the ssh client write them to ~/.ssh/known_hosts is redundant in this case. Here's where that gets exported by kubernetes gateways, specifically for these shared hostnames: https://github.com/mlibrary/nebula/blob/production/manifests/profile/kubernetes/keepalived.pp#L62-L71 If a MITM attacker manages to get in there, they would have a third host key that matches neither the primary nor the backup host—their key wouldn't be in /etc/ssh/ssh_known_hosts—and so the ssh client would still bail. We went with this because it seemed easier than figuring out how to dist host keys to servers that we're currently just letting be, but as we talk more about using terraform for VM creation, I think that all starts to change. I think managing our host keys on VM creation is, in my mind, right there next to managing the puppet pki without relying on the puppetserver. |
|
Thanks -- I discussed this a bit with @rrotter and I think the point about having |
Some of our hosts are redundant with keepalived in a primary/backup pattern, and we sometimes want to use them as jump hosts with ssh.
Usually ssh will proxy through the primary host, but in those rare moments where the backup is claiming the IP address, we still want to connect, even though the backup host has different ssh host keys.
Even when /etc/ssh/ssh_known_hosts is managed, the ssh client will also store keys in the user's known_hosts file, resulting in the possibility of failed ssh connections while e.g. in the middle of a reboot.
We can't configure ssh not to save the host keys, but we can configure it to write them to /dev/null. We also set the log level to ERROR because otherwise every single connection will yell that this is the first time it's seen this server and it's saving the host keys in /dev/null.