Fix SSH tunnel MTU with iptables

One of my server provides two incompatibles services regarding the MTU of his ethernet interface.

The first service is a SSH server that many embedded devices connect to with SSH tunnel enabled to let manage them remotely. This service is prone to hit the well know stall when big packets are exchanged inside the tunnel because the TCP MTU is too large and the ICMP messages to reduce them are dropped by some firewall between the device and the server. The usual way to solve this problem is to set the MTU of the ethernet interface to a bit lower than 1500, I use 1412. It work very well, but…

The second service is a TFTP server that provides a PXE image, including SYSLINUX and the Linux kernel, to boot a embedded device in testing mode by just disengaging the CF memory card (removing it is painful due to the mechanical design) and connecting a ethernet cable to it. Sadly the TFTP/PXE exchange require a MTU of 1500 on the ethernet interface to work, at least with the BIOS of this embedded device. The quick and dirty trick I used was to set the MTU of the ethernet interface of the server to 1500 just the time to boot the tested device and to revert back to a MTU of 1412 immediately  after to let the SSH tunnel work without stall.

The point to understand is that the interface MTU is used by default to set the TCP MTU, because the TCP MTU can’t be bigger than the interface MTU without fragmentation, but the TCP MTU can be smaller than the interface MTU without problem. This is in fact what will solve my problem for the two services: I want a interface MTU of 1500 and a TCP MTU of 1412. How to get that ?

The Linux kernel netfilter can set the TCP MTU on the initialization of a TCP stream using a iptables command. This is commonly used on router that use NAT between interfaces with different MTU (if we don’t look at the details). But in my case I want to only affect the SSH service and it’s not a router. I finally found that this command do the job:

iptables -t mangle -A OUTPUT -p tcp --sport 22 \
-m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1412