Disabling Offloading on Ubuntu 18.04 with Netplan and/or systemd-networkd
Why Disable Offloading?
Firstly, why not disable offloading? Offloading is a way for the operating system and your CPU to offload some of the work involved in transmitting packets (which can be significant, especially at higher speeds. The two main cases that I’ve come across for this are offloading the calculation of TCP checksums and handling fragmentation. Hardware offloading should be a good thing and it often is, I’ve only rarely found a need to disable it.
However, sometimes it just seems to break things and you can’t find a better work around. For example, I had a problem earlier where a particular host of a network was sending TCP segments and they were being routed to their destinations but then being discarded. I set up a packet capture on another remote host under our control and I was seeing a SYN arriving and then more SYNs as retransmission was attempted and no SYN-ACK was being sent back. This was isolated to this host, no other hosts on the sending network had this issue. I ruled out the other usual suspects - NAT, firewall rules etc.
Finally, I decided there was nothing left to try other than disable offloading and like that, the issue was gone! I had to disable checksum offloading, but you may have a slightly different issue.
Disabling Offloading Temporarily
This should work whatever you’re using to manage your network.
To do this, you’ll need to have ethtool installed:
sudo apt install ethtool
Once it’s installed, work out what interface is affected. For the sake of this post, we’ll say it’s eth0, but it could be something completely different (e.g. br1, vnet2, enp1s2)
My problem was caused by checksum offloading, which can be disabled by running this command:
sudo ethtool --offload eth0 rx off tx off
rx off disables for things you’re receiving.
tx off disables it for things you’re transmitting.
If the problem goes away, then you’ve found the cause. Great. Bear in mind though, when you next reboot, these settings will be reset to default and the problem will come back.
If this didn’t solve the problem, it may be worth looking at some of the features that can be enabled/disabled:
rx- receive (RX) checksumming
tx- transmit (TX) checksumming
tso- TCP segmentation offload
ufo- UDP fragmentation offload
gso- generic segmentation offload
gro- generic receive offload
lro- large receive offload
rxvlan- receive (RX) VLAN acceleration
txvlan- transmit (TX) VLAN acceleration
ntuple- receive (RX) ntuple filters and actions
rxhash- receive hashing offload
In general, it’s best to disable the minimum amount of features you need to resolve your issue rather than blanket disabling everything.
Netplan and systemd-networkd
Ubuntu 18.04 brought with it Canonical’s new answer to the problem of network configuration, netplan. I don’t particularly like netplan, but I also like to use the defaults wherever possible. On a server, the best choice is to use the systemd-networkd backend to netplan (the other choice is NetworkManager). You might argue that an even better choice is to cut netplan out of the loop and use systemd-networkd directly. In either case, I’m assuming in this post that you’re either using netplan and systemd-networkd together or systemd-networkd alone.
If you’ve used netplan, you’ll realise that for basic uses, it’s quite nice and neat, but for more advanced uses, you start to find the bugs and the missing features (though it is continuing to improve).
Disabling Offloading Permanently
Netplan doesn’t offer a way to turn off offloading within its configuration. Systemd-networkd does provide some options to disable offloading, although I wasn’t able to find options for all of the features available in ethtool, for this reason I’ve not investigated it further.
Luckily, Ubuntu 18.04 comes with the
networkd-dispatcher package. You
can use this to run scripts (like legacy ifup/ifdown scripts).
If you’re missing the package:
sudo apt install networkd-dispatcher
We need to create a script to run as the network comes up:
sudo nano /usr/lib/networkd-dispatcher/routable.d/10-disable-offloading
I put the following content in it:
#!/bin/sh ethtool --offload eth0 rx off tx off
You’ll need to change this if you’ve used other options or a different network interface.
Finally, ensure that it’s executable:
sudo chmod +x /usr/lib/networkd-dispatcher/routable.d/10-disable-offloading
If everything has worked, you’ll now be able to boot without issue.
Please note, I’ve located this configuration file in
/usr/lib/networkd-dispatcher due to
LP #1765152, you
/etc/networkd-dispatcher in other distributions or other documentation. The correct
location in Ubuntu 18.04 is in
/usr/lib/networkd-dispatcher, even if that seems a bit odd.