Disabling Offloading on Ubuntu 18.04 with Netplan and/or systemd-networkd

Why Disable Offloading?

Firstly, why not disable offloading? Offloading is a way for the operating system and your CPU to offload some of the work involved in transmitting packets (which can be significant, especially at higher speeds. The two main cases that I’ve come across for this are offloading the calculation of TCP checksums and handling fragmentation. Hardware offloading should be a good thing and it often is, I’ve only rarely found a need to disable it.

However, sometimes it just seems to break things and you can’t find a better work around. For example, I had a problem earlier where a particular host of a network was sending TCP segments and they were being routed to their destinations but then being discarded. I set up a packet capture on another remote host under our control and I was seeing a SYN arriving and then more SYNs as retransmission was attempted and no SYN-ACK was being sent back. This was isolated to this host, no other hosts on the sending network had this issue. I ruled out the other usual suspects - NAT, firewall rules etc.

Finally, I decided there was nothing left to try other than disable offloading and like that, the issue was gone! I had to disable checksum offloading, but you may have a slightly different issue.

Disabling Offloading Temporarily

This should work whatever you’re using to manage your network.

To do this, you’ll need to have ethtool installed:

sudo apt install ethtool

Once it’s installed, work out what interface is affected. For the sake of this post, we’ll say it’s eth0, but it could be something completely different (e.g. br1, vnet2, enp1s2)

My problem was caused by checksum offloading, which can be disabled by running this command:

sudo ethtool --offload eth0  rx off  tx off

rx off disables for things you’re receiving. tx off disables it for things you’re transmitting.

If the problem goes away, then you’ve found the cause. Great. Bear in mind though, when you next reboot, these settings will be reset to default and the problem will come back.

If this didn’t solve the problem, it may be worth looking at some of the features that can be enabled/disabled:

In general, it’s best to disable the minimum amount of features you need to resolve your issue rather than blanket disabling everything.

Netplan and systemd-networkd

Ubuntu 18.04 brought with it Canonical’s new answer to the problem of network configuration, netplan. I don’t particularly like netplan, but I also like to use the defaults wherever possible. On a server, the best choice is to use the systemd-networkd backend to netplan (the other choice is NetworkManager). You might argue that an even better choice is to cut netplan out of the loop and use systemd-networkd directly. In either case, I’m assuming in this post that you’re either using netplan and systemd-networkd together or systemd-networkd alone.

If you’ve used netplan, you’ll realise that for basic uses, it’s quite nice and neat, but for more advanced uses, you start to find the bugs and the missing features (though it is continuing to improve).

Disabling Offloading Permanently

Netplan doesn’t offer a way to turn off offloading within its configuration. Systemd-networkd does provide some options to disable offloading, although I wasn’t able to find options for all of the features available in ethtool, for this reason I’ve not investigated it further.

Luckily, Ubuntu 18.04 comes with the networkd-dispatcher package. You can use this to run scripts (like legacy ifup/ifdown scripts).

If you’re missing the package:

sudo apt install networkd-dispatcher

We need to create a script to run as the network comes up:

sudo nano /usr/lib/networkd-dispatcher/routable.d/10-disable-offloading

I put the following content in it:

#!/bin/sh
ethtool --offload eth0 rx off tx off

You’ll need to change this if you’ve used other options or a different network interface.

Finally, ensure that it’s executable:

sudo chmod +x /usr/lib/networkd-dispatcher/routable.d/10-disable-offloading

If everything has worked, you’ll now be able to boot without issue.

Please note, I’ve located this configuration file in /usr/lib/networkd-dispatcher due to LP #1765152, you may see /etc/networkd-dispatcher in other distributions or other documentation. The correct location in Ubuntu 18.04 is in /usr/lib/networkd-dispatcher, even if that seems a bit odd.