Fixing intermittent packet loss on a vSwitch with a periodic ARP request

Introduction

If you connect a Hetzner dedicated server to Hetzner cloud servers over a vSwitch, you may see short, irregular outages: traffic between the dedicated server and the cloud servers stops for a few seconds, then resumes on its own. Nothing is logged on either host, no interface counter moves, and the servers are otherwise healthy.

This tutorial covers how to confirm the cause from packet captures and how to fix it with one periodic ARP request. The underlying reason is specific to the EVPN/VXLAN fabric behind the vSwitch, and the fix is a few lines of systemd.

The symptom, in one line: traffic towards the dedicated server is discarded while its MAC/IP binding is missing from the router, and a Linux host with steady traffic never sends the ARP request that would refresh it.

Prerequisites

• A Hetzner dedicated server with a vSwitch VLAN interface, and one or more cloud servers in a Private Network attached to the same vSwitch
• root on the dedicated server
• tcpdump and iputils-arping

Example terminology

Every MAC below is masked: the host's own is written as aa:bb:cc:dd:ee:ff and the two gateway addresses as f2:aa:bb:cc:dd:01 and f2:aa:bb:cc:dd:02. The excerpts also drop tcpdump's ethertype ARP (0x0806), length 42: for width. The IP addresses, timestamps and ordering are as captured. Your own gateway MACs will be different, but they will follow the same pattern that matters here: two addresses differing only in the last octet.

Step 1 - Confirm the symptom

The outages are too short to catch by hand, so run a continuous probe and a continuous capture on both ends. A one-second ping is enough to detect it:

while :; do ping -n -c1 -W1 10.0.0.2 >/dev/null 2>&1 || \
  echo "$(date -u +%FT%TZ) miss"; sleep 1; done

and on the dedicated server, a rolling capture of just ARP and ICMP on the VLAN interface, which is cheap enough to leave running:

tcpdump -i enp41s0.4000 -nn -s 128 -w /var/log/vswitch-%Y%m%dT%H%M%SZ.pcap \
  -G 3600 'arp or icmp'

Note: -G rotates hourly, but combining it with -W does not recycle the files - tcpdump exits after the last one. Delete old files separately, for example with find ... -mmin +4320 -delete from a timer.

Capture on the cloud server too. The pair of captures is what makes the diagnosis possible, because it shows which direction is lost.

In our case the captures showed, for every incident:

• Packets from the dedicated server arrived at the Cloud server normally
• Packets to the dedicated server never arrived, although the Cloud server sent them
• No interface errors, drops or overruns on either host

Lost in one direction only, at the fabric, with both hosts healthy.

Step 2 - Look at the ARP on the VLAN

This is where the cause becomes visible. On a quiet vSwitch VLAN there is very little ARP, so you can read all of it:

tcpdump -nne -tttt -r /var/log/vswitch-20261002T052640Z.pcap arp

Two different routers ask for the dedicated server's address, on two different cadences, and they alternate in long runs - only one is active at a time:

05:05:51.390063 f2:aa:bb:cc:dd:01 > ff:ff:ff:ff:ff:ff, ARP, Request who-has 10.0.1.2 tell 10.0.1.1
05:05:51.390073 aa:bb:cc:dd:ee:ff > f2:aa:bb:cc:dd:01, ARP, Reply 10.0.1.2 is-at aa:bb:cc:dd:ee:ff
...
05:43:38.500922 f2:aa:bb:cc:dd:02 > ff:ff:ff:ff:ff:ff, ARP, Request who-has 10.0.1.2 tell 0.0.0.0
05:43:38.500927 aa:bb:cc:dd:ee:ff > f2:aa:bb:cc:dd:02, ARP, Reply 10.0.1.2 is-at aa:bb:cc:dd:ee:ff

Note the second one: sender IP 0.0.0.0. That is an RFC 5227 style probe. It comes from an access router that has no IP address of its own and participates in ARP learning passively, to reduce load on the router that does have one. Some operating systems ignore requests with a 0.0.0.0 sender; Linux answers them, as above.

Now the part that matters. Measure the interval between the first router's requests across a few days:

tcpdump -nne -tttt -r *.pcap arp | grep 'who-has 10.0.1.2 tell 10.0.1.1'

In 72 hours of capture, every interval was 1504.2 s give or take a few tenths of a second - except around each outage, where the request arrived late and was then repeated several times in quick succession:

05:05:51.390063   on cadence
05:30:55          next request due here - it did not arrive
05:31:06.439501   11.4 s late     answered in 11 us
05:31:07.121447   repeat          answered in 11 us
05:31:07.723512   repeat          answered in 11 us
05:31:08.327040   repeat          answered in 11 us

Inbound traffic was zero from 05:31:07 until 05:31:09, and resumed right after the last reply. During a longer incident the same pattern appeared with 84 requests over 114 seconds, every one answered within 15 microseconds.

So the host is answering promptly and the router is asking repeatedly. The outage lasts for as long as the binding is missing.

Step 3 - Why the host never asks on its own

Here is the detail that makes this hard to spot. Across the same 72 hours, the dedicated server sent no ARP request for its own gateway at all, except during the incidents.

That is not a misconfiguration. It is how Linux neighbour discovery works: a received packet that arrived via a next hop confirms that neighbour, so no solicitation is needed. With steady traffic the entry simply never leaves REACHABLE:

root@server:~# ip -s -s neigh show 10.0.1.1
10.0.1.1 dev enp41s0.4000 lladdr f2:aa:bb:cc:dd:01  ref 1 used 339876/0/339876probes 1 REACHABLE

The three figures after used are milliseconds since the entry was last used, confirmed and updated. confirmed is 0: it is being confirmed continuously by traffic. All the relevant sysctls are at their kernel defaults (base_reachable_time_ms = 30000, gc_stale_time = 60).

On an ordinary layer-2 network that is fine. On this fabric it is not, because the router relearns a binding only when traffic arrives for the address and then distributes it to the access routers by BGP. A host that never sends ARP gives the passive learners nothing to learn from.

Step 4 - Send a periodic ARP request

The fix is to revalidate the gateway on a schedule instead of relying on traffic confirmation. Install iputils-arping:

apt install -y iputils-arping

Check it works. The request must be a broadcast carrying your own address as the sender, which is what arping sends by default:

root@server:~# arping -c 1 -w 2 -I enp41s0.4000 10.0.1.1
ARPING 10.0.1.1 from 10.0.1.2 enp41s0.4000
Unicast reply from 10.0.1.1 [F2:AA:BB:CC:DD:01]  0.953ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

The arping from iputils is meant here. There is an unrelated program of the same name with different options; if yours does not print ARPING ... from as above, check which package provides it.

Create /usr/local/sbin/vswitch-arp-keepalive:

#!/usr/bin/env bash
set -u
IFACE="${IFACE:-enp41s0.4000}"
TARGET="${TARGET:-10.0.1.1}"
INTERVAL="${INTERVAL:-300}"

missed=0
while :; do
  started=$(date +%s)
  if arping -q -I "$IFACE" -c 1 -w 1 "$TARGET" >/dev/null 2>&1; then
    [ "$missed" -gt 0 ] && logger -t vswitch-arp "reply from $TARGET again after $missed missed"
    missed=0
  else
    missed=$((missed + 1))
    [ "$missed" -eq 1 ] && logger -t vswitch-arp "no ARP reply from $TARGET"
  fi
  left=$(( INTERVAL - ($(date +%s) - started) ))
  [ "$left" -gt 0 ] && sleep "$left"
done

chmod 755 /usr/local/sbin/vswitch-arp-keepalive

And /etc/systemd/system/vswitch-arp-keepalive.service:

[Unit]
Description=Periodic ARP request to the vSwitch gateway
After=network-online.target
Wants=network-online.target
ConditionPathExists=/usr/bin/arping

[Service]
Type=simple
ExecStart=/usr/local/sbin/vswitch-arp-keepalive
Restart=always
RestartSec=5
Nice=10

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now vswitch-arp-keepalive

300 seconds is a reasonable interval and there is no need to go lower. Resist the temptation to send one every second: it works, but it hides the problem instead of showing you whether it is gone, and it fills your captures.

The script logs only transitions, so an unanswered gateway ARP shows up in the journal. That is worth alerting on in its own right - during one 389-second outage the host sent 199 requests for the gateway and got 3 replies, which is a much clearer signal than "some pings failed".

Step 5 - Verify

Check that the requests are going out on cadence and being answered:

journalctl -t vswitch-arp --since -1d           # should be silent
TZ=UTC tcpdump -nne -tttt -r /var/log/vswitch-20261003T0926Z.pcap arp

Set TZ=UTC on the read as well as on the capture. tcpdump formats timestamps in local time, so on a host set to anything other than UTC the packet times will not line up with the timestamps in a support ticket.

09:31:11.388869 aa:bb:cc:dd:ee:ff > ff:ff:ff:ff:ff:ff, ARP, Request who-has 10.0.1.1 tell 10.0.1.2
09:31:11.389278 f2:aa:bb:cc:dd:01 > aa:bb:cc:dd:ee:ff, ARP, Reply 10.0.1.1 is-at f2:aa:bb:cc:dd:01
09:36:11.406827 aa:bb:cc:dd:ee:ff > ff:ff:ff:ff:ff:ff, ARP, Request who-has 10.0.1.1 tell 10.0.1.2

Note when grepping these yourself: tcpdump prints the broadcast target MAC between the address and tell, so the real line reads Request who-has 10.0.1.1 (ff:ff:ff:ff:ff:ff) tell 10.0.1.2 and a pattern of who-has 10.0.1.1 tell matches nothing.

One further sign that the keepalive is doing what it should: the routers stop asking. In our capture the last request from either of them was 110 seconds before the first keepalive request, and in the three days of captures we still hold neither asked for the address again, where previously one of them always did every 12 or 25 minutes. The broadcast requests keep the binding alive on their own.

Over those three days the count is 873 requests, 873 replies, and no other ARP on the segment at all - the keepalive and its answers are the only ARP left. Replies came back in a median of 0.40 ms, with the slowest at 48 ms. The last outage was on 2 October, well before that window opened.

About the observation window: the longest quiet period before the keepalive was 4 days 19 hours, which is why a couple of quiet days would not have meant much. The run after it reached 9 days 2 hours, nearly twice that, and in all of it not one gateway ARP request went unanswered. That is one host on one vSwitch rather than a controlled experiment, so leave the monitoring from Step 1 in place and judge it on your own traffic.

Not every vSwitch outage is this one

That 9-day run ended, and it is worth saying how, because the difference is what tells you whether this fix is the right one. Two short episodes followed, of about a minute and half a minute, each a cluster of three-to-five-second flaps rather than one clean blackhole. They differ from the fault above in three ways that matter:

• Both directions failed at once. The dedicated server lost the cloud servers and they lost it within the same second. The fault this tutorial is about is one-directional - traffic towards the dedicated server is discarded while its own traffic leaves normally - and that asymmetry is the whole diagnosis.
• Traffic between two cloud servers was lost too, over the Private Network, which never touches the vSwitch. No MAC/IP binding on the vSwitch can explain that.
• ARP stayed healthy. The keepalive's request 22 seconds before one episode was answered in 352 µs, and the next one after it in 315 µs. The binding was never in question.

So these were something else - a general network event rather than an expired binding - and the keepalive neither prevents nor is implicated in them. If your own outages look like that, the ARP cadence in Step 2 will be clean and this is not your fix. The monitoring is what lets you tell the two apart, which is the real reason to leave it running.

A note on synchronous replication

If you run a database with synchronous replication across the vSwitch, these outages cost more than their own length, and they do so in a way that is easy to misread. We measured this with netem in a lab, not on the vSwitch itself, so treat the numbers as indicative of the mechanism rather than as measurements of the fabric.

The loss here is one-way: traffic towards the dedicated server is discarded, traffic away from it is not. A primary replicating to a standby across that path therefore keeps receiving the standby's status updates while the standby receives no WAL at all. The connection looks healthy from both sides, so wal_sender_timeout never fires, nothing resets the socket, and the WAL the primary is trying to send is retransmitted under exponential RTO backoff. Commits waiting on that standby stall for the outage plus the accumulated backoff:

The benign-looking failure is the worse one: a clean partition self-corrects when wal_sender_timeout fires, while one-way loss has nothing to trip and grows faster than the outage it follows.

The lever is tcp_user_timeout on the replication connection, which is 0 by default - nothing bounds how long the kernel will keep retransmitting. Setting it to 5000 ms cut the same stall from 26.6 s to 11.3 s, with the commit returning 12 s before connectivity was restored. Two caveats: it is context=user, so an existing walsender keeps the old value until it is recycled; and if a cluster manager owns synchronous_standby_names, the stall is bounded by tcp_user_timeout plus that manager's polling interval, not by tcp_user_timeout alone. It also trades one long stall for a tail of shorter ones while the standby rejoins.

None of this replaces the keepalive above - it limits the damage of an outage rather than preventing one. But it is worth knowing that an ARP problem on the vSwitch can surface as a database write that hangs for half a minute, with nothing in the Postgres logs to connect the two.

Conclusion

A Linux host with steady traffic does not solicit its gateway, because neighbour discovery confirms the entry from received packets instead. On a vSwitch that is a problem, because the fabric learns the host's MAC/IP binding from ARP it sees on the segment, and will discard traffic towards an address whose binding has expired until it is relearned. Sending one broadcast ARP request every 300 seconds gives the fabric something to learn from and removes the outages.

Worth knowing if you are debugging something similar:

• Capture on both ends - the direction of loss is the whole diagnosis
• Read the ARP, not just the ICMP, and measure the intervals between requests; a cadence that is accurate to tenths of a second for days makes the one late request obvious
• Answering the access router's 0.0.0.0 probes is not sufficient. We answered 101 of 101 of them, in 4 to 15 microseconds, and still had the outages. The broadcast request appears to be the operative difference
• An unanswered ARP request to your own gateway is a precise alert condition, and much earlier than application errors
• A vSwitch can also lose traffic for reasons that have nothing to do with ARP. Loss in both directions, or loss between two cloud servers, rules this fault out before you start reading captures

Thanks to Hetzner support for explaining the EVPN/VXLAN behaviour behind this, which is what made the fix obvious rather than lucky.

License: MIT