Exploring a resilient infrastructure architecture using VMware ESXi, Ubuntu Server, and Keepalived (VRRP). This hands-on lab focuses on designing an active-passive server configuration with a shared Virtual IP (VIP), automated failure detection, and seamless service failover. The goal is to understand how enterprise environments minimize downtime, eliminate single points of failure at the application server level, and maintain service availability during unexpected outages.

Some engineering problems involve data arriving on the wire several times a second. Often, the machine discards this data before your code can process it. This aricle discusses one such scenario. It involves a router announcing itself and a gateway that needed to listen without responding. I will explain the two places in the Linux receive path where the system was discarding these announcements. ## The setting Our projects gateway is a hardware unit installed on ships. It can sit in-line on the vessel’s network. In this mode, the gateway takes over the address that the devices on a VLAN use as their default gateway, by convention “.1”. The ship’s existing router then moves to “.3”. The two devices share the “.1” address using the Virtual Router Redundancy Protocol (VRRP). The gateway acts as the master with a priority of 200, while the router acts as the backup with a priority of 100. If the gateway stops advertising, the router takes the “.1” address back in under two seconds. The router is the fallback that makes it acceptable to put a new box in the path of a bridge network at all. This arrangement has a simple precondition that is easy to violate: the router must speak VRRP. If the router lacks this and uses .1 as a static address, two machines may answer for the same address. This is not just a degraded mode; it is an outage, on exactly the kind of network where an outage is least welcome. Therefore, the gateway must not claim .1 until it hears a VRRP advertisement. Until then, the gateway observes the zone passively. Nothing on the network segment should change. While the requirement is simple to state, implementing it took more time. ## Why the obvious listener hears nothing A VRRP advertisement is a small IP packet with protocol number 112. It is sent to the multicast address 224.0.0.18 with a time-to-live of 255. The gateway already runs an nftables table with an input chain for packets addressed to itself, so the obvious listener is one rule in that chain: match protocol 112 from the router’s .3, and record the source. However, this rule will never match, because the packets are dropped before they reach it. Two filters drop these packets before they reach the input chain, and both exist for good reasons. The first is the network interface’s multicast filter. When a program joins a group, the kernel adds the group’s Ethernet address to the interface’s multicast list, and the driver programs that list into the card; the kernel joins the all-hosts group 224.0.0.1 on its own. A card that filters by that list, as network cards commonly do, discards frames for any other group before the kernel sees them. The Ethernet address for 224.0.0.18 is 01:00:5e:00:00:12, and until something on the host joins that group, it is not on the list. Whether a given card filters, and how, depends on the card and its driver; the kernel itself does not drop the frame at this point. The second filter is the IPv4 routing decision. When a multicast packet reaches the IP layer, the kernel checks if the receiving interface has joined the destination group. If it has not, and the host is not a multicast router, the routing lookup fails and the packet is dropped. In netfilter’s terms, this check sits between the prerouting hook and the input hook: prerouting sees the packet, the routing decision discards it, and input never does. Since 224.0.0.18 is a link-local group, the packet is never forwarded either, so the forward chain does not see it. Both filters are open when keepalived, the VRRP implementation the gateway uses, is running, because keepalived joins the VRRP group, 224.0.0.18 by default, on each interface it serves. This creates a trap. The program that would make the advertisements visible is the program that claims .1. A keepalived instance at priority 200 that hears the router at 100 takes .1 after its preemption delay; one that hears nothing takes .1 after about three advertisement intervals. Starting the speaker in order to listen is starting the speaker. The only safe listener is one that joins nothing and claims nothing. ## The alternatives that were set aside We considered several well-known methods to capture packets that the stack would normally drop. * One option is a raw IP socket for protocol 112. Because a raw socket only receives traffic that survives routing, it must join the group. This method works and is how a VRRP implementation listens. However, it requires the CAP_NET_RAW capability for a process that currently only holds CAP_NET_ADMIN. It also requires a new parser for VRRP packets. * Another option is a packet socket, which observes frames below the IP layer. This also requires the CAP_NET_RAW capability. Furthermore, it would force the process that enforces zones to inspect every frame on the segment just to find one specific packet type. * Finally, we considered a duplicate-address probe by sending an ARP request for .1. This answers a different question: it confirms if .1 is currently in use, but it does not confirm if the router speaks VRRP. VRRP support is the specific property required for safe failover. Each of these options introduces a new privilege, a new dependency, or evidence that does not fully meet our requirements. ## The answer: a set in prerouting, and an interface that listens The gateway already had a mechanism to record packets from the kernel’s packet path without using user-space code. This mechanism uses nftables dynamic sets, which is the same method used for flow evidence (part 1 of this series). The primary challenge was determining where to place the rule to capture packets before they were discarded. The solution is the prerouting chain, which executes before the routing decision occurs: set vrrp-heard { type ipv4_addr flags dynamic, timeout timeout 1h size 1024 } chain prerouting { type filter hook prerouting priority raw; policy accept; iifname "vlan10" ip saddr 10.10.10.3 ip protocol vrrp add @vrrp-heard { ip saddr } iifname "vlan20" ip saddr 10.10.20.3 ip protocol vrrp add @vrrp-heard { ip saddr } ... } (The renderer writes the set’s declaration on one line; it is spread out here.) We use one rule per routed interface. Each rule matches the interface, the router’s .3 address on that subnet, and protocol 112. These rules add the source to a set. They appear at the start of the chain so no later rule can end the packet’s path before they see it. They have no verdict, so the packet continues normally and is dropped by the routing decision. Readers of part 1 will recognise the shape: an “add” action never shares a rule with a verdict, and here there is no verdict to share it with. This handles the second filter. To solve the first filter, we configure each interface to accept all multicast frames. On the gateway, every interface in a routed zone is set to AllMulticast=yes. Other interfaces, observed or silent, remain at AllMulticast=no. A test of the network configuration script checks both. We apply this to the VLAN subinterfaces, not the trunk. The kernel’s VLAN driver passes this flag to the physical card. The frames reach the IP layer, the prerouting chain records the router, and the routing decision drops the packet. The gateway’s network process reads the set using nft -j, as it reads the flow sets. It records every zone where a router has been heard. This information is saved in a store that outlives both the one-hour set timeout and the process itself. Keepalived is configured only for zones where the router has been heard. A zone whose router has not been heard has no VRRP instance at all, shows a “waiting-for-router” status and is observed passively. The process already held CAP_NET_ADMIN to manage the nftables table. The listener needed no new capability, no raw socket and no new crate. The code used to read the set is the same code used to read flow sets. ## Testing it against a real kernel A rendered rule can be checked as text. However, verifying if the kernel inserts the correct addresses into the set is a separate issue. A second test performs this check within a network namespace. This test creates a pair of virtual interfaces and assigns three addresses to the far end. It then loads the rendered table and sends one VRRP advertisement to 224.0.0.18 from each address. These include the zone’s router at 10.10.10.3, another host on the same subnet at 10.10.10.9, and an address from a different zone’s router. Finally, the test lists the set. Only 10.10.10.3 may be in it, and only 10.10.10.3 is. Since nothing in the namespace joins the group, the test confirms that prerouting records a packet that no group membership admits. This process does not test a card’s filter, as a virtual Ethernet pair does not filter multicast. That part belongs to the gateway’s own network cards, and is to be checked on the shipboard hardware: advertisements arriving on a VLAN subinterface, with the setting switched off and then on. The end-to-end behavior was tested in the lab on 2026-10-25. When VRRP was disabled on the router and .1 was configured statically, the gateway rendered no VRRP instance, sent no advertisements, and made no claims. The zone remained in observation. When VRRP was re-enabled, the gateway heard the router and took the .1 address at priority 200, in-line, 226 seconds later. ## Two things the review found The first issue involved timing. The network process publishes a list of routers heard to a file. A root-owned helper reads this file to start or stop keepalived. After a restart of the network process, the helper briefly acted on the list written by the previous process. This caused VRRP speakers to start for about eight seconds before the new list replaced it. The speakers held in a fault state and claimed nothing, but a list that is only correct for the process that wrote it should say which process that was. Now, the file’s first line is the writing process’s invocation identifier. The helper acts on the list only when that identifier is the running process’s. Consequently, a restarted or stopped network process starts and stops no speaker. The second issue involved trust. The rule accepts any VRRP packet on the right interface from the right source address, and source addresses can be forged. A host on the segment could send one advertisement that appears to come from .3, and the gateway would then claim .1 beside a router holding it statically, which is the situation the whole mechanism exists to prevent. We accepted this as a residual risk and recorded it in the threat model, because a host on the segment that can forge a source address can already take .1 by answering ARP for it. The listener does not make that host more capable; it also does not make the gateway immune to it, and the documentation says so. ## What it costs * When an interface has AllMulticast=yes, it accepts every multicast frame on its segment. It then passes these frames to the kernel. The kernel drops most of them during the routing decision, just as it always has. On a segment with many discovery protocols, this creates extra work for the gateway’s processor. We have not measured the exact impact. * Heard once is remembered. If the router later stops advertising, the gateway does not return to a waiting state. Instead, it holds .1 at a priority below the router’s and raises an alert. This is intentional. A router that once spoke VRRP and has gone quiet is a fault to report, not a configuration to accommodate. Consequently, the listener answers the question once per zone rather than continuously. * The rule matches the source, interface, and protocol. It does not parse the advertisement. Therefore, it does not check the virtual router identifier or the advertised address. ## Lessons The Linux receive path teaches a specific lesson. A packet addressed to a multicast group that nothing on the host has joined is discarded twice before an input rule can see it. The interface filter drops it first, and the routing decision drops it second. A listener that must not join the group has to open the first filter explicitly and sit before the second. There is also a broader lesson about where to listen. When the tool that would let you observe something also changes it, the useful question is not how to make that tool quieter, but where in the system the thing is already visible to something that changes nothing. Here that was a hook that runs before the decision to drop the packet, and a set that already existed for a different purpose. The best addition to the gateway’s privileges, in this case, was none. ## References * RFC 5798, “Virtual Router Redundancy Protocol (VRRP) Version 3 for IPv4 and IPv6”: advertisements to `224.0.0.18`, protocol 112, TTL 255. * Linux kernel, `net/ipv4/ip_input.c`: the `prerouting` hook, then the input route lookup, then local delivery and the `input` hook. * Linux kernel, `net/ipv4/route.c`: the input route lookup for multicast destinations and the group membership check (`ip_check_mc_rcu`). * Linux kernel, `net/ipv4/igmp.c`: joining a group adds its Ethernet address to the interface’s multicast list (`ip_mc_filter_add`). * Linux kernel, `net/8021q/vlan_dev.c`: a VLAN device passes the all-multicast flag to its real device (`vlan_dev_change_rx_flags`). * keepalived, `keepalived/vrrp/vrrp_if.c`: the VRRP read socket joins the group (`if_join_vrrp_group`). * nftables wiki, “Netfilter hooks”, https://wiki.nftables.org/: the order of the `prerouting`, `input` and `forward` hooks around the routing decision. * systemd.network(5), `AllMulticast=`.

A complete guide to configuring BGP IP failover using BIRD 2 on Linux dedicated servers for true High Availability without Layer 2 restrictions.

Watch HariTortuga's VRChat clip titled "Wizards looking to relieve himself" on Twitch.

Design high availability routing. Share Virtual IPs using Keepalived and VRRP, and balance client traffic using round-robin DNS.

著者：飛鳥 創(著)　個人出版　2026/9/6(日)配信

ALT: a man in a yellow shirt is sitting in a chair holding a cup and pointing