HA @ Home

I, like many of you, have been working from home since the pandemic. As a network engineer, I worked from anywhere I needed to as the job demanded. I met with my team and customers over WebEx, Zoom, Meet and later Teams. I was already pandemic ready, so when I had to stay home, it really wasn't a huge change... except for the network.

At work I had build a global, highly redundant, scalable network that always provided best experience no matter what was thrown at it. At home... well, you've heard of the cobblers children? Don't get me wrong, the network worked. I could watch easily watch streaming shows all day long. However, throw streaming plus online gaming, online learning, et cetera and the network was not something I was going to show off.

I am not going to go through all of the changes/upgrades I made in this article. I will cover how I went from a single ISP point of failure to highly available, resilient internet connection.

First step... get another ISP. I do not plan to name the ISP's I ended up using. Depending on where you are in the world, you may only have access to a limited number of ISPs. For me, my original ISP (cable provider) was charging nearly $200 a month. The service wasn't bad, for the most part but this wasn't a business class service. So my first step was to go 5G. Getting that setup was fairly easy. The modem I was provided had the ability for an external antenna so I pick one up as well. That seemed to help some, but not as much as I would have hoped. While the service was good, I did notice that there was a large amount of jitter and the latency increased and the bandwidth increased. So I canceled the cable and ordered a 'fiber to the home ISP' that just happened to be expanding into my area.

Basic Setup

I run OpenBSD as my firewall (big surprise if you have read any of my other articles). Here is my basic setup.

#
# Firewall PF
#
all_nets = "192.168.0.0/16"
all6_nets = "2001:db8::/32"
vpn_subnet = "192.0.2.0/24"
wan_if="em3"
lan_if="em1"

set skip on lo
set skip on enc
set state-defaults pflow

table <inet_doh> persist file "/var/db/pf.internet_doh_servers"
table <internal_dns> persist file "/var/db/pf.internal_dns_servers"
table <banned> persist file "/var/db/pf.banned"
table <sshguard> persist
table <time_blocked> persist

Pretty standard, first specify macros, next setup 'skip' and other global settings, followed up by common tables.

# default block
block log
# block anything in the time_blocked table
block quick from <time_blocked>
# block bad hosts
block quick from <banned>
# capture NTP from any internal sources and respond for them
pass in proto udp from $all_nets to port 123 rdr-to 127.0.0.1 port 123
pass in inet6 proto udp from $all6_nets to port 123 rdr-to ::1 port 123

pass out inet6 proto icmp6 from self to $all6_nets

# pass all and pfsync carp protocol
pass out quick on carp proto carp
pass quick on $lan_if proto pfsync

Next setup the common rules. Default block and log, block time based addresses, quick block of banned addresses. Capture all NTP and respond from ourselves. And finally permit CARP and PFSYNC.

Finally specify my zones.

include "/etc/pf.d/pf.anchor.lan"
include "/etc/pf.d/pf.anchor.wan"

My actual configuration has a few more zones than this, but for this example we'll focus on 2.

# WAN Anchor
# NOTE: (WAN:0) will be different per FW pair
#

anchor "wan" on WAN {
  # block everything in
  block in log
  # match outbound traffic to WAN interfaces
  match out nat-to (WAN:0)
  # pass traffic out
  pass out set queue(DEFAULT, ACK)
  # stop DNS/DOH bypass
  block out quick log proto tcp to <inet_doh> port 443 ! tagged DNS
  block out quick log proto { udp tcp } to port { 53 853 } ! tagged DNS
  # allow ourselves out
  pass out proto { udp tcp } to port { 53 853 } tagged DNS set queue(DNS)
  pass out proto { tcp udp } from self user = "_unbound" set queue(DNS)
  pass out proto { tcp udp } to port { 80, 443, 8443 } set queue(WEB, ACK)
}

Anything tagged as a WAN interface will have this NAT applied. Basically it will block all inbound traffic and NAT all outbound traffic. It will block traffic to known DoH servers but allow my DNS servers out.

For the LAN interface

# PF LAN
anchor "lan" on LAN {
  pass log set queue(LAN_DEF,LAN_ACK)
  pass in inet6 from any to 64:ff9b::/96 af-to inet from (WAN:0) set queue(LAN_DEF,LAN_ACK)
  match in from <internal_dns> tag DNS set queue(LAN_DEF,LAN_ACK)
}

Anything on the LAN is good?! Right. For now let's assume so, so pass it. If traffic comes from my internal DNS servers tag it for the WAN rule processing.

One thing that may not be familiar is the use of the queue() syntax. More on that later when we talk about buffer bloat.

Now with the PF rules in place, just set the groups on the interfaces.

group WAN
inet autoconf
up

group LAN
inet 192.168.1.2/24
inet6 2001:db8::2 64
up

Don't forget to enable IP forwarding or the firewall will not work as expected.

# sysctl net.inet.ip.forwarding=1
# sysctl net.inet6.ip6.forwarding=1

Since we plan on having redundant firewalls to eliminate a device failure we will need to setup carp(4). This will provide a first hop redundancy protocol (FHRP). The setup of carp is very easy.

group LAN
vhid 3
carpdev em1
pass letmein
state master
inet 192.168.1.1/24
up

Match the group of the main interfaces. Set the ID. This much match on the peer router. Tie the carp interface to a physical interfaces and set a password (optional). Set the default state. Once should be 'master' the other 'backup'. Finally set the floating IP and bring the interface up.

By default the sysctl(8) for carp are setup to have carp enabled. To see what the current settings are use the following command

# sysctl net.inet.carp
net.inet.carp.allow=1
net.inet.carp.preempt=0
net.inet.carp.log=2

Buffer Bloat... simply put

Network devices, especially router, have different speed interfaces. Today it is not unusual to have 1G / 10G on a home LAN and less than that for the WAN interface. Sometime an order of magnitude smaller. To compensate for this difference bandwidth, routers use temporary storage, called buffers, so store packets while they wait their turn to be sent on the slower interfaces. Without special configurations, packets are processed in a FIFO (first in, first out) queue. Meaning that if a larger packet is queued before a smaller packet, the smaller packet will need to wait until the larger packet is sent before it gets a turn.

Now let's consider that the smaller packet is an ACK. And that ACK is being sent in response to a on-line game. Do we want that packet waiting for some other non-time sensitive packets... say a web page? Probably not.

Fighting the bloat... Queuing

Fortunately, OpenBSD has an answer ready to go. Since version 6.2 OpenBSD has a queuing mechanism in PF that will help us overcome buffer bloat (called FQ-CoDel for those that are curious). We can specify our upload speed of the main FIFO queue and setup flow based sub-queues for our traffic.

# WAN
queue INET on $wan_if bandwidth 95M max 95M
queue ACK parent INET bandwidth 1M max 10M flows 1024 qlimit 1024
queue DNS parent INET bandwidth 5M flows 1024 qlimit 1024
queue WEB parent INET bandwidth 50M flows 1024 qlimit 1024
queue SSH parent INET bandwidth 20M flows 1024 qlimit 1024
queue GAME parent INET bandwidth 20M flows 1024 qlimit 1024
queue DEFAULT parent INET bandwidth 90M flows 1024 qlimit 1024 default
# LAN
queue LAN on $lan_if bandwidth 1G
queue LAN_ACK parent LAN bandwidth 1G flows 1024 qlimit 1024
queue LAN_DEF parent LAN bandwidth 1G flows 1024 qlimit 1024 default

As mentioned the INET queue has a bandwidth set to match the upload speed of the connection as well as the max bandwidth setting. It is important, for queuing to work correctly that the bandwidth be set around 95% of the available upstream bandwidth.

Next create one or more child queues and set the bandwidth they can should consume. If max is not specified, the queues will be allowed to take more, if available. Here a qlimit and flow limit for each queue is also specified increasing from the default. This is optional. To see if a queue is getting maxed out and dropping traffic you can use the following command. Make notes of any dropped pkts or qlength being larger than 0.

# pfctl -v -sq
queue INET on em3 bandwidth 100M
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
queue ACK parent INET flows 1024 bandwidth 1M qlimit 1024
  [ pkts:   31459517  bytes: 1770765239  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]
queue DNS parent INET flows 1024 bandwidth 5M qlimit 1024
  [ pkts:     465724  bytes:   43885028  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]
queue WEB parent INET flows 1024 bandwidth 50M qlimit 1024
  [ pkts:    2913536  bytes: 2104425594  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]
queue SSH parent INET flows 1024 bandwidth 20M qlimit 1024
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]
queue GAME parent INET flows 1024 bandwidth 20M qlimit 1024
  [ pkts:       5751  bytes:     596945  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]
queue DEFAULT parent INET flows 1024 bandwidth 90M default qlimit 1024
  [ pkts:    1723990  bytes:  633385764  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]
queue LAN on em1 bandwidth 1G
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
queue LAN_ACK parent LAN flows 1024 bandwidth 1G qlimit 1024
  [ pkts:     749181  bytes:   50240090  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]
queue LAN_DEF parent LAN flows 1024 bandwidth 1G default qlimit 1024
  [ pkts: 1229933940  bytes: 320469964876  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]

The systat(1) command can also give you a real time look at the queues in action.

# systat queue

Got one... now repeat for two

Now that we have one firewall up and working we can repeat the process for the second firewall with the second ISP. After that we will need to tie them together.

Since both firewalls are up and functioning, let's make sure that CARP is working as well as PFSYNC. The command

# ifconfig carp

will show all carp and pfsync interfaces and their current state. On one firewall you should see the interface be MASTER and on the other firewall it should state BACKUP. If it states anything else, CARP is not working and this problem will need to be solved first.

Now to test failover of CARP. The easiest way to do this is on the master systems demote the carp interface. To check the current demotion use this command

# ifconfig -g carp
carp: carp demote count 0

Now let's demote the master firewall and check again.

# ifconfig -g carp carpdemote
# ifconfig -g carp
carp: carp demote count 1

Now check the status of the "backup" firewall. We should see the carp interfaces goto MASTER state. Checking the dmesg should also show the CARP interface state transition.

To remove the demotion us the following command

# ifconfig -g carp -carpdemote

If we have not enabled preempt in the sysctls, the MASTER status will not fail back over to the "primary" firewall until the "backup" firewall is demoted.

There may be reasons why we want to keep one firewall/ISP as the "primary" unless it has failed. In those cases make sure the set the net.inet.cap.preempt to 1.

Automatic demotion

Now that we have the firewalls and ISP setup and we've tested performance and failover we need failover to happen automatically. In OpenBSD this is easily accomplished with ifstated(8) and some shell scripts.

Before we get to the config, lets consider what we want to happen. First if the WAN interface is down, we can't pass packet we want the failover to occur. But what happens if the interface is up but he ISP's router isn't available. We should test for the default gateway to be up. Next what happens if the default gateway is up but there is no path to the Internet? Next what happens if there is a path to the Internet but performance is really bad? So here we have 4 tests...

• WAN interface up
• WAN default gateway present
• Internet accessable
• Internet performance ok

We can check all those items with ifstated daemon. Below is a config an helper script to continually test for the "health" of the connection.

# 
dmz_if_up = 'em3.link.up'
dmz_if_down = 'em3.link.down'

external_gw='"ping -q -c 1 -w 1 `route -n show -inet | grep default | awk \'{ print $2 }\'` >/dev/null 2>&1" every 60'

performance='"/root/check_perf.sh X.X.X.X >/dev/null 2>&1" every 60'

# ifstated starts up with the first defined state
state neutral {
        if $dmz_if_down {
                run "logger -st ifstated 'uplink down'"
                set-state demoted
        }
        if ! $external_gw {
                run "logger -st ifstated 'could not reach cable router'"
                set-state demoted
        }
        if ! $performance {
                run "logger -st ifstated 'perforamce degraded'"
                set-state demoted
        }
}

state demoted {
        init {
                run "ifconfig -g carp carpdemote" 
        }
        if $dmz_if_up && $external_gw && $performance {
                run "logger -st ifstated 'cable router ok'"
                # remove our carp demotion
                run "ifconfig -g carp -carpdemote" 
                set-state neutral
        }
}

## commands in the global scope are always run
if carp3.link.up
        run "logger -st ifstated 'carp3 is master'"
if carp3.link.down
        run "logger -st ifstated 'carp3 is backup'"

#!/bin/sh

if [ $# -lt 1 ]; then
        echo "${0##*/} <host>"
        exit 2
fi

LIMIT=${LIMIT:-200}
TRIES=${TRIES:-5}
WAIT=${WAIT:-1}

rtt=`ping -q -c ${TRIES} -w ${WAIT} ${1} 2>&1 | sed -n '/^round-trip/p' | cut -f4 -d" " | cut -f2 -d"/"`

if [ -z "${rtt}" ]; then
  echo "failed to get RTT" >&2
fi
rtt=${rtt%%.*}
if [ $rtt -gt $LIMIT ]; then
  echo "too high"
  exit 1
else
  echo "ok"
fi
exit 0

The first checks dmz_if_up/down check to make sure the WAN interface is up. The next check external_gw verifies that the gateways is pingable every 60 seconds. The final test performance checks for both internet reachability as well as performance. Select and IP or host what we want to ping every 60 seconds. The script below will send 5 pings and calculate the average response time. If any test fails then the performance/reachability is considered "failed" and state is set to demoted. When entering demoted state, the first thing that is ran is the ifconfig demotion of CARP. Once the tests all succeed, the state is change back and the demotion is reversed.

Conclusion

After running in this environment for a while it has worked pretty well. By and large my "home user community" has been generally happy. The environment seem to "do the right thing" when failures occur.

Since traffic is split across 2 different ISP's when a failover occurs, all sessions are dropped and have to be re-established. By and large this is not a big deal. But since the IP addresses on the WAN interfaces are different there are a ton of errors with pfsync making me wonder I should even be worrying about pfsync across these firewalls.