Anycast DNS...

When I decided to begin this article, I thought to my self "this will be quick.  I've setup Anycast DNS many time with commercial products.  How much more work could it be setting this up using opensource tools?" Well I found out how much work!  And that is a story for another article.  But with everything I "learned" I was able to get it working and so share it here, if only to remind myself how I did it.

Setting the stage

Here is an example of setting up Anycast DNS on an OpenBSD system utilizing relayd(8), bgpd(8) and nsd(8).  I assume that you are have already installed the basic OS on the server and are familiar with locating and editing various files on the server.

Configuration

First thing is to determine the Anycast IP address that will be used.  This address is one that will need to be to route within the internal network and not conflict with any internal or external address space.  For this example we'll use 192.0.2.1.   We will make that address an alias (secondary) address on our ethernet interface (re0 in this example).

ifconfig re0 alias 192.0.2.1 netmask 255.255.255.255

# /etc/hostname.if
# main address of device
inet 192.168.1.10 255.255.255.0
# Anycast address
inet alias 192.0.2.1 255.255.255.255

Next we will setup relayd(8).  There are two things we will do that may seem wrong.  First is we will setup our Anycast address on the same interface or our main address and second is we are going to route placing a route from that aliased address to the real address.  This may not make a lot of sense.  It doesn't.  But realize that we only care the we can get a static route into the FIB that bgpd(8) will then redistribute.  The fact that really doesn't do anything turns out not to be important.

# /etc/relayd.conf
# Anycast relayd.conf

# Our real address
DNSHOST="192.168.1.10"
ANYCAST="192.0.2.1/32"

log state changes
# enable this if you want to see howw your script is doing
#log host checks

table <dns_hosts> { $DNSHOST }
router "anycast_dns" {
  # this part adds a static route from the 192.0.2.1 address to
  # the 192.168.1.10 address which is how BGP will pick up the route
  # to advertise.
  route $ANYCAST
  forward to <dns_hosts> check script "/home/scripts/anycast_check.sh"
}

Next we will create the anycast_check.sh script.  This script can be placed anywhere that _relayd can get to it.  It will be executed with the permission of _relayd so anything it utilized in it's test _relayd must be able to run.

#!/bin/sh
# relayd(8) expects exit code of 0 on failure. non-0 on success
DNSHOST="192.168.1.10"
a=$(dig +tries=1 +timeout=1 +short @$DNSHOST hostname.bind CHAOS TXT | wc -l)
exit $a

Next we will configure our nsd(8) config and zone files.  This is a basic nsd.conf(5) config file.  You will defiantly need to modify it to meet your needs.

# nsd.conf
server:
        hide-version: yes
        verbosity: 1
        database: "" # disable database

        # listen on the Anycast IP address
        ip-address: 192.0.2.1

remote-control:
        control-enable: yes
        control-interface: /var/run/nsd.sock

zone:
        name: "example.com"
        zonefile: "zones/master/example.com"

; example.com
; this SOA is waaay out of wack for a real zone
; only used for testing
@ IN SOA ns1.example.com. noemail.example.com. (
  101        ; serial number
  60         ; refresh
  900        ; retry
  120        ; expire
  30         ; ttl
  )
@ IN NS ns1.example.com.
@ IN TXT "example zone"
ns1.example.com. IN A 192.0.2.1

Next we create the bgpd.conf(5) configuration file.  The key to this file is bgpd(8) needs to redistribute static address(es).  If there are some addresses that should not be redistributed you can create a filter or place those interfaces & addresses in a different rdomain(4) (way outside of this article).

# anycast bgpd.conf

# RFC 6996 specifices that ASN 64512 - 65534 are reserved for
# private use.
myas="65003"
myip="192.168.1.10"
# this will be the AS of the router that this system peers with
peeras="65001"
peerip="192.168.1.1"
AS $myas
router-id $myip
fib-update yes
log updates
# This will advertise the anycast route from relayd(8)
network inet static
# define peer(s) to use
# specify what address to peer on to make sure our anycast address
# isn't selected/used
group "peers" {
  remote-as $peeras
  neighbor $peerip
  local-address $myip
}
# we will not need to accept any routes
deny from any
# but we will send our route
allow to any

Finally we enable and start the daemons.

rcctl enable relayd bgpd nsd
rcctl start relayd bgpd nsd

Confirmation and Testing

Confirm that nsd(8) is working (and check the script at the same time)

# /home/scripts/anycast_check.sh || echo "Working"
Working

Confirm that relayd(8) is working.

# relayctl show routers
Id      Type            Name                            Avlblty Status
1       router          anycast_dns                             active
                        route: 192.0.2.1/32

Confirm that the route is in the FIB.  The key is the 4th line (192.0.2.1/32 192.168.1.10 UGS)

# netstat -rnfinet
Routing tables

Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
default            192.168.1.1        UGS       11   111081     -     8 re0  
192.0.2.1          00:11:22:33:44:55  UHLl       1    11033     -     1 re0  
192.0.2.1/32       192.0.2.1          UCn        0        0     -     4 re0  
192.0.2.1/32       192.168.1.10       UGS        0        0     -     8 re0  
127/8              127.0.0.1          UGRS       0        0 32768     8 lo0  
127.0.0.1          127.0.0.1          UHhl       1       16 32768     1 lo0  
192.168.1/24       192.168.1.10       UCn        5   287754     -     4 re0  

Confirm that bgpd(8) sees the route.

# bgpctl show rib
flags: * = Valid, > = Selected, I = via IBGP, A = Announced,
       S = Stale, E = Error
origin validation state: N = not-found, V = valid, ! = invalid
origin: i = IGP, e = EGP, ? = Incomplete

flags ovs destination          gateway          lpref   med aspath origin
AI*>    N 192.0.2.1/32         0.0.0.0           100     0 i

Now confirm that the 192.0.2.1 route disappears when you shut nsd(8) down.

Future refinements

• I have noted that if relayd(8) exists unexpectedly the route it inserted may not be removed.  In that case it is possible that nsd(8) also has a problem and you can end up in a situation where you are blackholing DNS requests.
• The anycast_check.sh script only check to see if the DNS server is running.  Not that any zones are valid or records are being returned.  That script should be modified to do more thorough checks.