Software Used:

• FreeBSD 14.2 (jail)

Introduction

If you can't observe what is happening then you can't find and fix what is broken

Setting up a logging server at this point may seem premature. But as we move forward, things will get more complicated and more interdependent. Having a logging server that can handle logs from all services will aide is troubleshooting what is going wrong.

Logging Discussion

What is it we want to get out of the logging server? At this point we just need a place to store logs from other systems. There are many logging servers that have nice graphical interfaces. We may get to self hosting one of those in the future, but for now we want a simple, syslog like logging server. Each log should be timestamped with the time it arrived and the hostname of the server sending the log. If no server hostname is available, we should us the IP address. In addition the syslog server should support TLS encrypted logs.

Update DNS with new address

To begin with we will need to be able to resolve the new log server. To do this update the example.org and 30.168.192.in-addr.arpa zone files on the DNS servers respectively.

logger     IN      A      192.168.30.12

12      IN      PTR      logger.example.org.

Don't forget to update the zone serial number and restart the named service.

Setting up the syslog jail

The syslog server will be created using a FreeBSD jail. If you have not already setup the Jail infrastructure review Episode 3 - Setting up FreeBSD Jail. For this jail we will be using the following file.

# Log server jail file

logger {
	# STARTUP / LOGGING
	exec.start = "/bin/sh /etc/rc";
	exec.stop = "/bin/sh /etc/rc.shutdown";
	exec.consolelog = "/var/log/jail_console_${name}.log";

	# PERMISSIONS
	allow.raw_sockets;
	exec.clean;
	mount.devfs;
	allow.reserved_ports = true;


	# HOSTNAME / PATH
	host.hostname = "${name}";
	path = "/usr/local/jails/containers/${name}";

	# NETWORK
	vnet;
	vnet.interface = "${epair}b";

	# NETWORKS/INTERFACES
	$id = "12";
	$ip = "192.168.10.${id}";
	$mask = "255.255.255.0";
	$gateway = "192.168.10.1";
	$bridge = "net-dmz";
	$epair = "epair${id}";

	exec.prestart += "ifconfig ${epair} create up";
	exec.prestart += "ifconfig ${epair}a up descr jail:${name}";
	exec.prestart += "ifconfig ${bridge} addm ${epair}a up";
	exec.start    += "ifconfig ${epair}b ${ip} netmask ${mask} up";
	exec.start    += "route add default ${gateway}";
	exec.poststop += "ifconfig ${bridge} deletem ${epair}a";
	exec.poststop += "ifconfig ${epair}a destroy";
}

Start the log server jail

service jail start logger

Configure the Log server

First we will create the /usr/local/etc/syslog.d directory if not already created.

[ -f /usr/local/etc/syslog.d ] || mkdir -p /usr/local/etc/syslog.d

Next create the directory to keep the logs in

[ -f /var/log/remote ] || mkdir -p /var/log/remote

Next create the config file for receiving the remote logs. The config file will be kept in the /usr/local/etc/syslog.d directory

# Send any remote logs to a the remote file
:hostname, startswith, "192.168"
*.*                               /var/log/remote/remote.log

Finally we will update the syslogd flags

sysrc syslogd_flags="-n -O rcf3164 "

Now we do not want the log file growing infinitely, so we need to rotate the log every so often. Everyday seems right. Let's also keep a week of old logs. Adjust the time frame and retention to suit your needs.

First we create a directory for our newsyslog config file.

[ -f /usr/local/etc/newsyslog.conf.d ] || mkdir -p /usr/local/etc/newsyslog.conf.d

Finally we create the config file. The newsyslog command gets run every hour so we do not need to make any additional changes.

/var/log/remote/remote.log     640 7 * $D0 CNTZ

Allocate the certificate

Review Episode 4 section Deploy Root cert and pull new cert to add a certificate to the logging server.

Enable and start the log server

Finally let's start the log server. Since syslog may already be started by default jail we will call the restart command to restart if necessary.

Then we will create the config file

service syslogd restart

Update Logging in existing servers

Firewall

First login to the firewall and su(1) to root. Update the /etc/syslog.conf file and add the following line to the bottom of the file.

# Send log messages to log server
*.*                                              @192.168.10.12

Finally, restart the syslogd service

rcctl restart syslogd

DNS/DHCP server

First, login to the DNS/DHCP server and su(1) to root. Create the /usr/local/etc/syslog.d directory if it doesn't exist.

[ -f /usr/local/etc/syslog.d ] || mkdir -p /usr/local/etc/syslog.d

Next create the syslog config file

Send log messages to log server
*.*                                          @192.168.10.12

Finally restart syslogd

service syslogd restart

Certificate Authority server

First, login to the CA server and su(1) to root. Create the /usr/local/etc/syslog.d directory if it doesn't exist.

[ -f /usr/local/etc/syslog.d ] || mkdir -p /usr/local/etc/syslog.d

Next create the syslog config file.

# Send log messages to log server
*.*                                          @192.168.10.12

Finally restart syslogd

service syslogd restart

Conclusion

Now there is an issue with something, there is only one place to go to review the logs for all servers.