In my earlier post about ELK over TLS, I left syslog as plaintext over UDP. I’ve since been looking into how to use TLS to encrypt the TCP transport for syslog, and mutual authentication using certificates as laid out in RFC 5425. I’m not really interested in using DTLS.

I just happen to be using the same VM as the CA and the rsyslog server, but the CA server should be a totally separate machine, probably even air-gapped; the client is a different VM.

A lot of this has come from the rsyslog tutorial; although I prefer the basic/legacy syntax to advanced/RainerScript.

Red Hat 8 CA:

So we install GnuTLS and its rsyslog driver, this also provides certtool to generate keys, you could equally use OpenSSL:

dnf install rsyslog-gnutls gnutls-utils

mkdir /root/rsyslog

certtool --generate-privkey --outfile /root/rsyslog/ca_key.pem
certtool --generate-self-signed --load-privkey /root/rsyslog/ca_key.pem --outfile /root/rsyslog/ca.pem

The CA’s public key ca.pem should be sent to the rsyslog clients and server, the private key ca_key.pem should be guarded like the crown jewels!

Once we receive Certificate Signing Requests from the server and clients (see below) we sign them and send back a signed certificate:

certtool --generate-certificate --load-request /root/rsyslog/server_csr.pem \
    --outfile /root/rsyslog/server_cert.pem --load-ca-certificate /root/rsyslog/ca.pem --load-ca-privkey /root/rsyslog/ca_key.pem
    
certtool --generate-certificate --load-request /root/rsyslog/client_csr.pem \
    --outfile /root/rsyslog/client_cert.pem --load-ca-certificate /root/rsyslog/ca.pem --load-ca-privkey /root/rsyslog/ca_key.pem

Red Hat 8 syslog server:

We install the same packages and generate a private key and CSR to send to the CA:

dnf install rsyslog-gnutls gnutls-utils

mkdir /root/rsyslog
mkdir -p /etc/pki/rsyslog/

certtool --generate-privkey --outfile /root/rsyslog/server_key.pem --bits 2048
certtool --generate-request --load-privkey /root/rsyslog/server_key.pem --outfile /root/rsyslog/server_csr.pem

We configure rsyslog to use the public/private keypair and the CA’s key. We’re using CN/SAN verification, so the certificate has to have a proper FQDN, at least for example elastic.local, not just a hostname or IP. The server doesn’t need to be able to resolve the client’s hostnames, so DNS is not required.

The ciphersuite by default is pretty good so I’ve not really looked into how to change that. We’re forcing the use of TLS (1.2/1.3 only by default) and using the standard tcp/6514 port. Our /etc/rsyslog.d/11-tls.conf looks like this:

# make gtls driver the default
$DefaultNetstreamDriver gtls

# certificate files
$DefaultNetstreamDriverCAFile /etc/pki/rsyslog/ca.pem
$DefaultNetstreamDriverCertFile /etc/pki/rsyslog/server_cert.pem
$DefaultNetstreamDriverKeyFile /etc/pki/rsyslog/server_key.pem

# tcp listener
$ModLoad imtcp
$InputTCPServerStreamDriverAuthMode x509/name
$InputTCPServerStreamDriverMode 1
$InputTCPServerRun 6514
$InputTCPServerStreamDriverPermittedPeer *.local 

# write to file
if $fromhost-ip startswith '192.168.1.' then /var/log/rsyslog.log

If the hostname in the client certificate does not match the *.local peer wildcard, the server will refuse the connection; again note it is not checking that the client IP resolves to that hostname.

SUSE 12 syslog client:

I thought I’d just use a different client OS to demonstrate the subtle differences, but its much the same as the RHEL server. The clients need to be able to resolve the server’s hostname, so a simple /etc/hosts entry will suffice, again no need for DNS.

zypper install rsyslog-module-gtls gnutls

echo '192.168.1.3 elastic elastic.local' >> /etc/hosts

mkdir /root/rsyslog
mkdir -p /etc/pki/rsyslog/

certtool --generate-privkey --outfile /root/rsyslog/client_key.pem --bits 2048
certtool --generate-request --load-privkey /root/rsyslog/client_key.pem --outfile /root/rsyslog/client_csr.pem

Our /etc/rsyslog.d/tls.conf looks like this:

# make gtls driver the default
$DefaultNetstreamDriver gtls

# certificate files
$DefaultNetstreamDriverCAFile /etc/pki/rsyslog/ca.pem
$DefaultNetstreamDriverCertFile /etc/pki/rsyslog/client_cert.pem
$DefaultNetstreamDriverKeyFile /etc/pki/rsyslog/client_key.pem

# gtls network stream driver
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverPermittedPeer elastic.local

# send everything
*.* @@192.168.1.3:6514

In the final line we’re telling the client machine’s rsyslog setup to send all syslog to the remote server - note we can use IP’s here, as the certificate authentication is done by the stream driver separately.

If the client certificate isn’t signed by the same CA as the server, then the server will refuse the connection. The client will only send to the elastic.local peer.