Unix Tutorial Projects: Centralised RSyslog

RSyslog RSyslog

Think I have enough time writing about ongoing efforts, learnings and projects of mine here on Unix Tutorial, so it makes perfect sense to introduce another regular thing: Unix Tutorial Projects.

Usually these are small enough things to fit into a calendar week, and at the end of each project I’ll be publishing a post like today’s with expectations and achievements thus far.

Centralised RSyslog

What do I mean by that? It’s going to be a server (initially – one of the Raspberry Pi systems in my home office) for collecting logs from the rest of my home office, automation, Unix Tutorial group and hosting/troubleshooting systems.

Main benefits of centralising logs are:

  • easier and quicker access to logs – no need to log into each system
  • greater flexibility reviewing logs – sometimes a quick grep across the same log on all the systems will reveal far more than looking at each individual log
  • better understanding of fundamental issues – if something is too noisy with logs, this is usually a good thing: there are issues that need to be fixed, rather than alers to be muted or individual servers to be ignored
  • easier analysis – you can forward all the logs into ELK (Elastic, Logstash, Kibana) setup to gain instant visualisation, etc
  • robust incident response – sometimes issues are so sever that your servers go offline or disks fail or data gets lost. Having a copy of all the important logs stored elsewhere (on your centalised logs server) will be a great tool for incident response

Now, I’m also a DataDog user myself so I realise this mini-project is a reinvention of a whell in some ways, but it seems like both a fun way to spend a few geeky evenings and a great way to improve my RSyslog understanding which I need for other projects.

My Plan for Centalised RSyslog Setup

I’m doing the following so far, let me know if you think I’m missing a step or two:

  • attach a USB stick to my Raspberry Pi for logs storage – always good to have logs on a separate filesystem, so if it gets 100% full you don’t impact the root filesystem
  • install rsyslog from standard Raspbian repos
  • open external port on my Internet router and allow RSyslog traffic on it
  • setup self-signed CA authority and generate keys/certs for RSyslog clients
  • setup encryption for shipping logs safely (TLS) – wonder if this will be taking a serious toll on my Raspberry Pi’s CPU
  • configure RSyslog to create separate log file for each server
  • update RSyslog on each of the monitored servers to ship logs to my home office log collector
  • log each RSyslog client into separate hostname based file
  • possibly take it even further and filter logs by application or message content
  • setup real-time monitoring of syslog logs with grc colouriser
  • write a bunch of Python scripts for assessing log volumes and predict storage requirements
  • I’ll update this post when I’m done.

See Also

  • Unix Tutorial Projects
  • RSyslog website
  • RSyslog: Separate Log File for Each Host
  • Review Latest Logs with tail and awk
  • Script: Save Your Session Log



Keep Learning

Follow me on Facebook, Twitter or Telegram:
Recommended
I learn with Educative: Educative
IT Consultancy
I'm a principal consultant with Tech Stack Solutions. I help with cloud architectrure, AWS deployments and automated management of Unix/Linux infrastructure. Get in touch!

Recent Tweets