Setup Graylog as central syslog server

10 Dec 2020

I was searching for information about running a local syslog server in my network when I stumbled on Graylog. I’m currently running an ElastiFlow server with dashboards for my netflow and haproxy syslog info. But this way my haproxy server stores all logs it self and using filebeat to send it to my ELK stack. I would like to use a central syslog server to store the log files. Using a central syslog server I can easily set up more servers to log to that server.

Using Docker

I decided to run all of it in Docker containers even though I have a dedicated VM for it. I like the isolation a docker container have and i can keep all the config and data in one folder (with some sub folders). Another thing is that they are not depending on my Host, I can store my docker-compose.yml file on a network disk or git-repo and then just install whatever Linux distro I like at the moment and then just start my service(s).

Installing Docker

Well I have already written a small guide on how to Install Docker and docker-compose on Debian 10 so I will not cover that part here again.

Docker compose

I like to write docker-compose files for my services/servers and Graylog already have one, so why complicate stuff lets start with theirs: (you can find it over at their site)

version: '3'
services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongo:
    image: mongo:4.2
    networks:
      - graylog
   # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.0
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    deploy:
      resources:
        limits:
          memory: 1g
    networks:
      - graylog
  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:4.2
    environment:
      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      # This can be set to an url/domain name if you have a local DNS
      - GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
    networks:
      - graylog
    restart: always
    depends_on:
      - mongo
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 1514:1514
      # Syslog UDP
      - 1514:1514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp
networks:
  graylog:
    driver: bridge

Set server timezone

elasticsearch:
      ...
      environment:
      # Set server timezone
      - TZ=Europe/Stockholm
      - GRAYLOG_TIMEZONE=Europe/Stockholm

Add persistent storage volume for services

Now lets change the docker-compose.yml to use persistent storage. I usually store my server data under /srv including my docker-compose.yml. This way I can move the content of /srv to anothe Linux server and just run docker-compose up -d in /srv folder and it will start my services.

mongo_db:
  ...
  volumes:
    - /srv/mongodb:/data/db
elasticsearch:
  ...
  volumes:
   - /srv/elasticsearch:/usr/share/elasticsearch/data
graylog:
  ...
  volumes:
   - /srv/graylog:/usr/share/graylog/data

Don’t forget to create all folders.

sudo mkdir -p /srv/{mongodb,elasticsearch,graylog}

Local config files

I like to configure my services using local files if possible. This way my docker-compose.yml isn’t cluttered with loads of environment variables.

So lets download and prep the default config files.

mkdir -p /srv/graylog/config
cd /srv/graylog/config
wget https://raw.githubusercontent.com/Graylog2/graylog-docker/4.0/config/graylog.conf
wget https://raw.githubusercontent.com/Graylog2/graylog-docker/4.0/config/log4j2.xml

Change permissions

Graylog is running with UID 1100, I don’t know about GID but I do change both UID and GID to 1100

sudo chown 1100:1100 -R /srv/graylog

Change Graylog admin password

Well the most important thing to change is your own password, that is the root_password_sha2 parameter in the config file.

To change the password you need a sha256 hash of your wanted password, this can be retrieved by piping your password to sha256sum in Linux

echo -n "superhardpassword" | sha256sum
85dee518689f3693440b4ae448d75b5beae363655f6a21e1dcdb164d266984bf  -

Well not entirely true, you get the sha256 hash, spaces and a dash. You only want the contigious part up until the first space. If you want to script this, the ending parts can be removed using cut like this

echo -n "superhardpassword" | sha256sum | cut -d" " -f1
85dee518689f3693440b4ae448d75b5beae363655f6a21e1dcdb164d266984bf

so the final string in the graylog.conf would be this

root_password_sha2 = 85dee518689f3693440b4ae448d75b5beae363655f6a21e1dcdb164d266984bf

Don’t forget to change password_secret to some long random string, the comment in the default graylog.conf recomends at least 64 characters.

I made a script for this:

#!/bin/bash
# graylog_passwd.sh

CONF=/srv/graylog/config/graylog.conf

echo -n "Enter a new password: "
PWD=$(head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1)

echo "root_password_sha2 = $PWD"

sed -i "s/root_password_sha2 = .*/root_password_sha2 = $PWD/" $CONF

SALT=$(tr -dc 'A-Za-z0-9!"#$%&'\''()*+,-./:;<=>?@[\]^_`{|}~' </dev/urandom | head -c 64  ; echo)

echo "password_secret = $SALT"

sed -i "s/password_secret = .*/password_secret = $SALT/" $CONF

Graylog and Elasticsearch error

When I finally started my Graylog services I got errors about Graylog not being able to find Elasticsearch server. I finally solved this by exposing elasticsearch ports and instruct graylog to use my server IP and to use elasticsearch version 7.

graylog:
  ...
  environment:
    ...
    - GRAYLOG_ELASTICSEARCH_HOSTS=http://192.168.1.20:9200
    - GRAYLOG_ELASTICSEARCH_VERSION=7

I could have put this configuration in the graylog.conf, but I decided to add this configuration to the docker-compose file. This will keep my networking settings in one place and the graylog specific configuration in another.

Everything you can configure in graylog.conf you can configure using GRAYLOG_* envrionment variable. Just remember the environment variable is all upper case and the settings variable in graylog.conf is lower case.

Syslog UDP input

Some quick notes to setup Syslog UDP input, this is needed to receive Syslog messages.

First you go to System -> Inputs

At the inputs configuration screen you choose an input from the drop-down list and then press Launch new input

This will bring up the Syslog UDP configuration dialog

Give the input a name and the port where the docker container is listening by default 1514.

And then we need a final change to the docker-compose.yml

graylog:
  ...
  ports:
    ...
   # Syslog TCP
    - 514:1514
   # Syslog UDP
    - 514:1514/udp

I changed both TCP and UDP ports, this way the none root Graylog service can listen on 1514 port and Docker will make the actual server listen to the default Syslog port 514.

Testing Syslog

A simple way to test the syslog is to send a message, this can be done using logger on a Linux machine

logger -n 192.168.0.20 "My message"

This will send a syslog message to the server with IP 192.168.0.20, if you don’t use the standard syslog port you can specify what port using the -P <port> parameter like this

logger -n 192.168.0.20 -P 1514 "MyMessage"

Troubleshooting

If you want to troubleshoot the Syslog input you can temporarly stop the input and create a new RAW input that will listen on the port 1514. This way you will receive the unprocessed messages in Graylog. Just remember to disable or remove the RAW input when you enables the Syslog input again.

The final docker-compose.yml

version: '3'

services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongo:
    image: mongo:4.2
    container_name: mongodb
    volumes:
      - /srv/mongodb:/data/db
    networks:
      - graylog

  # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.x$
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.0
    container_name: elasticsearch
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - /srv/elasticsearch:/usr/share/elastisearch/data
    ulimits:
      memlock:
        soft: -1
        hard: -1
    deploy:
      resources:
        limits:
          memory: 1g
    ports:
      - 9200:9200
    networks:
      - graylog

  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:4.2
    container_name: graylog
    environment:
      # Set server timezone
      - TZ=Europe/Stockholm
      - GRAYLOG_TIMEZONE=Europe/Stockholm
      - GRAYLOG_HTTP_EXTERNAL_URI=http://192.168.0.20:9000/
      - GRAYLOG_ELASTICSEARCH_HOSTS=http://192.168.0.20:9200
      - GRAYLOG_ELASTICSEARCH_VERSION=7
    volumes:
      - /srv/graylog/graylog:/usr/share/graylog/data
    networks:
      - graylog
    restart: always
    depends_on:
      - mongo
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 514:1514
      # Syslog UDP
      - 514:1514/udp
      # GELF TCP
      - 12201:12201

networks:
  graylog: