UDM to Graylog and timezones

I’ve started hooking up my various systems into my Graylog instance and ran into a hiccup with my UDM. Because the log format for /var/log/messages isn’t timezone aware, when the UDM sends its syslogs to Graylog, they get misinterpreted as though they were in UTC. I went on a wild-goose chase trying to fix this one, though the end solution was surprisingly simple.

Attempt 1: Change Unifi’s timezone to UTC

I figured if Graylog was interpreting my logs as UTC, I could just set them to be in UTC and avoid the whole issue.

Controller timezone

This introduced a second problem: the timezone of the Unifi Controller no longer matched the timezone of the UDM itself, resulting in mixed timezones in /var/log/messages.

# tail /var/log/messages
Oct 19 18:59:15 ... (a message from the UDM)
Oct 20 02:59:15 ... (a message from the controller)
Oct 19 18:59:15 ... (another message from the UDM)
Oct 20 02:59:15 ... (another message from the controller)

Perhaps it’s an issue with the controller and UDM having mismatched timezones? I went digging through the Ubiquiti community forums and found posts related syslogs and timezones, but no satisfactory conclusion. Out of curiosity, I attempted to adjust the timezone on the UDM to match the controller, only to find UTC wasn’t even an option!

UDM timezone

Okay, we’ll have to find another approach.

Attempt 2: Have Graylog input the syslogs with a specific timezone

If I can’t properly send the timestamps, maybe I can correct them on the receiving side of things. I go googling for graylog input timezone offset and come across a few threads that touch on the subject (1, 2, 3, 4), but I find conflicting answers and inconsistent results. Mucking around with the proposed solutions proves fruitless, so I take a step back. There just isn’t time to spend wading through half-baked forum posts.

Attempt 3: Reach out to the Unifi community regarding timezones

I didn’t quite expect this one to work, but I tried reaching out to the Ubiquiti forums, making an effort to detail out my problem, screenshots to help illustrate the point, firmware version numbers, etc. No replies, but that outcome was honestly expected.

Attempt 4: Modify the default timezone of the Graylog instance itself

Looking into Graylog’s configuration, I come across root_timezone = UTC. The DockerHub documentation details the GRAYLOG_ prefix that one can use to convert the option into an environment variable.

services:
  graylog:
    environment:
+      - GRAYLOG_ROOT_TIMEZONE=America/Vancouver

That’s a step forward! https://graylog.home.lan/system/overview/ now shows both my browser timezone and the admin user’s timezone as America/Vancouver, but the server timezone still shows as UTC. There’s no obvious option for this, but I know whatever option it is will be part of the underlying base image from which the Graylog image has been composed.

Googling change docker container timezone suggests the following might work:

services:
  graylog:
    environment:
+      - GRAYLOG_ROOT_TIMEZONE=America/Vancouver
+      - TZ=America/Vancouver

A quick container restart and it does! Moreover it does affect the default timezone Graylog applies to naive datetimes. Phew!

Reminders to self

Lastly, if I had been able to adjust the timezone on the UDM, I might have tricked myself into doing the same for all my devices and not discovered the env vars that truly fixed the issue. All the more reason to talk with others and see how the details differ.