I’ve started hooking up my various systems into my Graylog instance and ran into a hiccup
with my UDM. Because the log format for
/var/log/messages isn’t timezone aware, when the
UDM sends its syslogs to Graylog, they get misinterpreted as though they were in UTC. I
went on a wild-goose chase trying to fix this one, though the end solution was surprisingly
Attempt 1: Change Unifi’s timezone to UTC
I figured if Graylog was interpreting my logs as UTC, I could just set them to be in UTC and avoid the whole issue.
This introduced a second problem: the timezone of the Unifi Controller no longer matched
the timezone of the UDM itself, resulting in mixed timezones in
# tail /var/log/messages Oct 19 18:59:15 ... (a message from the UDM) Oct 20 02:59:15 ... (a message from the controller) Oct 19 18:59:15 ... (another message from the UDM) Oct 20 02:59:15 ... (another message from the controller)
Perhaps it’s an issue with the controller and UDM having mismatched timezones? I went digging through the Ubiquiti community forums and found posts related syslogs and timezones, but no satisfactory conclusion. Out of curiosity, I attempted to adjust the timezone on the UDM to match the controller, only to find UTC wasn’t even an option!
Okay, we’ll have to find another approach.
Attempt 2: Have Graylog input the syslogs with a specific timezone
If I can’t properly send the timestamps, maybe I can correct them on the receiving
side of things. I go googling for
graylog input timezone offset and come across a
few threads that touch on the subject (1,
but I find conflicting answers and inconsistent results. Mucking around with the
proposed solutions proves fruitless, so I take a step back. There just isn’t time to
spend wading through half-baked forum posts.
Attempt 3: Reach out to the Unifi community regarding timezones
I didn’t quite expect this one to work, but I tried reaching out to the Ubiquiti forums, making an effort to detail out my problem, screenshots to help illustrate the point, firmware version numbers, etc. No replies, but that outcome was honestly expected.
Attempt 4: Modify the default timezone of the Graylog instance itself
Looking into Graylog’s configuration, I come across
root_timezone = UTC.
The DockerHub documentation details the
GRAYLOG_ prefix that one can use to convert the option into an environment variable.
services: graylog: environment: + - GRAYLOG_ROOT_TIMEZONE=America/Vancouver
That’s a step forward!
https://graylog.home.lan/system/overview/ now shows both
my browser timezone and the admin user’s timezone as America/Vancouver, but the
server timezone still shows as UTC. There’s no obvious option for this, but I
know whatever option it is will be part of the underlying base image from which
the Graylog image has been composed.
change docker container timezone suggests the following might work:
services: graylog: environment: + - GRAYLOG_ROOT_TIMEZONE=America/Vancouver + - TZ=America/Vancouver
A quick container restart and it does! Moreover it does affect the default timezone Graylog applies to naive datetimes. Phew!
Reminders to self
- The problem isn’t always with the system you suspect
- The documentation you need isn’t always in one place
- Community forums might help you, mislead you, or ignore you
- Sometimes it helps to both tighten and broaden the focus of your search
Lastly, if I had been able to adjust the timezone on the UDM, I might have tricked myself into doing the same for all my devices and not discovered the env vars that truly fixed the issue. All the more reason to talk with others and see how the details differ.