Discovering the problem
I’ve had an issue sitting open for a long while that I’d yet to address. If Pi-hole starts up and system-apps'
/etc/resolv.conf had been
overwritten to use the DHCP settings of
nameserver 10.0.0.11, Pi-hole would fail to initialize. The work-around I’d discovered was to
/etc/resolv.conf, start Pi-hole, and then not care about whatever changes DHCP would make to it until I next had to
restart Pi-hole, which was seldom-to-never. Low risk, functional work-around, nothing depending on it: low priority.
That was until I started seeing other oddities that all came back to this one issue. I was working on getting Prometheus Alertmanager up and running and found it was unable to connect to the endpoints that were otherwise accessible from within my browser.
- Do these endpoints work from…
- My browser? Yes.
- system-apps? Yes.
- The Prometheus container? No.
- From the Prometheus container, can I…
- curl an arbitrary remote URL? Oh, the container doesn’t have curl.
- wget an arbitrary remote URL? No.
- ping any arbitrary remote IP? Yes.
- ping an IP within my private subnet? Yes.
Okay, this is really starting to sound like a DNS issue, which is what clues me in to the problem I had been having with Pi-hole.
cat the Prometheus container’s resolv.conf and see it has
nameserver 127.0.0.11. Odd, but I suspect that’s perhaps
specific to Docker and just a coincidence that it’s
.11 matches system-apps. A quick google reveals that yes, this is the expected
nameserver for a Docker container.
So let’s try something, because I now suspect that Docker spun up the container using the incorrect nameserver allocated to system-apps. I adjust system-apps' resolv.conf, restart Prometheus, and everything’s working fine.
An inelegant fix that introduces complexity
Checking my original GitLab issue for the problem, I see I’d previously found
Debian’s relevant documentation for pinning changes in resolv.conf. Alertnatively, it’s possible
to configure the nameserver used by a Docker container.
I opt for the former as it’s less hacky and implement it with a
supersede command in
/etc/dhcp/dhclient.conf as per Debian’s
root@system-apps:~# cat /etc/dhcp/dhclient.conf | grep "^supersede" supersede domain-name "hosts.lan"; supersede domain-search "hosts.lan"; supersede domain-name-servers 127.0.0.1;
Restarting the Prometheus container and everything starts polling without issue.
Trying not to go too deep into the rabbit hole
I could just stop here, but there is nagging thought that I can’t address without going down a rabbit hole for which I obviously have
no time, but… Is this solution the most elegant? I suspect that there’s a way of doing this while introducing less complexity. Maybe
there’s some Pi-hole option that allows it to start up with resolv.conf
nameserver 10.0.0.11 instead of requiring
nameserver 127.0.0.1? Attempting to do so causes
DNS resolution is currently unavailable.
Others report the same problem and workaround, and a solution I should investigate has been posted. The Pi-hole documentation even marks this option as recommended.
Sets your container’s resolve settings to localhost so it can resolve DHCP hostnames from Pi-hole’s DNSMasq, also fixes common resolution errors on container restart.
As a quick check, I revert the changes I’ve made to my dhclient.conf and resolv.conf, then I apply the following to my Pi-hole’s docker-compose.yml:
services: pihole: + dns: + - "127.0.0.1"
That seems to do it and is notably more elegant than modifying dhclient.conf and resolv.conf.
Three points come out of this whole exercise:
- A solution might fix your problem, but introduce complexity in doing so. It might not be the right solution.
- Thoroughly read through the documentation of any new software you’re about to use, especially as the individual components one adopts become increasingly complex.
- When developing software, if a setting is recommended, it should probably be enabled by default. I’m curious as to why the Pi-hole developers didn’t in this case.