Pi-hole and resolv.conf
Discovering the problem
I’ve had an issue sitting open for a long while that I’d yet to address. If Pi-hole starts up and system-apps' /etc/resolv.conf
had been
overwritten to use the DHCP settings of nameserver 10.0.0.11
, Pi-hole would fail to initialize. The work-around I’d discovered was to
manually modify /etc/resolv.conf
, start Pi-hole, and then not care about whatever changes DHCP would make to it until I next had to
restart Pi-hole, which was seldom-to-never. Low risk, functional work-around, nothing depending on it: low priority.
That was until I started seeing other oddities that all came back to this one issue. I was working on getting Prometheus Alertmanager up and running and found it was unable to connect to the endpoints that were otherwise accessible from within my browser.
- Do these endpoints work from…
- My browser? Yes.
- system-apps? Yes.
- The Prometheus container? No.
- From the Prometheus container, can I…
- curl an arbitrary remote URL? Oh, the container doesn’t have curl.
- wget an arbitrary remote URL? No.
- ping any arbitrary remote IP? Yes.
- ping an IP within my private subnet? Yes.
Okay, this is really starting to sound like a DNS issue, which is what clues me in to the problem I had been having with Pi-hole.
I cat
the Prometheus container’s resolv.conf and see it has nameserver 127.0.0.11
. Odd, but I suspect that’s perhaps
specific to Docker and just a coincidence that it’s .11
matches system-apps. A quick google reveals that yes, this is the expected
nameserver for a Docker container.
So let’s try something, because I now suspect that Docker spun up the container using the incorrect nameserver allocated to system-apps. I adjust system-apps' resolv.conf, restart Prometheus, and everything’s working fine.
An inelegant fix that introduces complexity
Checking my original GitLab issue for the problem, I see I’d previously found
Debian’s relevant documentation for pinning changes in resolv.conf. Alertnatively, it’s possible
to configure the nameserver used by a Docker container.
I opt for the former as it’s less hacky and implement it with a supersede
command in /etc/dhcp/dhclient.conf
as per Debian’s
documentation.
root@system-apps:~# cat /etc/dhcp/dhclient.conf | grep "^supersede"
supersede domain-name "hosts.lan";
supersede domain-search "hosts.lan";
supersede domain-name-servers 127.0.0.1;
Restarting the Prometheus container and everything starts polling without issue.
Trying not to go too deep into the rabbit hole
I could just stop here, but there is nagging thought that I can’t address without going down a rabbit hole for which I obviously have
no time, but… Is this solution the most elegant? I suspect that there’s a way of doing this while introducing less complexity. Maybe
there’s some Pi-hole option that allows it to start up with resolv.conf nameserver 10.0.0.11
instead of requiring
nameserver 127.0.0.1
? Attempting to do so causes DNS resolution is currently unavailable
.
Others report the same problem and workaround, and a solution I should investigate has been posted. The Pi-hole documentation even marks this option as recommended.
--dns=127.0.0.1
RecommendedSets your container’s resolve settings to localhost so it can resolve DHCP hostnames from Pi-hole’s DNSMasq, also fixes common resolution errors on container restart.
As a quick check, I revert the changes I’ve made to my dhclient.conf and resolv.conf, then I apply the following to my Pi-hole’s docker-compose.yml:
services:
pihole:
+ dns:
+ - "127.0.0.1"
That seems to do it and is notably more elegant than modifying dhclient.conf and resolv.conf.
Three points come out of this whole exercise:
- A solution might fix your problem, but introduce complexity in doing so. It might not be the right solution.
- Thoroughly read through the documentation of any new software you’re about to use, especially as the individual components one adopts become increasingly complex.
- When developing software, if a setting is recommended, it should probably be enabled by default. I’m curious as to why the Pi-hole developers didn’t in this case.