FTL Crash

[roger-small]

Versions

Both Servers are identical including software:
Pi-hole version is v5.17.3 (Latest: v5.17.3)
web version is v5.21 (Latest: v5.21)
FTL version is v5.24 (Latest: v5.24)

Platform

• Ubuntu 22.04.3 LTS (GNU/Linux 6.5.11-6-pve x86_64)

• ProxMox Container

Expected behavior

I expected to be able to let it run and not crash (Not trying to be sarcastic, just filling out the form. I do appreciate you guys and your service)

Actual behavior / bug

This morning, I logged into the web admin console for two of my servers to check on the status of the servers. We just installed these servers last week. The Top Permitted Domains and Query Types portion of the dashboard was taking a long time to view. I closed out the web page and figured I'd look at it later. About 15 minutes later, I started getting reports that multiple devices were going down and becoming unresponsive.

Steps to reproduce

Unable to reproduce, although this happened on two separate servers at the same time.

Debug Token

Server 1 (GPM)

• URL: https://tricorder.pi-hole.net/75l5ld4F/
  Server 2 (GPN)
• URL: https://tricorder.pi-hole.net/8t3M7iwf/

Screenshots

If applicable, add screenshots to help explain your problem.

Server 1 pihole-FTL.log (Call this one GPM)

[2024-01-26 08:56:10.461 40749M] INFO: FTL replaced 1 invalid characters with ~ in the query "gpm~ap42.local"
[2024-01-26 08:56:10.491 40749M] INFO: FTL replaced 1 invalid characters with ~ in the query "gpm~ap42"
[2024-01-26 08:56:10.766 40749M] INFO: FTL replaced 1 invalid characters with ~ in the query "gpm~ap55.local"
[2024-01-26 08:56:10.792 40749M] INFO: FTL replaced 1 invalid characters with ~ in the query "gpm~ap55"
[2024-01-26 08:56:10.818 40749M] INFO: FTL replaced 1 invalid characters with ~ in the query "gpm~ap55.local"
[2024-01-26 08:56:10.841 40749M] INFO: FTL replaced 1 invalid characters with ~ in the query "gpm~ap55"
[2024-01-26 08:56:12.453 40749M] INFO: FTL replaced 1 invalid characters with ~ in the query "gpm~ap61.local"
[2024-01-26 08:56:12.480 40749/T40768] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2024-01-26 08:56:12.480 40749/T40768] ---------------------------->  FTL crashed!  <----------------------------
[2024-01-26 08:56:12.480 40749/T40768] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2024-01-26 08:56:12.480 40749/T40768] Please report a bug at https://github.com/pi-hole/FTL/issues
[2024-01-26 08:56:12.480 40749/T40768] and include in your report already the following details:
[2024-01-26 08:56:12.480 40749/T40768] FTL has been running for 343587 seconds
[2024-01-26 08:56:12.480 40749/T40768] FTL branch: master
[2024-01-26 08:56:12.480 40749/T40768] FTL version: v5.24
[2024-01-26 08:56:12.480 40749/T40768] FTL commit: 01227311
[2024-01-26 08:56:12.480 40749/T40768] FTL date: 2024-01-06 16:47:56 +0000
[2024-01-26 08:56:12.480 40749/T40768] FTL user: started as pihole, ended as pihole
[2024-01-26 08:56:12.480 40749/T40768] Compiled for x86_64 (compiled on CI) using gcc (Debian 8.3.0-6) 8.3.0
[2024-01-26 08:56:12.480 40749/T40768] Process details: MID: 40749
[2024-01-26 08:56:12.480 40749/T40768]                  PID: 40749
[2024-01-26 08:56:12.480 40749/T40768]                  TID: 40768
[2024-01-26 08:56:12.480 40749/T40768]                  Name: telnet-IPv4-0
[2024-01-26 08:56:12.480 40749/T40768] Received signal: Segmentation fault
[2024-01-26 08:56:12.480 40749/T40768]      at address: 0x7f0aa1117000
[2024-01-26 08:56:12.480 40749/T40768]      with code:  SEGV_ACCERR (Invalid permissions for mapped object)
[2024-01-26 08:56:12.481 40749/T40768] Backtrace:
[2024-01-26 08:56:12.481 40749/T40768] B[0000]: /usr/bin/pihole-FTL(generate_backtrace+0x41) [0x55a74ee24771]
[2024-01-26 08:56:12.483 40749/T40768] L[0000]: N/A (0x5c771 -> addr2line 0x5c771 -e /usr/bin/pihole-FTL)
[2024-01-26 08:56:12.483 40749/T40768] B[0001]: /usr/bin/pihole-FTL(+0x5cccd) [0x55a74ee24ccd]
[2024-01-26 08:56:12.485 40749/T40768] L[0001]: N/A (0x5cccd -> addr2line 0x5cccd -e /usr/bin/pihole-FTL)
[2024-01-26 08:56:12.485 40749/T40768] B[0002]: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0aa5042520]
[2024-01-26 08:56:12.485 40749/T40768] B[0003]: /usr/bin/pihole-FTL(getTopDomains+0x135) [0x55a74ee26a15]
[2024-01-26 08:56:12.487 40749/T40768] L[0003]: N/A (0x5ea15 -> addr2line 0x5ea15 -e /usr/bin/pihole-FTL)
[2024-01-26 08:56:12.487 40749/T40768] B[0004]: /usr/bin/pihole-FTL(process_request+0x1d9) [0x55a74ee2a539]
[2024-01-26 08:56:12.489 40749/T40768] L[0004]: N/A (0x62539 -> addr2line 0x62539 -e /usr/bin/pihole-FTL)
[2024-01-26 08:56:12.489 40749/T40768] B[0005]: /usr/bin/pihole-FTL(+0x62d84) [0x55a74ee2ad84]
[2024-01-26 08:56:12.490 40749/T40768] L[0005]: N/A (0x62d84 -> addr2line 0x62d84 -e /usr/bin/pihole-FTL)
[2024-01-26 08:56:12.490 40749/T40768] B[0006]: /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f0aa5094ac3]
[2024-01-26 08:56:12.490 40749/T40768] B[0007]: /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x7f0aa5126850]
[2024-01-26 08:56:12.491 40749/T40768] ------ Listing content of directory /dev/shm ------
[2024-01-26 08:56:12.491 40749/T40768] File Mode User:Group      Size  Filename
[2024-01-26 08:56:12.491 40749/T40768] rwxrwxrwx root:root       260  .
[2024-01-26 08:56:12.491 40749/T40768] rwxr-xr-x root:root       480  ..
[2024-01-26 08:56:12.491 40749/T40768] rw------- pihole:pihole     4K  FTL-per-client-regex
[2024-01-26 08:56:12.491 40749/T40768] rw------- pihole:pihole   541K  FTL-dns-cache
[2024-01-26 08:56:12.491 40749/T40768] rw------- pihole:pihole     8K  FTL-overTime
[2024-01-26 08:56:12.491 40749/T40768] rw------- pihole:pihole   222M  FTL-queries
[2024-01-26 08:56:12.491 40749/T40768] rw------- pihole:pihole   315K  FTL-upstreams
[2024-01-26 08:56:12.491 40749/T40768] rw------- pihole:pihole   946K  FTL-clients
[2024-01-26 08:56:12.491 40749/T40768] rw------- pihole:pihole    42M  FTL-domains
[2024-01-26 08:56:12.491 40749/T40768] rw------- pihole:pihole    26M  FTL-strings
[2024-01-26 08:56:12.492 40749/T40768] rw------- pihole:pihole    16  FTL-settings
[2024-01-26 08:56:12.492 40749/T40768] rw------- pihole:pihole   248  FTL-counters
[2024-01-26 08:56:12.492 40749/T40768] rw------- pihole:pihole    88  FTL-lock
[2024-01-26 08:56:12.492 40749/T40768] ----------------------------------------------------

Server 2 pihole-FTL.log (Call this one GPN)

[2024-01-26 08:56:27.409 48625M] INFO: FTL replaced 1 invalid characters with ~ in the query "gpn~ap217"
[2024-01-26 08:56:27.879 48625M] INFO: FTL replaced 1 invalid characters with ~ in the query "gpn~apadmin.local"
[2024-01-26 08:56:27.912 48625M] INFO: FTL replaced 1 invalid characters with ~ in the query "gpn~apadmin"
[2024-01-26 08:56:27.942 48625M] INFO: FTL replaced 1 invalid characters with ~ in the query "gpn~apadmin.local"
[2024-01-26 08:56:27.969 48625M] INFO: FTL replaced 1 invalid characters with ~ in the query "gpn~apadmin"
[2024-01-26 08:56:29.647 48625M] INFO: FTL replaced 1 invalid characters with ~ in the query "gpn~ap115.local"
[2024-01-26 08:56:29.656 48625/T48633] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2024-01-26 08:56:29.656 48625/T48633] ---------------------------->  FTL crashed!  <----------------------------
[2024-01-26 08:56:29.656 48625/T48633] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2024-01-26 08:56:29.656 48625/T48633] Please report a bug at https://github.com/pi-hole/FTL/issues
[2024-01-26 08:56:29.656 48625/T48633] and include in your report already the following details:
[2024-01-26 08:56:29.656 48625/T48633] FTL has been running for 343499 seconds
[2024-01-26 08:56:29.656 48625/T48633] FTL branch: master
[2024-01-26 08:56:29.656 48625/T48633] FTL version: v5.24
[2024-01-26 08:56:29.656 48625/T48633] FTL commit: 01227311
[2024-01-26 08:56:29.656 48625/T48633] FTL date: 2024-01-06 16:47:56 +0000
[2024-01-26 08:56:29.656 48625/T48633] FTL user: started as pihole, ended as pihole
[2024-01-26 08:56:29.657 48625/T48633] Compiled for x86_64 (compiled on CI) using gcc (Debian 8.3.0-6) 8.3.0
[2024-01-26 08:56:29.657 48625/T48633] Process details: MID: 48625
[2024-01-26 08:56:29.657 48625/T48633]                  PID: 48625
[2024-01-26 08:56:29.657 48625/T48633]                  TID: 48633
[2024-01-26 08:56:29.657 48625/T48633]                  Name: telnet-IPv4-1
[2024-01-26 08:56:29.657 48625/T48633] Received signal: Segmentation fault
[2024-01-26 08:56:29.657 48625/T48633]      at address: 0x7fac415ee000
[2024-01-26 08:56:29.657 48625/T48633]      with code:  SEGV_ACCERR (Invalid permissions for mapped object)
[2024-01-26 08:56:29.657 48625/T48633] Backtrace:
[2024-01-26 08:56:29.658 48625/T48633] B[0000]: /usr/bin/pihole-FTL(generate_backtrace+0x41) [0x55f6ca076771]
[2024-01-26 08:56:29.660 48625/T48633] L[0000]: N/A (0x5c771 -> addr2line 0x5c771 -e /usr/bin/pihole-FTL)
[2024-01-26 08:56:29.660 48625/T48633] B[0001]: /usr/bin/pihole-FTL(+0x5cccd) [0x55f6ca076ccd]
[2024-01-26 08:56:29.661 48625/T48633] L[0001]: N/A (0x5cccd -> addr2line 0x5cccd -e /usr/bin/pihole-FTL)
[2024-01-26 08:56:29.662 48625/T48633] B[0002]: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fac42842520]
[2024-01-26 08:56:29.662 48625/T48633] B[0003]: /usr/bin/pihole-FTL(getTopDomains+0x135) [0x55f6ca078a15]
[2024-01-26 08:56:29.663 48625/T48633] L[0003]: N/A (0x5ea15 -> addr2line 0x5ea15 -e /usr/bin/pihole-FTL)
[2024-01-26 08:56:29.663 48625/T48633] B[0004]: /usr/bin/pihole-FTL(process_request+0x1d9) [0x55f6ca07c539]
[2024-01-26 08:56:29.665 48625/T48633] L[0004]: N/A (0x62539 -> addr2line 0x62539 -e /usr/bin/pihole-FTL)
[2024-01-26 08:56:29.665 48625/T48633] B[0005]: /usr/bin/pihole-FTL(+0x62d84) [0x55f6ca07cd84]
[2024-01-26 08:56:29.667 48625/T48633] L[0005]: N/A (0x62d84 -> addr2line 0x62d84 -e /usr/bin/pihole-FTL)
[2024-01-26 08:56:29.667 48625/T48633] B[0006]: /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fac42894ac3]
[2024-01-26 08:56:29.667 48625/T48633] B[0007]: /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x7fac42926850]
[2024-01-26 08:56:29.667 48625/T48633] ------ Listing content of directory /dev/shm ------
[2024-01-26 08:56:29.667 48625/T48633] File Mode User:Group      Size  Filename
[2024-01-26 08:56:29.667 48625/T48633] rwxrwxrwx root:root       260  .
[2024-01-26 08:56:29.667 48625/T48633] rwxr-xr-x root:root       480  ..
[2024-01-26 08:56:29.667 48625/T48633] rw------- pihole:pihole     4K  FTL-per-client-regex
[2024-01-26 08:56:29.667 48625/T48633] rw------- pihole:pihole   676K  FTL-dns-cache
[2024-01-26 08:56:29.667 48625/T48633] rw------- pihole:pihole     8K  FTL-overTime
[2024-01-26 08:56:29.667 48625/T48633] rw------- pihole:pihole   229M  FTL-queries
[2024-01-26 08:56:29.668 48625/T48633] rw------- pihole:pihole   315K  FTL-upstreams
[2024-01-26 08:56:29.668 48625/T48633] rw------- pihole:pihole   602K  FTL-clients
[2024-01-26 08:56:29.668 48625/T48633] rw------- pihole:pihole    28M  FTL-domains
[2024-01-26 08:56:29.668 48625/T48633] rw------- pihole:pihole    21M  FTL-strings
[2024-01-26 08:56:29.668 48625/T48633] rw------- pihole:pihole    16  FTL-settings
[2024-01-26 08:56:29.668 48625/T48633] rw------- pihole:pihole   248  FTL-counters
[2024-01-26 08:56:29.668 48625/T48633] rw------- pihole:pihole    88  FTL-lock

Additional context

Add any other context about the problem here.

We have Add-Block disabled for now, as we were recommended to use Pi-Hole as the local DHCP/DNS server for our school instead of relying on external DHCP / DNS.
These two servers are running on ProxMox VE 8.1.3 (Linux 6.5.11-6-pve kernel) servers as containers. Each Pi-Hole instance has 8GB of memory and 8 cores allocated to them. The physical host processors are Intel(R) Xeon(R) CPU E5-2697 v4

[DL6ER]

Thanks for your report and sorry for the delay in replying - it's busy times!

Could you please run the following commands and tell me the output (if there is any, but I do hope so):

addr2line 0x5cccd -e /usr/bin/pihole-FTL

addr2line 0x5ea15 -e /usr/bin/pihole-FTL

addr2line 0x62539 -e /usr/bin/pihole-FTL

[roger-small]

Good afternoon, thanks for taking the time to look at this issue. I'm assuming I've done something wrong with the setup or something, especially as both servers crashed after I loaded the admin page.... but I appreciate your time in looking at the issue. One thing I am working on is renaming all of my AP's. Looking at the log file, there are miles of logs that "replaced invalid characters in hostname". All of our ap's have a space in them, and we have 20 or so at each location, and they're also our most "vocal" devices on our networks. I still wouldn't think that would cause a problem. Also, the resources to the VM were way over provisioned, even during the heaviest usage, the CPU utilization never went above 20% and it was using less than 1 GB of the assigned 8GB... Let me know if there is anything other information that is needed.

Server 1 (GPM):
root@gpm-pi-hole:# addr2line 0x5cccd -e /usr/bin/pihole-FTL
/__w/FTL/FTL/src/signals.c:242
root@gpm-pi-hole:# addr2line 0x5ea15 -e /usr/bin/pihole-FTL
/__w/FTL/FTL/src/api/api.c:229
root@gpm-pi-hole:~# addr2line 0x62539 -e /usr/bin/pihole-FTL
/__w/FTL/FTL/src/api/request.c:55

Server 2 (GPN):
root@gpn-pi-hole:# addr2line 0x5cccd -e /usr/bin/pihole-FTL
/__w/FTL/FTL/src/signals.c:242
root@gpn-pi-hole:# addr2line 0x5ea15 -e /usr/bin/pihole-FTL
/__w/FTL/FTL/src/api/api.c:229
root@gpn-pi-hole:~# addr2line 0x62539 -e /usr/bin/pihole-FTL
/__w/FTL/FTL/src/api/request.c:55

[DL6ER]

Thanks.

I still wouldn't think that would cause a problem

Spaces are illegal in hostnames which are nothing else than domains. There's also no possibility for something like www.github.com. Some routers/systems allow you to specify such standard-violating hostnames but Pi-hole should be able to handle them.

Looking at the lines you posted, we see that the error happens here:

FTL/src/api/api.c

Line 229 in 0122731

temparray[domainID][0] = domainID;

which is actually bit puzzling as this shouldn't be able to crash... The only reason I could imagine right now would be if you have an outrageous number of domains in your memory - however, we have seen reports of Pi-hole being employed at the network provider-level and there was never such a crash anywhere.

As such, I am not sure which direction we could go right now to provide a fix for the issue you are seeing. However, what I would recommend at this point would be you trying the all-new Pi-hole v6.0 which is currently in late beta phase and should go live soon(ish).

See more details here: https://pi-hole.net/blog/2023/10/09/pi-hole-v6-beta-testing/

As the entirety of the API has been rewritten, I doubt this crash will still exist.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Please comment or update this issue or it will be closed in 5 days.