Unhandled fault: page domain fault (0x81b) at 0x00465000 when starting pihole-FTL on recent kernel

All,

This week I upgraded my PogoPlug Pro to Debian Bullseye. The plug is running pi-hole which has been running fine for more than a year now. This was on kernel linux-4.14.198-oxnas-tld-1. Bullseye is running the same kernel and pi-hole runs fine as well.
Today I decided to upgrade to the most recent kernel I can find here (linux-5.4.224-oxnas-tld-1). Everything works but pihole-FTL fails with the following error at startup:

root@pogo01:/# /etc/init.d/pihole-FTL start
Not running

[  103.153164][ T1224] 8<--- cut here ---
[  103.156915][ T1224] Unhandled fault: page domain fault (0x81b) at 0x00465000
[  103.163927][ T1224] pgd = f9cf97d2
[  103.167308][ T1224] [00465000] *pgd=661c0831, *pte=63ae859f, *ppte=63ae8e6e
[  103.174243][ T1224] Internal error: : 81b [#1] PREEMPT SMP ARM
[  103.180042][ T1224] Modules linked in: rt2800pci rt2800mmio rt2800lib rt2x00pci rt2x00mmio rt2x00lib mac80211 cfg80211 rfkill dwmac_generic uas
[  103.192875][ T1224] CPU: 0 PID: 1224 Comm: pihole-FTL Not tainted 5.4.210-oxnas-tld-1 #1.0
[  103.201093][ T1224] Hardware name: Generic DT based system
[  103.206561][ T1224] PC is at v6_coherent_kern_range+0x4/0x2c
[  103.212195][ T1224] LR is at arm_syscall+0x188/0x3b4
[  103.217131][ T1224] pc : [<c010e6b0>]    lr : [<c010635c>]    psr: 60000013
[  103.224056][ T1224] sp : c61aff90  ip : 00000000  fp : bebdfb1c
[  103.229944][ T1224] r10: 00000000  r9 : c61ae000  r8 : c01011e4
[  103.235832][ T1224] r7 : 00466000  r6 : 00760000  r5 : c61ae000  r4 : 00000002
[  103.243018][ T1224] r3 : 00001000  r2 : c7189c00  r1 : 00466000  r0 : 00465000
[  103.250201][ T1224] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[  103.257991][ T1224] Control: 00c5787d  Table: 6617400a  DAC: 00000051
[  103.264396][ T1224] Process pihole-FTL (pid: 1224, stack limit = 0x2d7865ad)
[  103.271408][ T1224] Stack: (0xc61aff90 to 0xc61b0000)
[  103.276436][ T1224] ff80:                                     bebdfa70 00000000 00000000 000f0002
[  103.285271][ T1224] ffa0: c01011e4 c0101000 bebdfa70 00000000 00465000 00760000 00000000 00000014
[  103.294105][ T1224] ffc0: bebdfa70 00000000 00000000 000f0002 00000016 b6d8baac b6fcd9b8 bebdfb1c
[  103.302940][ T1224] ffe0: 00000000 bebdfa60 b6fa8fbc b6fa8fd4 60000010 00465000 00000000 00000000
[  103.311779][ T1224] [<c010e6b0>] (v6_coherent_kern_range) from [<c010635c>] (arm_syscall+0x188/0x3b4)
[  103.320957][ T1224] [<c010635c>] (arm_syscall) from [<c0101000>] (ret_fast_syscall+0x0/0x54)
[  103.329351][ T1224] Exception stack(0xc61affa8 to 0xc61afff0)
[  103.335073][ T1224] ffa0:                   bebdfa70 00000000 00465000 00760000 00000000 00000014
[  103.343905][ T1224] ffc0: bebdfa70 00000000 00000000 000f0002 00000016 b6d8baac b6fcd9b8 bebdfb1c
[  103.352734][ T1224] ffe0: 00000000 bebdfa60 b6fa8fbc b6fa8fd4
[  103.358458][ T1224] Code: ee070f15 e12fff1e e12fff1e e3c0001f (ee070f3a)
[  103.365217][ T1224] ---[ end trace 85d702e36acf4e5d ]---
Segmentation fault

Not sure who to ask for help on this subject, here? or the pihole people?, because this could be a kernel (?) issue.

Anyway, I would appreciate any thoughts on this.
Many thanks in advance ... and happy new year ;)

kennywest,

I think you should try pihole community (and of course users here who use pihole), to see which kernel versions people have reported running, and whether a segfault was seen lately.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Thank you for your reply. I asked the pi-hole community for help ... let's see if we can find a solution. In the meantime I reverted to the 4.14.198 kernel which seems to work without any problems.

maybe it's too late for a reply, but I've got the same error using kernel 5.4.224-oxnas-tld-1 and after some google search I found

(one) explanation for page_domain_fault.

I changed kernel configuration and removed

CONFIG_CPU_SW_DOMAIN_PAN

After installing the modified kernel I do not see this error anymore.
I'm unsure if this is the right solution but for the moment pihole seems to work.

Hi hej,

> After installing the modified kernel I do not see
> this error anymore.
> I'm unsure if this is the right solution but for
> the moment pihole seems to work.

That's good to know. Look like it is the right diagnosis.

Because CONFIG_CPU_SW_DOMAIN_PAN is a security feature, I think it is pi-hole code that need to be updated..

The poster Stefan recommended a solution:

Quote
https://stackoverflow.com/questions/39515407/how-to-handle-a-page-domain-fault-in-a-self-written-character-device-kernel-modu
As Tsyvarev mentioned, the input buffer needs to be copied from user space to kernel space via copy_from_user. After memcpy is replaced by copy_from_user the module works fine.

Perhaps updating your pi-hole package for Debian to se if it has been fixed?

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 04/17/2023 05:10PM by bodhi.

Hi Bodhi,

after a little more googling I found CONFIG_CPU_SW_DOMAIN_PAN breakage on ARM11 MPCore.

Quote
https://lists.infradead.org/pipermail/linux-arm-kernel/2016-January/400752.html
Russel King:
Having thought about this some more, I'm coming to the conclusion that
the only sane solution here is to change the help text for SW_PAN such
that if you want to run a kernel on ARM11 MPcore, you must disable
SW_PAN.

It's a thread from 2016 and I don't know the current state, especially for openWRT.
A quick check shows that CONFIG_CPU_SW_DOMAIN_PAN is enabled for oxnas target, but CONFIG_UACCESS_WITH_MEMCPY is disabled. I've read something about an incompatibility between these two config variables, but this is a (maybe) outdated information, too. https://patchwork.kernel.org/project/linux-arm-kernel/patch/20151203110041.GK8644@n2100.arm.linux.org.uk/

For me disabling CONFIG_CPU_SW_DOMAIN_PAN is my current workaround.


-hermann

BTW, I could install pihole from the pihole repository without compiling a private version of pihole-FTL.

Hi hermann,

> after a little more googling I found
>
> CONFIG_CPU_SW_DOMAIN_PAN breakage on ARM11
> MPCore.
>
>

Quote
https://lists.infradead.org/pipermail/linux-arm-kernel/2016-January/400752.html
> Russel King:

> Having thought about this some more, I'm coming to
> the conclusion that
> the only sane solution here is to change the help
> text for SW_PAN such
> that if you want to run a kernel on ARM11 MPcore,
> you must disable
> SW_PAN.
>

Ah. That's more like it. It's must be part of the reason that OXNAS has been removed from mainline, because they don't want to fix the SMP supports for OXNAS anymore. ARM11 MPCore has quite a bit of problems. Kernel developers kept finding they have to patch the SMP code to run with ARM11 MPcore, so they gave up on it. Also because there has been no activity on OXNAS in mainline for a long time.

I'll give it a close read.

>
> It's a thread from 2016 and I don't know the
> current state, especially for openWRT.
> A quick check shows that
> CONFIG_CPU_SW_DOMAIN_PAN is enabled for
> oxnas target, but
> CONFIG_UACCESS_WITH_MEMCPY is disabled.
> I've read something about an incompatibility
> between these two config variables, but this is a
> (maybe) outdated information, too.
> https://patchwork.kernel.org/project/linux-arm-kernel/patch/20151203110041.GK8644@n2100.arm.linux.org.uk/

I think the solution to remove memcpy is a sound solution. Sooner or later, pihole should clean up their code to use the kernel standard.

> For me disabling CONFIG_CPU_SW_DOMAIN_PAN
> is my current workaround.
> pihole-FTL.

I'll take a look at OpenWrt kernel config.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

Hi hermann,

Could you rebuild your kernel with these:

# CONFIG_UACCESS_WITH_MEMCPY is not set
CONFIG_CPU_SW_DOMAIN_PAN=y

And run pihole.

Without looking at pihole code, I hope it might switch to a different kernel call.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

at the risk of hijacking the thread - i personally would migrate away from pihole and move to a better solution - as shown below

Adguardhome : a self contained binary for arm5 architecture is available if you dont want to do the compiling yourself

or

Blocky : this would need some tweaking to the code

I appreciate that this doesnt contribute to the finding and reporting of optimisation of the Bhodi-acious kernel. but IMHO they are far superior to pihole

:HIJACK

============================
Breaking stuff since 1994 :-)
============================



Edited 1 time(s). Last edit at 04/18/2023 04:15PM by Gravelrash.

Gravelrash,

> at the risk of hijacking the thread

Not at all :)

> - i personally
> would migrate away from pihole and move to a
> better solution - as shown below
>
> Adguardhome
> : a self contained binary for arm5 architecture is
> available if you dont want to do the compiling
> yourself

It should run on armv6 which the Pogo Pro V3 is based on.

> or
>
> Blocky
> : this would need some tweaking to the code
>
> I appreciate that this doesnt contribute to the
> finding and reporting of optimisation of the
> Bhodi-acious kernel. but IMHO they are far
> superior to pihole
>
> :HIJACK

:)) thanks for the suggestion.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)

bodhi Wrote:
-------------------------------------------------------
> It should run on armv6 which the Pogo Pro V3 is
> based on.
>
It runs on just about any hardware architecture you can imagine :)

if i was running it on the arm6 versions you support - id be looking to up the swap file to around 756MB and set the swappiness quite aggressively - it is a bit of a ram hog when you do a blocklist update so i would make allowance for it. ibe been testing it on the L-50 and it runs fine and dandy plenty of ram to spare and i only see spikes in the ram usage when i ask it to go get updates .. otherwise it should run well on 128M.. its runs like a champ on the L-50 :))

============================
Breaking stuff since 1994 :-)
============================



Edited 1 time(s). Last edit at 04/18/2023 04:47PM by Gravelrash.

Hi bodhi,

no success!

root@pihole:~# zcat < /proc/config.gz | egrep 'UACCES|DOMAIN_PAN'
CONFIG_CPU_SW_DOMAIN_PAN=y
# CONFIG_UACCESS_WITH_MEMCPY is not set
root@pihole:~#

and

root@pihole:~# strace pihole-FTL
execve("/usr/bin/pihole-FTL", ["pihole-FTL"], 0xbeb10dc0 /* 14 vars */) = 0
brk(NULL)                               = 0x1a75000
uname({sysname="Linux", nodename="pihole.fritz.box", ...}) = 0
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/lib/tls/v6l/libm.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/tls/v6l", 0xbedc51b0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/lib/tls/libm.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/tls", 0xbedc51b0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/lib/v6l/libm.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/v6l", 0xbedc51b0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/lib/libm.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib", {st_mode=S_IFDIR|S_ISGID|0775, st_size=4096, ...}) = 0
openat(AT_FDCWD, "tls/v6l/libm.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "tls/libm.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "v6l/libm.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "libm.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=29167, ...}) = 0
mmap2(NULL, 29167, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb6fd9000
close(3)                                = 0
openat(AT_FDCWD, "/lib/arm-linux-gnueabi/libm.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\300u\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=595472, ...}) = 0
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6fd7000
mmap2(NULL, 659588, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6f0e000
mprotect(0xb6f9f000, 61440, PROT_NONE)  = 0
mmap2(0xb6fae000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x90000) = 0xb6fae000
close(3)                                = 0
openat(AT_FDCWD, "/usr/local/lib/librt.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "tls/v6l/librt.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "tls/librt.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "v6l/librt.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "librt.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/arm-linux-gnueabi/librt.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0@\26\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=30668, ...}) = 0
mmap2(NULL, 94720, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6ef6000
mprotect(0xb6efc000, 65536, PROT_NONE)  = 0
mmap2(0xb6f0c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0xb6f0c000
close(3)                                = 0
openat(AT_FDCWD, "/usr/local/lib/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "tls/v6l/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "tls/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "v6l/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/arm-linux-gnueabi/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\370\323\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=124628, ...}) = 0
mmap2(NULL, 188840, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6ec7000
mprotect(0xb6ee5000, 61440, PROT_NONE)  = 0
mmap2(0xb6ef4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1d000) = 0xb6ef4000
close(3)                                = 0
openat(AT_FDCWD, "/usr/local/lib/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "tls/v6l/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "tls/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "v6l/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/arm-linux-gnueabi/libpthread.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\230M\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=141496, ...}) = 0
mmap2(NULL, 180824, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6e9a000
mprotect(0xb6eb4000, 61440, PROT_NONE)  = 0
mmap2(0xb6ec3000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19000) = 0xb6ec3000
mmap2(0xb6ec5000, 4696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb6ec5000
close(3)                                = 0
openat(AT_FDCWD, "/usr/local/lib/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "tls/v6l/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "tls/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "v6l/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/arm-linux-gnueabi/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\\y\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1340264, ...}) = 0
mmap2(NULL, 1409596, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6d41000
mprotect(0xb6e85000, 61440, PROT_NONE)  = 0
mmap2(0xb6e94000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x143000) = 0xb6e94000
mmap2(0xb6e97000, 8764, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb6e97000
close(3)                                = 0
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6fd5000
set_tls(0xb6fd54d0)                     = 0
mprotect(0xb6e94000, 8192, PROT_READ)   = 0
mprotect(0xb6ec3000, 4096, PROT_READ)   = 0
mprotect(0xb6ef4000, 4096, PROT_READ)   = 0
mprotect(0xb6f0c000, 4096, PROT_READ)   = 0
mprotect(0xb6fae000, 4096, PROT_READ)   = 0
mprotect(0x4bb000, 3141632, PROT_READ|PROT_WRITE|PROT_EXEC) = 0
mprotect(0x4bb000, 3141632, PROT_READ|PROT_EXEC) = 0
cacheflush(0x4bb000, 0x7ba000, 0)       = ?
+++ killed by SIGSEGV +++
Segmentation fault

so I will stay with CONFIG_CPU_SW_DOMAIN_PAN disabled

-hermann

> so I will stay with CONFIG_CPU_SW_DOMAIN_PAN
> disabled

Sounds fine. Trading that security feature for more flexibility is not too bad. As long as you don't run strange software (strictly run Debian packages), then it's OK.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)