Linux: CONFIG_SECCOMP panic with LLD
Forked off from https://github.com/ClangBuiltLinux/continuous-integration/pull/81
With my fork of the Pi kernel, I can successfully build, boot, and run a kernel compiled with Clang on both 4.14 and 4.20. Additionally, thanks to the top two patches, I can link the kernel successfully with lld. However, it panics before getting to userspace. I have pulled a log, which I am happy to triage as needed but honestly, I don't know where to start.
nathanchance
All 30 comments
The oops' all look the same
[ 2.401795] PC is at 0x80008274
[ 2.401812] LR is at do_seccomp+0x138/0x73c
so ARM always has a dedicated register for its parent stack frame, "the link register," LR. So LR is do_seccomp+0x138 meaning if you disassemble the object file from the source file that defines do_seccomp (use grep or ag to find which translation unit it comes from), then look at the instruction at 0x138 bytes from the start of do_seccomp label, it's probably making a call into another function (on ARM, a call instruction is bl "branch and link" which I think of as call+set LR). So do_seccomp is jumping to 0x80008274. Looks like the kernels unwinder wasn't able to tell you what that is.
I don't know what seccomp is, and haven't looked, but maybe LLD is broken with seccomp? Maybe if seccomp can be disabled, you might be able to boot, then we can go from there.
nickdesaulniers
on 6 Dec 2018
Thank you very much for that analysis! Disabling CONFIG_SECCOMP allows it to boot successfully and everything appears to be functioning normally (been running for about five minutes without issues). I'll re-enable CONFIG_SECCOMP and poke my head around the disassembly and see what it looks like.
nathanchance
on 7 Dec 2018
I wasn't able to fully dive into my analysis tonight since I didn't have all the time I was hoping for but I uploaded the full disassembly of kernel/seccomp.o and a new stacktrace in case someone wants to glance them over.
https://gist.github.com/75872ee94ec8cbea55e279ab32f4d8e2
https://gist.github.com/7af41fcbb2f804a5f65bbedb81a1462a
It did occur to me that @kees is the seccomp maintainer, he may have some insight into this (although, there's no guarantee that this is isolated to seccomp, it could be a global issue that is just happening with seccomp in this config).
nathanchance
on 7 Dec 2018
Yes @kees can help then. Next question, is this isolated to arm32? I assume it's maybe not in any defconfig, so we don't have CI coverage of it?
nickdesaulniers
on 7 Dec 2018
Thanks for the disassembly. Because it's the disassembly of an object file that hasn't been linked yet, the addresses are not fully resolved yet, so the addresses from the stack trace don't quite line up with the disassembled object file. That's ok.
[ 2.377552] LR is at do_seccomp+0x1a8/0x768
(note, this is different from the first trace, took me a few minutes to figure that out, screws up the below math)
000004f8 <do_seccomp>:
0x1a8 + 0x4f8 == 0x6a0
6a0: e3500000 cmp r0, #0
Now another special thing is that the LR points to the next instruction after the child stack frame returns, we need to subtract 4 bytes (1 word on arm32).
0x6a0 - 0x1 == 0x69c
69c: ebfffffe bl 0 <arm_copy_from_user>
aha, there's a branch and link instruction (bl) which would have set the LR to 0x6a0. Looks like arm_copy_from_user is called twice from do_seccomp, why does this second one fail? Why is it special? do_seccomp might not immediately call arm_copy_from_user, but through inlining it may be the case.
Next idea to try; can you find the two invocations of arm_copy_from_user (or maybe copy_from_user) in do_seccomp? If you comment out one invocation and maybe return early, or jump to some error handler, or put some dummy value in the variable it's trying to read into, does booting progress? (We're trying to isolate the problematic source)
nickdesaulniers
on 7 Dec 2018
Also, I'll bet that arm_copy_from_user is implemented in assembly, which may be why it doesn't show up next to the PC in the unwind (I'm not sure how that's fixed, honestly).
nickdesaulniers
on 7 Dec 2018
Next question, is this isolated to arm32? I assume it's maybe not in any defconfig, so we don't have CI coverage of it?
CONFIG_SECCOMP=y is in multi_v7_defconfig and arm64 defconfig so I guess real hardware is exploiting this in a way QEMU isn't. I can't say if this is isolated to just arm32 because I don't have an arm64 board.
Also, I'll bet that
arm_copy_from_useris implemented in assembly, which may be why it doesn't show up next to the PC in the unwind (I'm not sure how that's fixed, honestly).
Correct: https://github.com/nathanchance/pi-kernel/blob/rpi-4.14.y/arch/arm/lib/copy_from_user.S
aha, there's a branch and link instruction (
bl) which would have set the LR to 0x6a0. Looks likearm_copy_from_useris called twice fromdo_seccomp, why does this second one fail? Why is it special?do_seccompmight not immediately callarm_copy_from_user, but through inlining it may be the case.
Yeah that's interesting... I see three instances of copy_from_user (which are turned into arm_copy_from_user) and none of them are directly in do_seccomp so inlining must have happened.
https://github.com/nathanchance/pi-kernel/blob/rpi-4.14.y/kernel/seccomp.c#L422
https://github.com/nathanchance/pi-kernel/blob/rpi-4.14.y/kernel/seccomp.c#L428
https://github.com/nathanchance/pi-kernel/blob/rpi-4.14.y/kernel/seccomp.c#L909
Since CONFIG_COMPAT isn't relevant for a 32-bit kernel, it must be the last two.
Next idea to try; can you find the two invocations of
arm_copy_from_user(or maybecopy_from_user) indo_seccomp? If you comment out one invocation and maybe return early, or jump to some error handler, or put some dummy value in the variable it's trying to read into, does booting progress? (We're trying to isolate the problematic source)
The problematic one appears to be in in seccomp_prepare_user_filter, as I can successfully boot with this diff (although my modules didn't load so I wasn't able to verify the version string successfully):
diff --git a/kernel/seccomp.c b/kernel/seccomp.c
index 075e344a87c3..6e243d9886dc 100644
--- a/kernel/seccomp.c
+++ b/kernel/seccomp.c
@@ -425,9 +425,11 @@ seccomp_prepare_user_filter(const char __user *user_filter)
fprog.filter = compat_ptr(fprog32.filter);
} else /* falls through to the if below. */
#endif
+#if 0
if (copy_from_user(&fprog, user_filter, sizeof(fprog)))
goto out;
filter = seccomp_prepare_filter(&fprog);
+#endif
out:
return filter;
}
CONFIG_SECCOMP=y is in multi_v7_defconfig and arm64 defconfig so I guess real hardware is exploiting this in a way QEMU isn't.
That would be bad, and peculiar if something like that was the linker's fault. But the Pi's config is not the same as our defconfigs (and we should verify booting the pi kernel in qemu).
CONFIG_MMU changes the implementation of raw_copy_from_user. Is that set for our QEMU tested defconfigs, but not for the pi, or vice-versa? (I feel like that would have blown up in other ways, and it's not clear to my why this lone call site is problematic).
nickdesaulniers
on 9 Dec 2018
and we should verify booting the pi kernel in qemu
Yeah this doesn't really seem to easy (at least with their config). Doing some research, it seems like everyone is using this repo and its build script uses arch/arm/configs/versatile_defconfig as a base, not arch/arm/configs/bcm2709_defconfig (because it's only concerned with running RPI userspace, not its kernel).
I will try to do more research on this tomorrow and I'll double check all of the results I've posted so far since I agree that it's super bizarre that JUST changing the linker is resulting in this issue...
CONFIG_MMUchanges the implementation ofraw_copy_from_user. Is that set for our QEMU tested defconfigs, but not for the pi, or vice-versa? (I feel like that would have blown up in other ways, and it's not clear to my why this lone call site is problematic).
It is default y on both arm and arm64 so I assume QEMU should be using it.
nathanchance
on 9 Dec 2018
Does this fail under qemu too? Have you compared the disassemble between ld.bfd and ld.lld?
There is nothing "special" about this copy_from_user() that I can see.
It looks like the Oops is at address 8000000d -- which seems like a strange address: it's the kernel base plus 14, which is pretty small... I would assume there is a dereference from some structure address but the address is very wrong. Perhaps the linker failed to apply the right offset, and used only the kernel base offset?
kees
on 12 Dec 2018
Can you do an objdump of vmlinux, instead, and include the copy_from_user() code? That should give us a view into exactly what the linked result looks like, and what it's trying to do at that address.
kees
on 12 Dec 2018
Yeah this doesn't really seem to easy (at least with their config).
You should be able to run 'make kvm_guest.config' to enable the additional options you need for any kernel inside of qemu (kvm or emulated, using "-m virt" instead of -m versatile). In addition you will want 鈥–ONFIG_DEBUG_INFO=y, CONFIG_DEBUG_INFO=y and possibly CONFIG_DEBUG_INFO=n, to allow you attach gdb to the qemu instance and use the provided tools from the kernel.
arndb
on 12 Dec 2018
I will debug this best I can over the weekend or early next week, thank you both for the tips!
nathanchance
on 13 Dec 2018
Can you do an objdump of vmlinux, instead, and include the copy_from_user() code? That should give us a view into exactly what the linked result looks like, and what it's trying to do at that address.
Sorry for taking so long to get back to this :(
Here are the various snippets, along with the full dump here:
804c0124 <__copy_from_user_memcpy>:
804c0124: e92d4ff0 push {r4, r5, r6, r7, r8, r9, sl, fp, lr}
804c0128: e24dd004 sub sp, sp, #4
804c012c: e1a0600d mov r6, sp
804c0130: e1a08000 mov r8, r0
804c0134: e7cc601f bfc r6, #0, #13
804c0138: e1a04002 mov r4, r2
804c013c: e1a05001 mov r5, r1
804c0140: e5960008 ldr r0, [r6, #8]
804c0144: e3500000 cmp r0, #0
804c0148: 0a000066 beq 804c02e8 <__copy_from_user_memcpy+0x1c4>
804c014c: e5969004 ldr r9, [r6, #4]
804c0150: e3590000 cmp r9, #0
804c0154: 1a000003 bne 804c0168 <__copy_from_user_memcpy+0x44>
804c0158: e596000c ldr r0, [r6, #12]
804c015c: e5900338 ldr r0, [r0, #824] ; 0x338
804c0160: e280003c add r0, r0, #60 ; 0x3c
804c0164: eb0d49b8 bl 8081284c <down_read>
804c0168: e3540000 cmp r4, #0
804c016c: e58d6000 str r6, [sp]
804c0170: 0a000051 beq 804c02bc <__copy_from_user_memcpy+0x198>
804c0174: e7e86655 ubfx r6, r5, #12, #9
804c0178: e1a0baa5 lsr fp, r5, #21
804c017c: e1a0000d mov r0, sp
804c0180: e3a0100c mov r1, #12
804c0184: e7cc0011 bfi r0, r1, #0, #13
804c0188: e5900000 ldr r0, [r0]
804c018c: e5900338 ldr r0, [r0, #824] ; 0x338
804c0190: e5900024 ldr r0, [r0, #36] ; 0x24
804c0194: e790a18b ldr sl, [r0, fp, lsl #3]
804c0198: e35a0000 cmp sl, #0
804c019c: 0a000019 beq 804c0208 <__copy_from_user_memcpy+0xe4>
804c01a0: e21a0002 ands r0, sl, #2
804c01a4: 1a000017 bne 804c0208 <__copy_from_user_memcpy+0xe4>
804c01a8: e3060804 movw r0, #26628 ; 0x6804
804c01ac: e3001e3c movw r1, #3644 ; 0xe3c
804c01b0: e34800d0 movt r0, #32976 ; 0x80d0
804c01b4: e34810e1 movt r1, #32993 ; 0x80e1
804c01b8: e5900000 ldr r0, [r0]
804c01bc: e5911000 ldr r1, [r1]
804c01c0: e060062a rsb r0, r0, sl, lsr #12
804c01c4: e0800180 add r0, r0, r0, lsl #3
804c01c8: e0810100 add r0, r1, r0, lsl #2
804c01cc: e280701c add r7, r0, #28
804c01d0: e1a00007 mov r0, r7
804c01d4: eb0d541c bl 8081524c <_raw_spin_lock>
804c01d8: e7cba01f bfc sl, #0, #12
804c01dc: e24a0481 sub r0, sl, #-2130706432 ; 0x81000000
804c01e0: e7900106 ldr r0, [r0, r6, lsl #2]
804c01e4: e2000003 and r0, r0, #3
804c01e8: e3500003 cmp r0, #3
804c01ec: 0a00001f beq 804c0270 <__copy_from_user_memcpy+0x14c>
804c01f0: f57ff05b dmb ish
804c01f4: e1d700b0 ldrh r0, [r7]
804c01f8: e2800001 add r0, r0, #1
804c01fc: e1c700b0 strh r0, [r7]
804c0200: f57ff04a dsb ishst
804c0204: e320f004 sev
804c0208: e3590000 cmp r9, #0
804c020c: 1a000004 bne 804c0224 <__copy_from_user_memcpy+0x100>
804c0210: e59d0000 ldr r0, [sp]
804c0214: e590000c ldr r0, [r0, #12]
804c0218: e5900338 ldr r0, [r0, #824] ; 0x338
804c021c: e280003c add r0, r0, #60 ; 0x3c
804c0220: ebf2c44b bl 80171354 <up_read>
804c0224: e59d0000 ldr r0, [sp]
804c0228: e5900008 ldr r0, [r0, #8]
804c022c: e2401001 sub r1, r0, #1
804c0230: e1a00005 mov r0, r5
804c0234: eb0cdbe3 bl 807f71c8 <__get_user_1>
804c0238: e1801009 orr r1, r0, r9
804c023c: e3500000 cmp r0, #0
804c0240: 13000006 movwne r0, #6
804c0244: e3510000 cmp r1, #0
804c0248: 1a000005 bne 804c0264 <__copy_from_user_memcpy+0x140>
804c024c: e59d0000 ldr r0, [sp]
804c0250: e590000c ldr r0, [r0, #12]
804c0254: e5900338 ldr r0, [r0, #824] ; 0x338
804c0258: e280003c add r0, r0, #60 ; 0x3c
804c025c: eb0d497a bl 8081284c <down_read>
804c0260: e3a00000 mov r0, #0
804c0264: e3500000 cmp r0, #0
804c0268: 0affffc3 beq 804c017c <__copy_from_user_memcpy+0x58>
804c026c: ea00001a b 804c02dc <__copy_from_user_memcpy+0x1b8>
804c0270: e1a00005 mov r0, r5
804c0274: e1a01005 mov r1, r5
804c0278: e7df061f bfc r0, #12, #20
804c027c: e2606a01 rsb r6, r0, #4096 ; 0x1000
804c0280: e1a00008 mov r0, r8
804c0284: e1560004 cmp r6, r4
804c0288: 81a06004 movhi r6, r4
804c028c: e1a02006 mov r2, r6
804c0290: eb0cde6a bl 807f7c40 <memcpy>
804c0294: f57ff05b dmb ish
804c0298: e0544006 subs r4, r4, r6
804c029c: e0855006 add r5, r5, r6
804c02a0: e1d700b0 ldrh r0, [r7]
804c02a4: e0888006 add r8, r8, r6
804c02a8: e2800001 add r0, r0, #1
804c02ac: e1c700b0 strh r0, [r7]
804c02b0: f57ff04a dsb ishst
804c02b4: e320f004 sev
804c02b8: 1affffad bne 804c0174 <__copy_from_user_memcpy+0x50>
804c02bc: e3a04000 mov r4, #0
804c02c0: e3590000 cmp r9, #0
804c02c4: 1a000004 bne 804c02dc <__copy_from_user_memcpy+0x1b8>
804c02c8: e59d0000 ldr r0, [sp]
804c02cc: e590000c ldr r0, [r0, #12]
804c02d0: e5900338 ldr r0, [r0, #824] ; 0x338
804c02d4: e280003c add r0, r0, #60 ; 0x3c
804c02d8: ebf2c41d bl 80171354 <up_read>
804c02dc: e1a00004 mov r0, r4
804c02e0: e28dd004 add sp, sp, #4
804c02e4: e8bd8ff0 pop {r4, r5, r6, r7, r8, r9, sl, fp, pc}
804c02e8: e1a00008 mov r0, r8
804c02ec: e1a01005 mov r1, r5
804c02f0: e1a02004 mov r2, r4
804c02f4: eb0cde51 bl 807f7c40 <memcpy>
804c02f8: e3a04000 mov r4, #0
804c02fc: eafffff6 b 804c02dc <__copy_from_user_memcpy+0x1b8>
...
804c0310 <arm_copy_from_user>:
804c0310: e352003f cmp r2, #63 ; 0x3f
804c0314: 8a000000 bhi 804c031c <arm_copy_from_user+0xc>
804c0318: ea0cd643 b 807f5c2c <__copy_from_user_std>
804c031c: eaffff80 b 804c0124 <__copy_from_user_memcpy>
...
807f5c2c <__copy_from_user_std>:
807f5c2c: e1a036ad lsr r3, sp, #13
807f5c30: e1a03683 lsl r3, r3, #13
807f5c34: e5933008 ldr r3, [r3, #8]
807f5c38: e091c002 adds ip, r1, r2
807f5c3c: e2433001 sub r3, r3, #1
807f5c40: 315c0003 cmpcc ip, r3
807f5c44: 23a01000 movcs r1, #0
807f5c48: e320f014 csdb
807f5c4c: e3a03000 mov r3, #0
807f5c50: e92d401d push {r0, r2, r3, r4, lr}
807f5c54: e2522004 subs r2, r2, #4
807f5c58: ba000032 blt 807f5d28 <__copy_from_user_std+0xfc>
807f5c5c: e210c003 ands ip, r0, #3
807f5c60: f5d1f000 pld [r1]
807f5c64: 1a000038 bne 807f5d4c <__copy_from_user_std+0x120>
807f5c68: e211c003 ands ip, r1, #3
807f5c6c: 1a000042 bne 807f5d7c <__copy_from_user_std+0x150>
807f5c70: e252201c subs r2, r2, #28
807f5c74: e92d01e0 push {r5, r6, r7, r8}
807f5c78: ba000013 blt 807f5ccc <__copy_from_user_std+0xa0>
807f5c7c: f5d1f000 pld [r1]
807f5c80: e2522060 subs r2, r2, #96 ; 0x60
807f5c84: f5d1f01c pld [r1, #28]
807f5c88: ba000002 blt 807f5c98 <__copy_from_user_std+0x6c>
807f5c8c: f5d1f03c pld [r1, #60] ; 0x3c
807f5c90: f5d1f05c pld [r1, #92] ; 0x5c
807f5c94: f5d1f07c pld [r1, #124] ; 0x7c
807f5c98: e4913004 ldr r3, [r1], #4
807f5c9c: e4914004 ldr r4, [r1], #4
807f5ca0: e4915004 ldr r5, [r1], #4
807f5ca4: e4916004 ldr r6, [r1], #4
807f5ca8: e4917004 ldr r7, [r1], #4
807f5cac: e4918004 ldr r8, [r1], #4
807f5cb0: e491c004 ldr ip, [r1], #4
807f5cb4: e491e004 ldr lr, [r1], #4
807f5cb8: e2522020 subs r2, r2, #32
807f5cbc: e8a051f8 stmia r0!, {r3, r4, r5, r6, r7, r8, ip, lr}
807f5cc0: aafffff3 bge 807f5c94 <__copy_from_user_std+0x68>
807f5cc4: e3720060 cmn r2, #96 ; 0x60
807f5cc8: aafffff2 bge 807f5c98 <__copy_from_user_std+0x6c>
807f5ccc: e212c01c ands ip, r2, #28
807f5cd0: e26cc020 rsb ip, ip, #32
807f5cd4: 108ff00c addne pc, pc, ip
807f5cd8: ea000011 b 807f5d24 <__copy_from_user_std+0xf8>
807f5cdc: e320f000 nop {0}
807f5ce0: e4913004 ldr r3, [r1], #4
807f5ce4: e4914004 ldr r4, [r1], #4
807f5ce8: e4915004 ldr r5, [r1], #4
807f5cec: e4916004 ldr r6, [r1], #4
807f5cf0: e4917004 ldr r7, [r1], #4
807f5cf4: e4918004 ldr r8, [r1], #4
807f5cf8: e491e004 ldr lr, [r1], #4
807f5cfc: e08ff00c add pc, pc, ip
807f5d00: e320f000 nop {0}
807f5d04: e320f000 nop {0}
807f5d08: e4803004 str r3, [r0], #4
807f5d0c: e4804004 str r4, [r0], #4
807f5d10: e4805004 str r5, [r0], #4
807f5d14: e4806004 str r6, [r0], #4
807f5d18: e4807004 str r7, [r0], #4
807f5d1c: e4808004 str r8, [r0], #4
807f5d20: e480e004 str lr, [r0], #4
807f5d24: e8bd01e0 pop {r5, r6, r7, r8}
807f5d28: e1b02f82 lsls r2, r2, #31
807f5d2c: 14d13001 ldrbne r3, [r1], #1
807f5d30: 24d14001 ldrbcs r4, [r1], #1
807f5d34: 24d1c001 ldrbcs ip, [r1], #1
807f5d38: 14c03001 strbne r3, [r0], #1
807f5d3c: 24c04001 strbcs r4, [r0], #1
807f5d40: 24c0c001 strbcs ip, [r0], #1
807f5d44: e28dd008 add sp, sp, #8
807f5d48: e8bd8011 pop {r0, r4, pc}
807f5d4c: e26cc004 rsb ip, ip, #4
807f5d50: e35c0002 cmp ip, #2
807f5d54: c4d13001 ldrbgt r3, [r1], #1
807f5d58: a4d14001 ldrbge r4, [r1], #1
807f5d5c: e4d1e001 ldrb lr, [r1], #1
807f5d60: c4c03001 strbgt r3, [r0], #1
807f5d64: a4c04001 strbge r4, [r0], #1
807f5d68: e052200c subs r2, r2, ip
807f5d6c: e4c0e001 strb lr, [r0], #1
807f5d70: baffffec blt 807f5d28 <__copy_from_user_std+0xfc>
807f5d74: e211c003 ands ip, r1, #3
807f5d78: 0affffbc beq 807f5c70 <__copy_from_user_std+0x44>
807f5d7c: e3c11003 bic r1, r1, #3
807f5d80: e35c0002 cmp ip, #2
807f5d84: e491e004 ldr lr, [r1], #4
807f5d88: 0a000032 beq 807f5e58 <__copy_from_user_std+0x22c>
807f5d8c: ca000063 bgt 807f5f20 <__copy_from_user_std+0x2f4>
807f5d90: e252201c subs r2, r2, #28
807f5d94: ba000025 blt 807f5e30 <__copy_from_user_std+0x204>
807f5d98: e92d03e0 push {r5, r6, r7, r8, r9}
807f5d9c: f5d1f000 pld [r1]
807f5da0: e2522060 subs r2, r2, #96 ; 0x60
807f5da4: f5d1f01c pld [r1, #28]
807f5da8: ba000002 blt 807f5db8 <__copy_from_user_std+0x18c>
807f5dac: f5d1f03c pld [r1, #60] ; 0x3c
807f5db0: f5d1f05c pld [r1, #92] ; 0x5c
807f5db4: f5d1f07c pld [r1, #124] ; 0x7c
807f5db8: e4914004 ldr r4, [r1], #4
807f5dbc: e4915004 ldr r5, [r1], #4
807f5dc0: e4916004 ldr r6, [r1], #4
807f5dc4: e4917004 ldr r7, [r1], #4
807f5dc8: e1a0342e lsr r3, lr, #8
807f5dcc: e2522020 subs r2, r2, #32
807f5dd0: e4918004 ldr r8, [r1], #4
807f5dd4: e4919004 ldr r9, [r1], #4
807f5dd8: e491c004 ldr ip, [r1], #4
807f5ddc: e491e004 ldr lr, [r1], #4
807f5de0: e1833c04 orr r3, r3, r4, lsl #24
807f5de4: e1a04424 lsr r4, r4, #8
807f5de8: e1844c05 orr r4, r4, r5, lsl #24
807f5dec: e1a05425 lsr r5, r5, #8
807f5df0: e1855c06 orr r5, r5, r6, lsl #24
807f5df4: e1a06426 lsr r6, r6, #8
807f5df8: e1866c07 orr r6, r6, r7, lsl #24
807f5dfc: e1a07427 lsr r7, r7, #8
807f5e00: e1877c08 orr r7, r7, r8, lsl #24
807f5e04: e1a08428 lsr r8, r8, #8
807f5e08: e1888c09 orr r8, r8, r9, lsl #24
807f5e0c: e1a09429 lsr r9, r9, #8
807f5e10: e1899c0c orr r9, r9, ip, lsl #24
807f5e14: e1a0c42c lsr ip, ip, #8
807f5e18: e18ccc0e orr ip, ip, lr, lsl #24
807f5e1c: e8a013f8 stmia r0!, {r3, r4, r5, r6, r7, r8, r9, ip}
807f5e20: aaffffe3 bge 807f5db4 <__copy_from_user_std+0x188>
807f5e24: e3720060 cmn r2, #96 ; 0x60
807f5e28: aaffffe2 bge 807f5db8 <__copy_from_user_std+0x18c>
807f5e2c: e8bd03e0 pop {r5, r6, r7, r8, r9}
807f5e30: e212c01c ands ip, r2, #28
807f5e34: 0a000005 beq 807f5e50 <__copy_from_user_std+0x224>
807f5e38: e1a0342e lsr r3, lr, #8
807f5e3c: e491e004 ldr lr, [r1], #4
807f5e40: e25cc004 subs ip, ip, #4
807f5e44: e1833c0e orr r3, r3, lr, lsl #24
807f5e48: e4803004 str r3, [r0], #4
807f5e4c: cafffff9 bgt 807f5e38 <__copy_from_user_std+0x20c>
807f5e50: e2411003 sub r1, r1, #3
807f5e54: eaffffb3 b 807f5d28 <__copy_from_user_std+0xfc>
807f5e58: e252201c subs r2, r2, #28
807f5e5c: ba000025 blt 807f5ef8 <__copy_from_user_std+0x2cc>
807f5e60: e92d03e0 push {r5, r6, r7, r8, r9}
807f5e64: f5d1f000 pld [r1]
807f5e68: e2522060 subs r2, r2, #96 ; 0x60
807f5e6c: f5d1f01c pld [r1, #28]
807f5e70: ba000002 blt 807f5e80 <__copy_from_user_std+0x254>
807f5e74: f5d1f03c pld [r1, #60] ; 0x3c
807f5e78: f5d1f05c pld [r1, #92] ; 0x5c
807f5e7c: f5d1f07c pld [r1, #124] ; 0x7c
807f5e80: e4914004 ldr r4, [r1], #4
807f5e84: e4915004 ldr r5, [r1], #4
807f5e88: e4916004 ldr r6, [r1], #4
807f5e8c: e4917004 ldr r7, [r1], #4
807f5e90: e1a0382e lsr r3, lr, #16
807f5e94: e2522020 subs r2, r2, #32
807f5e98: e4918004 ldr r8, [r1], #4
807f5e9c: e4919004 ldr r9, [r1], #4
807f5ea0: e491c004 ldr ip, [r1], #4
807f5ea4: e491e004 ldr lr, [r1], #4
807f5ea8: e1833804 orr r3, r3, r4, lsl #16
807f5eac: e1a04824 lsr r4, r4, #16
807f5eb0: e1844805 orr r4, r4, r5, lsl #16
807f5eb4: e1a05825 lsr r5, r5, #16
807f5eb8: e1855806 orr r5, r5, r6, lsl #16
807f5ebc: e1a06826 lsr r6, r6, #16
807f5ec0: e1866807 orr r6, r6, r7, lsl #16
807f5ec4: e1a07827 lsr r7, r7, #16
807f5ec8: e1877808 orr r7, r7, r8, lsl #16
807f5ecc: e1a08828 lsr r8, r8, #16
807f5ed0: e1888809 orr r8, r8, r9, lsl #16
807f5ed4: e1a09829 lsr r9, r9, #16
807f5ed8: e189980c orr r9, r9, ip, lsl #16
807f5edc: e1a0c82c lsr ip, ip, #16
807f5ee0: e18cc80e orr ip, ip, lr, lsl #16
807f5ee4: e8a013f8 stmia r0!, {r3, r4, r5, r6, r7, r8, r9, ip}
807f5ee8: aaffffe3 bge 807f5e7c <__copy_from_user_std+0x250>
807f5eec: e3720060 cmn r2, #96 ; 0x60
807f5ef0: aaffffe2 bge 807f5e80 <__copy_from_user_std+0x254>
807f5ef4: e8bd03e0 pop {r5, r6, r7, r8, r9}
807f5ef8: e212c01c ands ip, r2, #28
807f5efc: 0a000005 beq 807f5f18 <__copy_from_user_std+0x2ec>
807f5f00: e1a0382e lsr r3, lr, #16
807f5f04: e491e004 ldr lr, [r1], #4
807f5f08: e25cc004 subs ip, ip, #4
807f5f0c: e183380e orr r3, r3, lr, lsl #16
807f5f10: e4803004 str r3, [r0], #4
807f5f14: cafffff9 bgt 807f5f00 <__copy_from_user_std+0x2d4>
807f5f18: e2411002 sub r1, r1, #2
807f5f1c: eaffff81 b 807f5d28 <__copy_from_user_std+0xfc>
807f5f20: e252201c subs r2, r2, #28
807f5f24: ba000025 blt 807f5fc0 <__copy_from_user_std+0x394>
807f5f28: e92d03e0 push {r5, r6, r7, r8, r9}
807f5f2c: f5d1f000 pld [r1]
807f5f30: e2522060 subs r2, r2, #96 ; 0x60
807f5f34: f5d1f01c pld [r1, #28]
807f5f38: ba000002 blt 807f5f48 <__copy_from_user_std+0x31c>
807f5f3c: f5d1f03c pld [r1, #60] ; 0x3c
807f5f40: f5d1f05c pld [r1, #92] ; 0x5c
807f5f44: f5d1f07c pld [r1, #124] ; 0x7c
807f5f48: e4914004 ldr r4, [r1], #4
807f5f4c: e4915004 ldr r5, [r1], #4
807f5f50: e4916004 ldr r6, [r1], #4
807f5f54: e4917004 ldr r7, [r1], #4
807f5f58: e1a03c2e lsr r3, lr, #24
807f5f5c: e2522020 subs r2, r2, #32
807f5f60: e4918004 ldr r8, [r1], #4
807f5f64: e4919004 ldr r9, [r1], #4
807f5f68: e491c004 ldr ip, [r1], #4
807f5f6c: e491e004 ldr lr, [r1], #4
807f5f70: e1833404 orr r3, r3, r4, lsl #8
807f5f74: e1a04c24 lsr r4, r4, #24
807f5f78: e1844405 orr r4, r4, r5, lsl #8
807f5f7c: e1a05c25 lsr r5, r5, #24
807f5f80: e1855406 orr r5, r5, r6, lsl #8
807f5f84: e1a06c26 lsr r6, r6, #24
807f5f88: e1866407 orr r6, r6, r7, lsl #8
807f5f8c: e1a07c27 lsr r7, r7, #24
807f5f90: e1877408 orr r7, r7, r8, lsl #8
807f5f94: e1a08c28 lsr r8, r8, #24
807f5f98: e1888409 orr r8, r8, r9, lsl #8
807f5f9c: e1a09c29 lsr r9, r9, #24
807f5fa0: e189940c orr r9, r9, ip, lsl #8
807f5fa4: e1a0cc2c lsr ip, ip, #24
807f5fa8: e18cc40e orr ip, ip, lr, lsl #8
807f5fac: e8a013f8 stmia r0!, {r3, r4, r5, r6, r7, r8, r9, ip}
807f5fb0: aaffffe3 bge 807f5f44 <__copy_from_user_std+0x318>
807f5fb4: e3720060 cmn r2, #96 ; 0x60
807f5fb8: aaffffe2 bge 807f5f48 <__copy_from_user_std+0x31c>
807f5fbc: e8bd03e0 pop {r5, r6, r7, r8, r9}
807f5fc0: e212c01c ands ip, r2, #28
807f5fc4: 0a000005 beq 807f5fe0 <__copy_from_user_std+0x3b4>
807f5fc8: e1a03c2e lsr r3, lr, #24
807f5fcc: e491e004 ldr lr, [r1], #4
807f5fd0: e25cc004 subs ip, ip, #4
807f5fd4: e183340e orr r3, r3, lr, lsl #8
807f5fd8: e4803004 str r3, [r0], #4
807f5fdc: cafffff9 bgt 807f5fc8 <__copy_from_user_std+0x39c>
807f5fe0: e2411001 sub r1, r1, #1
807f5fe4: eaffff4f b 807f5d28 <__copy_from_user_std+0xfc>
807f5fe8: d4d4d4d4 ldrble sp, [r4], #1236 ; 0x4d4
807f5fec: d4d4d4d4 ldrble sp, [r4], #1236 ; 0x4d4
807f5ff0: d4d4d4d4 ldrble sp, [r4], #1236 ; 0x4d4
807f5ff4: d4d4d4d4 ldrble sp, [r4], #1236 ; 0x4d4
807f5ff8: d4d4d4d4 ldrble sp, [r4], #1236 ; 0x4d4
807f5ffc: d4d4d4d4 ldrble sp, [r4], #1236 ; 0x4d4
Getting the Pi kernel to boot in QEMU is still a WIP... A multi_v7_defconfig kernel boots when linked with LLD just fine: https://travis-ci.com/nathanchance/continuous-integration/jobs/162884028
nathanchance
on 8 Jan 2019
If the only difference is the linker, than the disassembly without relocations shouldn't show us any differences. The only way I know to print relocations is via [llvm-]objdump's -r flag. Either LLD is creating a bad relocation, or dropping a section it should not be.
Also, from the problematic section I wonder if user_filter is a C style string that could be printk'd or is just a bag of bytes? If it's a C string, maybe its value is a clue. It's possible that something else went wrong in the kernel, which then gave userspace a bad value, which passed it back into the kernel, at which point we then observe a failure (from a cascade of failures).
nickdesaulniers
on 9 Oct 2019
I will have to try and pull out my Pi and see if I can reproduce this with the latest tip of tree source.
nathanchance
on 11 Oct 2019
Looks like CrOS is now hitting this issue: https://bugs.chromium.org/p/chromium/issues/detail?id=1020633#c39
nickdesaulniers
on 18 Nov 2019
@m-gupta reports the diff in https://bugs.chromium.org/p/chromium/issues/detail?id=1020633#c47 works. RFT
nickdesaulniers
on 19 Nov 2019
I'll try to get my Pi up and running tonight or tomorrow to see if that diff resolves this issue as well!
nathanchance
on 19 Nov 2019
I can confirm that Kees' patch resolves this issue on my Raspberry Pi. Feel free to add:
Tested-by: Nathan Chancellor <[email protected]>
before submitting to the mailing list. I'd also recommend a CC to stable so this gets automatically picked up.
nathanchance
on 20 Nov 2019
nickdesaulniers
on 22 Nov 2019
FWIW, based on a suggestion by Kees, I checked and it looks like lld is placing .fixup before _stext while bfd places .fixup between _stext and _etext. It's not clear if that's relevant.
nickdesaulniers
on 8 Feb 2020
FWIW, based on a suggestion by Kees, I checked and it looks like lld is placing
.fixupbefore_stextwhile bfd places.fixupbetween_stextand_etext. It's not clear if that's relevant.
That'll put it in unmapped memory (stuff before _stext is, I think, unmapped and made available to the allocator). Regardless, I found the problem: there should be no .fixup section at all. All other instances but one were renamed to .text.fixup in c4a84ae39b4a5 (v4.1). Patch sent.
https://lore.kernel.org/lkml/202002071754.F5F073F1D@keescook/
kees
on 8 Feb 2020
Thanks @kees . I quickly tested this on one ARM32 chromebook quickly and I can now ssh into it with this fix. Starting a full round of testing.
m-gupta
on 8 Feb 2020
Sent to ARM patch tracker: https://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=8958/1
kees
on 10 Feb 2020
--- a/arch/arm/lib/copy_from_user.S
+++ b/arch/arm/lib/copy_from_user.S
@@ -118,7 +118,7 @@ ENTRY(arm_copy_from_user)
ENDPROC(arm_copy_from_user)
- .pushsection .fixup,"ax"
+ .pushsection .text.fixup,"ax"
.align 0
copy_abort_preamble
ldmfd sp!, {r1, r2, r3}
Unrelated, but .align 0 (added by https://github.com/torvalds/linux/commit/fadab0943d1c5b652a66858bb99b204fedaad96b) below looks weird to me. It is a no-op, though.
MaskRay
on 11 Feb 2020
Unrelated, but .align 0 (added by torvalds@fadab09) below looks weird to me. It is a no-op, though.
@MaskRay I think it "disables alignment requirements":
https://sourceware.org/binutils/docs/as/Align.html#Align
nickdesaulniers
on 13 Feb 2020
nickdesaulniers
on 21 Feb 2020
fixed: f87b1c49bc675da30d8e1e8f4b60b800312c7b90 (one year old bug, woot!)
nickdesaulniers
on 19 Mar 2020
GKH has sent it back to 4.4, too.
nickdesaulniers
on 19 Mar 2020
Related issues
tpgxyz
路
4Comments
nickdesaulniers
路
3Comments
nickdesaulniers
路
3Comments
tpgxyz
路
4Comments
nathanchance
路
4Comments
Most helpful comment
I can confirm that Kees' patch resolves this issue on my Raspberry Pi. Feel free to add:
before submitting to the mailing list. I'd also recommend a CC to stable so this gets automatically picked up.