Monero: Strange crash this morning

[agentpatience]

See photo different crash error than yesterday.

[moneromooo-monero]

gdb /path/to/monerod core*
bt full

[selsta]

Try gdb /path/to/monerod core*

It seems like you entered gdb /path/to/monerod*

[agentpatience]

[agentpatience]

Something still not right, typing “bt full” in gdb says no stack.

[moneromooo-monero]

It's not finding the core file. Possibly it's kept somewhere else by something like systemd or so.
If you're running that, try:

coredumpctl debug monerod

[agentpatience]

coredumpctl debug monerod
No match found.

[moneromooo-monero]

coredumpctl list
coredumpctl debug XXX

And replace XXX with the full name of the binary the list command will print.

[agentpatience]

I since restarted and have not seen that again yet tho it is still crashing with “Killed”. I think that’s a separate issue. I ran the commands you asked for but it comes back as “no core dumps found”. This is Ubuntu 20.10 desktop minimal install - maybe have to turn anything on?

[arizvisa]

@agentpatience, look for any ".core" files in the directory you ran the monerod binary (or even "core*" might match it). make sure you back it up too.

nonetheless, if you run sysctl kernel.core_pattern, it'll list what your distro is doing with core files (i.e. dropping it in a file, or passing it to some program). If it's being passed to systemd-coredump, then you can use coredumpctl as @moneromooo-monero mentioned. If you use coredumpctl info /path/to/monerod, it'll print out information about the corefile which should include the backtrace.

If it's being kill'd and you couldn't figure out the signal number, dmesg or journalctl -ke might dump out whether your kernel's oom killer signalled monerod.

[arizvisa]

Heh. Friend just informed me that ubuntu is using "apport" (https://wiki.ubuntu.com/Apport), and will occasionally set the ulimit -c to 0. If that's the case, we should probably document somewhere to ensure ulimit -c unlimited is set.

Nonetheless, "corrupted size vs. prev_size" implies memory corruption. So if this issue is reproduceable, this could be serious...

[moneromooo-monero]

It is in the README.

[voidzero]

maybe log a warning when ulimit -c is 0 so the user can be made aware.

[moneromooo-monero]

People freak out for random things, like the initial message that tells you the log settings. Telling them "core files can't get generated because your limit is 0" will freak some other set of people out, I'm sure of that.

[arizvisa]

also, regardless of the core limit being 0, the pid's corefile will still get piped to things like apport or systemd-coredump. so those tools will still be able to snag the backtrace and core from the signal'd process, it's just the kernel won't write its standard corefile to disk by default.