Go-ipfs: Segmentation fault `ipfs daemon --mount`

[karlek]

Version/Platform/Processor information (from ipfs version --all):

go-ipfs version: 0.4.3-dev-1c41cac
Repo version: 4
System version: amd64/linux
Golang version: devel +495e3c6 Tue May 17 04:02:11 2016 +0000

Type (bug, feature, meta, test failure, question): bug
Area (api, commands, daemon, fuse, etc): daemon
Priority (from P0: functioning, to P4: operations on fire): P2

Description:
I ran a ipfs get over night on a 13~GiB large hash, and when I woke up it hade crashed with a segmentation fault after 20%.

The ipfs get produced this output: Error: unexpected EOF

This was the output from the ipfs daemon:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]

goroutine 10460213 [running]:
panic(0xb5ce40, 0xc4200140d0)
    /home/_/go/src/runtime/panic.go:500 +0x18c
github.com/ipfs/go-ipfs/routing/dht.(*IpfsDHT).findProvidersAsyncRoutine(0xc4201d66e0, 0x104e7e0, 0xc42968e340, 0xc43b082030, 0x22, 0x3, 0xc43b31cf60)
    /home/_/Desktop/privgo/src/github.com/ipfs/go-ipfs/routing/dht/routing.go:385 +0x5bc
created by github.com/ipfs/go-ipfs/routing/dht.(*IpfsDHT).FindProvidersAsync
    /home/_/Desktop/privgo/src/github.com/ipfs/go-ipfs/routing/dht/routing.go:306 +0x16e

[whyrusleeping]

@karlek thanks for reporting! After this happened, was the daemon still running? or did the process exit?

[karlek]

The deamon didn't continue running, it crashed with the panic.
Sorry was out drinking :beer:

[whyrusleeping]

No worries, this one is tricky. Thanks for the info!

[whyrusleeping]

@karlek what version of go are you running? It doesnt appear to be a release version.

[whyrusleeping]

My first instinct is that the error is a 'hidden nil' where its a typed nil pointer returned through an interface. This would make it past the err != nil check on routing.go:380 and panic on line 385, but the panic trace should look different...

Proof of concept of the mentioned 'hidden nil' bug:

package main

import (
    "fmt"
)

type myErr struct {
    a int
}

func (e *myErr) Error() string {
    return fmt.Sprint(e.a)
}

func main() {
    var err error
    err = (*myErr)(nil)

    if err != nil {
        fmt.Println(err.Error())
    }
}

Results in:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x401043]

goroutine 1 [running]:
panic(0x48ff40, 0xc42000a0c0)
        /home/whyrusleeping/go/src/runtime/panic.go:500 +0x1a1
main.(*myErr).Error(0x0, 0x0, 0x0)
        /tmp/x8HM4.go:12 +0x43
main.main()
        /tmp/x8HM4.go:20 +0x48
exit status 2

Where we can clearly see that the panic is a result of the error being nil.

[whyrusleeping]

Pushing a PR that tries to debug and diagnose my suspicions here, It will print an error message and _not_ crash your daemon. :)

[karlek]

@whyrusleeping I use an old development version golang; haven't downloaded the latest version in a while.

[whyrusleeping]

@karlek if you don't mind, could you update to a newer release version of go and rebuild ipfs? The possibility exists that your old version of go has a bug in it thats causing this issue (especially since its not a release build).

[karlek]

Absolutely. Will write here if the bug reappear.

[Kubuxu]

The crash should be solved but we added error log in place it would appear. We would love to know what is causing such an unusual behaviour, so if you could report back if the logs contain info about this bug, we would really appreciate it.

[karlek]

I'm a new user of ipfs, and haven't figured out how to read the output the ipfs log tail yet.
How do I supply you the logs, i.e which commands should I run to extract the relevant information?

[Kubuxu]

It will be in output of ipfs daemon.

[karlek]

I haven't encountered this bug since it happened. Just to let you know!

[whyrusleeping]

@karlek Thanks for pinging back on this. I think that i'm going to go ahead and call it closed. Likely cause being a development version of the go compiler.