Go-ipfs: "panic: counter already signaled done" in go-libp2p-kad-dht addPeerToQuery

Created on 10 Mar 2019 · 5Comments · Source: ipfs/go-ipfs

Version information:

go-ipfs version: 0.4.19-
Repo version: 7
System version: amd64/windows
Golang version: go1.11.5

Windows 7, fwiw.

Type:

panic

Description:

I'm running go-ipfs pin add --recursive --progress in a loop, pinning hashes that are also pinned on a remote server, against a daemon running on the same machine. Every few hours the daemon hits this panic:

panic: counter already signaled done. use a new counter.

goroutine 2351179 [running]:
gx/ipfs/QmQNQhNmY4STU1MURjH9vYEMpx2ncMS4gbwxXWtrEjzVAq/go-todocounter.(*todoCounter).Increment(0xc003d04540, 0xc000000001)
        /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmQNQhNmY4STU1MURjH9vYEMpx2ncMS4gbwxXWtrEjzVAq/go-todocounter/counter.go:86 +0xb6
gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht.(*dhtQueryRunner).addPeerToQuery(0xc00d958580, 0xc0126e6450, 0x22)
        /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht/query.go:200 +0x110
gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht.(*dhtQueryRunner).Run(0xc00d958580, 0x138ef40, 0xc0077b7f80, 0xc004d3fe90, 0x3, 0x3, 0x0, 0x0, 0x0)
        /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht/query.go:136 +0x14a
gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht.(*dhtQuery).Run(0xc004d3fef0, 0x138ef40, 0xc0077b7f80, 0xc004d3fe90, 0x3, 0x3, 0x0, 0x0, 0x0)
        /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht/query.go:70 +0x109
gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht.(*IpfsDHT).GetClosestPeers.func2(0xc0012c7800, 0xc00e867da0, 0xc004d3fef0, 0x138efc0, 0xc0015e7bc0, 0xc004d3fe90, 0x3, 0x3, 0xc008891a40, 0x22)
        /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht/lookup.go:96 +0xe2
created by gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht.(*IpfsDHT).GetClosestPeers
        /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht/lookup.go:92 +0x2fd

I also reported this over at libp2p/go-libp2p-kad-dht#292, but I wanted to get it on the radar here as well.

Source

hcs64

👍3

Most helpful comment

For now, you'll have to build the latest go-ipfs master. The recommended way is to:

Clone the source.
Run make build
Install the ./cmd/ipfs/ipfs binary.

Alternatively, you should now now be able to just run GO111MODULE=on go get github.com/ipfs/go-ipfs@master (although this will give you a slightly non-standard build).

Stebalien on 27 Mar 2019

👍2

All 5 comments

also just got this:

panic: counter already signaled done. use a new counter.
goroutine 57218162 [running]:
gx/ipfs/QmQNQhNmY4STU1MURjH9vYEMpx2ncMS4gbwxXWtrEjzVAq/go-todocounter.(*todoCounter).Increment(0xc004ee7230, 0xc000000001)
        /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmQNQhNmY4STU1MURjH9vYEMpx2ncMS4gbwxXWtrEjzVAq/go-todocounter/counter.go:86 +0xaf
gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht.(*dhtQueryRunner).addPeerToQuery(0xc0009bac60, 0xc0093346c0, 0x22)
        /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht/query.go:200 +0x109
gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht.(*dhtQueryRunner).Run(0xc0009bac60, 0x19c8e40, 0xc006d2f880, 0xc004d3e5a0, 0x3, 0x3, 0x0, 0x0, 0x0)
        /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht/query.go:136 +0x143
gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht.(*dhtQuery).Run(0xc004d3e600, 0x19c8e40, 0xc006d2f880, 0xc004d3e5a0, 0x3, 0x3, 0x0, 0x0, 0x0)
        /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht/query.go:70 +0x102
gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht.(*IpfsDHT).GetClosestPeers.func2(0xc00b685560, 0xc002f642a0, 0xc004d3e600, 0x19c8ec0, 0xc00b685500, 0xc004d3e5a0, 0x3, 0x3, 0xc004eb7500, 0x22)
        /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht/lookup.go:96 +0xdb
created by gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht.(*IpfsDHT).GetClosestPeers
        /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht/lookup.go:92 +0x2f6

my system is different:

go-ipfs version: 0.4.19-
Repo version: 7
System version: amd64/linux
Golang version: go1.11.5

maybe this helps investigating the issue

ligi on 20 Mar 2019

This has been fixed in the latest go-libp2p-kad-dht (https://github.com/ipfs/go-ipfs/pull/6109).

Stebalien on 20 Mar 2019

👍2

Is there any workaround or is waitinbg for a new release of go-ipfs required?

Error: unexpected EOF                                                                                 
Error: unexpected EOF                                                                                   
Mar 26 17:51:36 sedric ipfs[22945]: panic: counter already signaled done. use a new counter.                                                                  Mar 26 17:51:36 sedric ipfs[22945]: goroutine 1697054 [running]:                                                                                              
Mar 26 17:51:36 sedric ipfs[22945]: gx/ipfs/QmQNQhNmY4STU1MURjH9vYEMpx2ncMS4gbwxXWtrEjzVAq/go-todocounter.(*todoCounter).Increment(0xc002f06600, 0xc000000001)
Mar 26 17:51:36 sedric ipfs[22945]:         /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmQNQhNmY4STU1MURjH9vYEM
px2ncMS4gbwxXWtrEjzVAq/go-todocounter/counter.go:86 +0xaf                                                       
Mar 26 17:51:36 sedric ipfs[22945]: gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht.(*dhtQueryRunner).addPeerToQuery(0xc0001adc30, 0x
c0138e5440, 0x22)                                                                                             
Mar 26 17:51:36 sedric ipfs[22945]:         /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJi
kWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht/query.go:200 +0x109                                                                                                 
Mar 26 17:51:36 sedric ipfs[22945]: gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht.(*dhtQueryRunner).Run(0xc0001adc30, 0x19c8e40, 0x
c0007f5340, 0xc001e11380, 0x3, 0x3, 0x0, 0x0, 0x0)                                                                           
Mar 26 17:51:36 sedric ipfs[22945]:         /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJi
kWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht/query.go:136 +0x143                                                  
Mar 26 17:51:36 sedric ipfs[22945]: gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht.(*dhtQuery).Run(0xc001e11410, 0x19c8e40, 0xc0007f
5340, 0xc001e11380, 0x3, 0x3, 0x0, 0x0, 0x0)                                                                                                                 
Mar 26 17:51:36 sedric ipfs[22945]:         /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJi
kWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht/query.go:70 +0x102                                                   
Mar 26 17:51:36 sedric ipfs[22945]: gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht.(*IpfsDHT).GetClosestPeers.func2(0xc005945c80, 0x
c0016f5fa0, 0xc001e11410, 0x19c8ec0, 0xc005945c20, 0xc001e11380, 0x3, 0x3, 0xc0134b7410, 0x22)                  
Mar 26 17:51:36 sedric ipfs[22945]:         /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJi
kWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht/lookup.go:96 +0xdb
Mar 26 17:51:36 sedric ipfs[22945]: created by gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJikWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht.(*IpfsDHT).GetClosestPeers
Mar 26 17:51:36 sedric ipfs[22945]:         /home/steb/projects/go/src/github.com/ipfs/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmdR6WN3TUEAVQ9KWE2UiFJi
kWTbUvgBJay6mjB4yUJebq/go-libp2p-kad-dht/lookup.go:92 +0x2f6

go-ipfs version: 0.4.19-
Repo version: 7
System version: amd64/linux
Golang version: go1.11.5

Debian GNU/Linux buster/sid

Mikaela on 26 Mar 2019

For now, you'll have to build the latest go-ipfs master. The recommended way is to:

Clone the source.
Run make build
Install the ./cmd/ipfs/ipfs binary.

Alternatively, you should now now be able to just run GO111MODULE=on go get github.com/ipfs/go-ipfs@master (although this will give you a slightly non-standard build).

Stebalien on 27 Mar 2019

👍2

Simpler: