Go-ipfs: Daemon panic: concurrent write to websocket connection

[Kycklingar]

Version information:

go-ipfs version: 0.4.20-rc1-495f12c02
Repo version: 7
System version: amd64/linux
Golang version: go1.12.2

Type:

panic

Description:

Daemon is ready
panic: concurrent write to websocket connection

goroutine 77927234 [running]:
github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc005d79740, 0xc0112c2101, 0x0, 0x0, 0x0, 0xc00264cd10, 0xc00264cd10)
    /home/kyck/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:591 +0x77a
github.com/gorilla/websocket.(*messageWriter).Close(0xc005d79740, 0x155ea80, 0xc00264cd28)
    /home/kyck/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:709 +0x56
github.com/gorilla/websocket.(*Conn).prepWrite(0xc002691e40, 0x2, 0x10000c00ae47800, 0x0)
    /home/kyck/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:459 +0x20d
github.com/gorilla/websocket.(*Conn).NextWriter(0xc002691e40, 0x2, 0x8f30e9, 0xc00025e350, 0x10, 0x92658d)
    /home/kyck/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:494 +0x39
github.com/gorilla/websocket.(*Conn).WriteMessage(0xc002691e40, 0x2, 0xc0045b7000, 0x14, 0x1000, 0x10, 0xc012dfce00)
    /home/kyck/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:755 +0x73
github.com/libp2p/go-ws-transport.(*Conn).Write(0xc005d79530, 0xc0045b7000, 0x14, 0x1000, 0xc00264ceb8, 0x8f3978, 0x10)
    /home/kyck/go/pkg/mod/github.com/libp2p/[email protected]/conn.go:74 +0x59
bufio.(*Writer).Flush(0xc016ea16c0, 0xc016ea16c0, 0xc011bbef60)
    /usr/local/go/src/bufio/bufio.go:590 +0x7b
github.com/multiformats/go-multistream.delimWriteBuffered(0x7feef2b2b208, 0xc005d79680, 0xc011bbef60, 0x12, 0x12, 0x1, 0x0)
    /home/kyck/go/pkg/mod/github.com/multiformats/[email protected]/multistream.go:64 +0xb9
github.com/multiformats/go-multistream.handshake.func1(0xc0072fdc20, 0x7feef2b2b150, 0xc005d79680)
    /home/kyck/go/pkg/mod/github.com/multiformats/[email protected]/client.go:49 +0xab
created by github.com/multiformats/go-multistream.handshake
    /home/kyck/go/pkg/mod/github.com/multiformats/[email protected]/client.go:48 +0x84

[Stebalien]

Dammit! I thought that was fixed.

[Stebalien]

Are you seeing any other stack traces? I'm having trouble reproducing this.

---

I'm pretty sure this isn't a concurrent write/close issue. It _looks_ like a concurrent write issue when negotiating the security transport but I'm not sure how this is possible.

[inetic]

For what it's worth, we're seeing this. But that's IPFS v0.4.19 and on Android.

[Stebalien]

Thanks. Unfortunately, I'm still going to need to do some work to figure out what's actually causing this.

[Kycklingar]

Are you seeing any other stack traces? I'm having trouble reproducing this.

Nope, that's all I got. It's only ever happened twice, both on the same day.

[songjiayang]

@Kycklingar Same with me, no panic now.

[Stebalien]

Interesting... I wonder if this can only be triggered by a non-ipfs websocket client (someone may have been scanning).

[songjiayang]

@Stebalien seem look like it , but ipfs node should not panic.

[Stebalien]

I agree. I'm just trying to reproduce it so I can fix it.

[Stebalien]

@inetic can you reproduce this reliably? Could you try building https://github.com/ipfs/go-ipfs/pull/6263?

[hsn10]

I have this crash daily, running with _--enable-pubsub-experiment --enable-namesys-pubsub --enable-mplex-experiment_

go-ipfs version: 0.4.22-
panic: concurrent write to websocket connection

goroutine 42561526 [running]:
github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc00dad2540, 0x1, 0x0, 0x0, 0x0, 0x1, 0xc00c5fb900)
        pkg/mod/github.com/gorilla/[email protected]/conn.go:591 +0x781
github.com/gorilla/websocket.(*messageWriter).Close(0xc00dad2540, 0x11f8649, 0x3)
        pkg/mod/github.com/gorilla/[email protected]/conn.go:709 +0x5d
github.com/gorilla/websocket.(*Conn).prepWrite(0xc00740adc0, 0x2, 0x8aa4bd8, 0x3f)
        pkg/mod/github.com/gorilla/[email protected]/conn.go:459 +0x214
github.com/gorilla/websocket.(*Conn).NextWriter(0xc00740adc0, 0x2, 0xc01d5dafc0, 0x8aa4bd8, 0x0, 0x1000000014e1fc0)
        pkg/mod/github.com/gorilla/[email protected]/conn.go:494 +0x40
github.com/gorilla/websocket.(*Conn).WriteMessage(0xc00740adc0, 0x2, 0xc01d5dafc0, 0x20, 0x40, 0x10, 0xc018931de0)
        pkg/mod/github.com/gorilla/[email protected]/conn.go:755 +0x7a
github.com/libp2p/go-ws-transport.(*Conn).Write(0xc00dd0ff50, 0xc01d5dafc0, 0x20, 0x40, 0x10946c0, 0x11d1be0, 0x2ab4620)
        pkg/mod/github.com/libp2p/[email protected]/conn.go:74 +0x60
bytes.(*Buffer).WriteTo(0xc00db3a6c0, 0x79393c8, 0xc00dad2450, 0x2ab4620, 0xc00db3a6c0, 0x8ba401)
        /usr/lib/go/src/bytes/buffer.go:242 +0xbf
io.copyBuffer(0x79393c8, 0xc00dad2450, 0x14c4920, 0xc00db3a6c0, 0x0, 0x0, 0x0, 0xc018bd7370, 0x438d5d, 0x1382308)
        /usr/lib/go/src/io/io.go:384 +0x355
io.Copy(...)
        /usr/lib/go/src/io/io.go:364
github.com/multiformats/go-multistream.SelectProtoOrFail.func1(0x1200771, 0xa, 0x7d5a580, 0xc00dad2450, 0xc00ec41d40)
        pkg/mod/github.com/multiformats/[email protected]/client.go:27 +0x16b
created by github.com/multiformats/go-multistream.SelectProtoOrFail
        pkg/mod/github.com/multiformats/[email protected]/client.go:23 +0x9c

[MatthewSteeples]

I also see this crash just running ipfs daemon --enable-pubsub-experiment --enable-namesys-pubsub

go-ipfs version: 0.4.22-
Repo version: 7
System version: arm/linux
Golang version: go1.12.7

Running on a Raspberry Pi 3B+

[Stebalien]

@MatthewSteeples could you try running go-ipfs master? We believe this may be fixed.

[bonedaddy]

This issue is also happening with general usage of libp2p websockets. I'm getting this issue outside of go-ipfs as well. https://github.com/libp2p/go-ws-transport/issues/56 is my bug report on go-ws-transport

This is a bug within go-ws-transport and makes websockets completely unusable. I have a custom node that's crashing every 30 minutes due to bug.

OP opened report using go-ws-transport v0.0.2 and it's still present in the latest master branch

[Stebalien]

Damn. Our gateways were "fixed" by updating but apparently we need to figure out where the issue actually is.

[ander7agar]

I have the same error. It happened for the first time last Sunday, it had version 0.4.21. I updated to the latest version 0.4.22, and so far it has continued to happen 2 to 3 times per day.

[Mithgol]

It starts happening more often, almost every hour.

[KireinaHoro]

Confirms for 0.4.22 on Windows 10, installed from chocolatey. Encountered that over 4 times in a few hours...

Daemon is started with no other arguments. Seems like it should happen more frequently if the node's serving data.

[pstroinski]

Same here, happens once or twice a day, ipfs 0.4.22 on ubuntu 18.04:
panic: concurrent write to websocket connection
goroutine 36495687 [running]:
github.com/gorilla/websocket.(messageWriter).flushFrame(0xbc615c0, 0x160d2801, 0x0, 0x0, 0x0, 0x8f06390, 0x0)
pkg/mod/github.com/gorilla/[email protected]/conn.go:591 +0x6e4
github.com/gorilla/websocket.(messageWriter).Close(0xbc615c0, 0x17ffdc2c, 0x844390f)
pkg/mod/github.com/gorilla/[email protected]/conn.go:709 +0x4c
github.com/gorilla/websocket.(Conn).prepWrite(0x21241380, 0x2, 0x0, 0x77)
pkg/mod/github.com/gorilla/[email protected]/conn.go:459 +0x1aa
github.com/gorilla/websocket.(Conn).NextWriter(0x21241380, 0x2, 0x8f06390, 0x2f, 0x0, 0x1000b01)
pkg/mod/github.com/gorilla/[email protected]/conn.go:494 +0x2d
github.com/gorilla/websocket.(Conn).WriteMessage(0x21241380, 0x2, 0x1a765dc0, 0x22, 0x40, 0xa026aec, 0xe740c918)
pkg/mod/github.com/gorilla/[email protected]/conn.go:755 +0x4f
github.com/libp2p/go-ws-transport.(Conn).Write(0xbc612a0, 0x1a765dc0, 0x22, 0x40, 0x8b0d7c0, 0x8c10080, 0xe740c918)
pkg/mod/github.com/libp2p/[email protected]/conn.go:74 +0x42
bytes.(*Buffer).WriteTo(0xbc8d520, 0xe7456828, 0xbc61500, 0xe740c918, 0xbc8d520, 0xe7456801, 0x1)
/usr/lib/go/src/bytes/buffer.go:242 +0x9e
io.copyBuffer(0xe7456828, 0xbc61500, 0x8f05fd0, 0xbc8d520, 0x0, 0x0, 0x0, 0xe71ceb0, 0xe71ceb8, 0x80a36f9, ...)
/usr/lib/go/src/io/io.go:384 +0x2d6
io.Copy(...)
/usr/lib/go/src/io/io.go:364
github.com/multiformats/go-multistream.SelectProtoOrFail.func1(0x8c318ae, 0xc, 0xe74567a8, 0xbc61500, 0xbf981c0)
pkg/mod/github.com/multiformats/[email protected]/client.go:27 +0x136
created by github.com/multiformats/go-multistream.SelectProtoOrFail
pkg/mod/github.com/multiformats/[email protected]/client.go:23 +0x71

[auronsan]

same with me too xD . I thought only me got this ipfs 0.4.22

ipfs_1        | Changing user to ipfs
ipfs_1        | ipfs version 0.4.22
ipfs_1        | Found IPFS fs-repo at /data/ipfs
ipfs_1        | Initializing daemon...
ipfs_1        | go-ipfs version: 0.4.22-
ipfs_1        | Repo version: 7
ipfs_1        | System version: amd64/linux
ipfs_1        | Golang version: go1.12.7
ipfs_1        | Swarm listening on /ip4/127.0.0.1/tcp/4001
ipfs_1        | Swarm listening on /ip4/172.25.0.2/tcp/4001
ipfs_1        | Swarm listening on /p2p-circuit
ipfs_1        | Swarm announcing /ip4/127.0.0.1/tcp/4001
ipfs_1        | Swarm announcing /ip4/172.25.0.2/tcp/4001
ipfs_1        | API server listening on /ip4/0.0.0.0/tcp/5001
ipfs_1        | WebUI: http://0.0.0.0:5001/webui
ipfs_1        | Gateway (readonly) server listening on /ip4/0.0.0.0/tcp/8080
ipfs_1        | Daemon is ready
ipfs_1        | panic: concurrent write to websocket connection
ipfs_1        | 
ipfs_1        | goroutine 4704739 [running]:
ipfs_1        | github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc008b83a10, 0xc007486c01, 0x0, 0x0, 0x0, 0x1, 0xc007486cf0)
ipfs_1        |     /go/pkg/mod/github.com/gorilla/[email protected]/conn.go:591 +0x77a
ipfs_1        | github.com/gorilla/websocket.(*messageWriter).Close(0xc008b83a10, 0x1, 0x1)
ipfs_1        |     /go/pkg/mod/github.com/gorilla/[email protected]/conn.go:709 +0x56
ipfs_1        | github.com/gorilla/websocket.(*Conn).prepWrite(0xc0046ae6e0, 0x2, 0xc007486d58, 0x982cc5)
ipfs_1        |     /go/pkg/mod/github.com/gorilla/[email protected]/conn.go:459 +0x20d
ipfs_1        | github.com/gorilla/websocket.(*Conn).NextWriter(0xc0046ae6e0, 0x2, 0x1, 0xc007486e10, 0x97af39, 0x100000000000008)
ipfs_1        |     /go/pkg/mod/github.com/gorilla/[email protected]/conn.go:494 +0x39
ipfs_1        | github.com/gorilla/websocket.(*Conn).WriteMessage(0xc0046ae6e0, 0x2, 0xc00d99cdc0, 0x22, 0x40, 0x10, 0xc00e387b20)
ipfs_1        |     /go/pkg/mod/github.com/gorilla/[email protected]/conn.go:755 +0x73
ipfs_1        | github.com/libp2p/go-ws-transport.(*Conn).Write(0xc008b83680, 0xc00d99cdc0, 0x22, 0x40, 0x163dfa0, 0x1795ea0, 0x7f8f66e2b380)
ipfs_1        |     /go/pkg/mod/github.com/libp2p/[email protected]/conn.go:74 +0x59
ipfs_1        | bytes.(*Buffer).WriteTo(0xc008696a20, 0x7f8f66e3a9a8, 0xc008b83980, 0x7f8f66e2b380, 0xc008696a20, 0xe25e01)
ipfs_1        |     /usr/local/go/src/bytes/buffer.go:242 +0xb8
ipfs_1        | io.copyBuffer(0x7f8f66e3a9a8, 0xc008b83980, 0x1a98d00, 0xc008696a20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x9a59d6)
ipfs_1        |     /usr/local/go/src/io/io.go:384 +0x34e
ipfs_1        | io.Copy(...)
ipfs_1        |     /usr/local/go/src/io/io.go:364
ipfs_1        | github.com/multiformats/go-multistream.SelectProtoOrFail.func1(0x17c705d, 0xc, 0x7f8f66e3a8f0, 0xc008b83980, 0xc0011b3260)
ipfs_1        |     /go/pkg/mod/github.com/multiformats/[email protected]/client.go:27 +0x164
ipfs_1        | created by github.com/multiformats/go-multistream.SelectProtoOrFail
ipfs_1        |     /go/pkg/mod/github.com/multiformats/[email protected]/client.go:23 +0x95

I think this is reference from gorilla websocket
https://github.com/googollee/go-socket.io/issues/182

[jaller94]

Version information:

ipfs version 0.4.22 from the Manjaro Linux repo.

go-ipfs version: 0.4.22-4e981576b-dirty
Repo version: 7
System version: amd64/linux
Golang version: go1.12.8

Description:

1. I ran ipfs daemon.
2. I ran ipfs mount.
3. Within 2 hours, the IPFS daemon crashed. This left the mount points in the status "/ipns: Transport endpoint is not connected".

Stacktrace:

panic: concurrent write to websocket connection

goroutine 2156447 [running]:
github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc001605530, 0x1, 0x0, 0x0, 0x0, 0xc0001325ee, 0xc00732ccf0)
    /build/go-ipfs/src/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:591 +0x77c
github.com/gorilla/websocket.(*messageWriter).Close(0xc001605530, 0x558fde865d9a, 0x10)
    /build/go-ipfs/src/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:709 +0x58
github.com/gorilla/websocket.(*Conn).prepWrite(0xc001c86dc0, 0x2, 0x0, 0xc00732cd00)
    /build/go-ipfs/src/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:459 +0x20f
github.com/gorilla/websocket.(*Conn).NextWriter(0xc001c86dc0, 0x2, 0x7f3828b036f8, 0xc00ab750e0, 0xc00ab750e0, 0x1007f3828b036f8)
    /build/go-ipfs/src/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:494 +0x3b
github.com/gorilla/websocket.(*Conn).WriteMessage(0xc001c86dc0, 0x2, 0xc02075d500, 0x22, 0x40, 0x10, 0xc002116500)
    /build/go-ipfs/src/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:755 +0x75
github.com/libp2p/go-ws-transport.(*Conn).Write(0xc0016052f0, 0xc02075d500, 0x22, 0x40, 0x558fdfb012c0, 0x558fdfc59200, 0x7f3828ac68c0)
    /build/go-ipfs/src/go-ipfs/vendor/github.com/libp2p/go-ws-transport/conn.go:74 +0x5b
bytes.(*Buffer).WriteTo(0xc005c99800, 0x7f38280f0460, 0xc0016054a0, 0x7f3828ac68c0, 0xc005c99800, 0x558fded14b01)
    /usr/lib/go/src/bytes/buffer.go:242 +0xba
io.copyBuffer(0x7f38280f0460, 0xc0016054a0, 0x558fdfca2a20, 0xc005c99800, 0x0, 0x0, 0x0, 0x558fdee010ce, 0x2, 0x2)
    /usr/lib/go/src/io/io.go:384 +0x350
io.Copy(...)
    /usr/lib/go/src/io/io.go:364
github.com/multiformats/go-multistream.SelectProtoOrFail.func1(0x558fdf3d17fd, 0xc, 0x7f38280f03a8, 0xc0016054a0, 0xc009b9d5c0)
    /build/go-ipfs/src/go-ipfs/vendor/github.com/multiformats/go-multistream/client.go:27 +0x166
created by github.com/multiformats/go-multistream.SelectProtoOrFail
    /build/go-ipfs/src/go-ipfs/vendor/github.com/multiformats/go-multistream/client.go:23 +0x97

[aschmahmann]

I was running into this issue very frequently on Windows 10 with go-ipfs 0.4.22 (e.g. more than once per hour), but after upgrading to master this occurs much less frequently. While the issue is not actually fixed, it might be an easy stop gap for those of us having issues.

go-ipfs version: 0.5.0-dev-410e7e878-dirty
Repo version: 7
System version: amd64/windows
Golang version: go1.13

[djdv]

Adding my logs to this thread in case they're useful.
Context: started the daemon, added 2 directories (~80GB each), wanted to make sure they were pinned along with a 3rd hash, so I ran the following which eventually resulted in a panic.

The dirty portions of the commit relate to changes in badger and reuseport (to build on this platform). Nothing else should be out of the ordinary.
(ref: https://github.com/ipfs/go-ipfs/issues/6642#issue-493470642)

go-ipfs version: 0.5.0-dev-410e7e878-dirty
Repo version: 7
System version: amd64/solaris
Golang version: go1.12.10

ipfs pin add --progress X Y Z
panic: concurrent write to websocket connection

goroutine 2952978 [running]:
github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc0093b90e0, 0xc0093b9501, 0x0, 0x0, 0x0, 0xfffffd7fef2e4d98, 0x0)
        src/github.com/ipfs/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:610 +0x62f
github.com/gorilla/websocket.(*messageWriter).Close(0xc0093b90e0, 0x40, 0xc0009cfd00)
        src/github.com/ipfs/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:724 +0x62
github.com/gorilla/websocket.(*Conn).beginMessage(0xc01a2f8f20, 0xc0093b9530, 0x2, 0xc0009cfda0, 0x786589)
        src/github.com/ipfs/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:473 +0x244
github.com/gorilla/websocket.(*Conn).NextWriter(0xc01a2f8f20, 0x2, 0xfffffd7fee3caf28, 0x38, 0x200, 0x1c0)
        src/github.com/ipfs/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:513 +0x53
github.com/gorilla/websocket.(*Conn).WriteMessage(0xc01a2f8f20, 0x2, 0xc016a87fc0, 0x20, 0x40, 0x10, 0xc00bc4bc50)
        src/github.com/ipfs/go-ipfs/vendor/github.com/gorilla/websocket/conn.go:766 +0x73
github.com/libp2p/go-ws-transport.(*Conn).Write(0xc00a2a3b90, 0xc016a87fc0, 0x20, 0x40, 0x10924a0, 0x11d3000, 0xfffffd7feecc90e8)
        src/github.com/ipfs/go-ipfs/vendor/github.com/libp2p/go-ws-transport/conn.go:74 +0x59
bytes.(*Buffer).WriteTo(0xc0093b9440, 0xfffffd7feeb91958, 0xc00b200120, 0xfffffd7feecc90e8, 0xc0093b9440, 0x927401)
        /usr/lib/golang/1.12/src/bytes/buffer.go:242 +0xb8
io.copyBuffer(0xfffffd7feeb91958, 0xc00b200120, 0x14cc680, 0xc0093b9440, 0x0, 0x0, 0x0, 0xc00bc4b340, 0x437cd6, 0x1382758)
        /usr/lib/golang/1.12/src/io/io.go:384 +0x34e
io.Copy(...)
        /usr/lib/golang/1.12/src/io/io.go:364
github.com/multiformats/go-multistream.SelectProtoOrFail.func1(0x1201aa0, 0xa, 0xfffffd7feeb918c0, 0xc00b200120, 0xc01bf6e240)
        src/github.com/ipfs/go-ipfs/vendor/github.com/multiformats/go-multistream/client.go:27 +0x164
created by github.com/multiformats/go-multistream.SelectProtoOrFail
        src/github.com/ipfs/go-ipfs/vendor/github.com/multiformats/go-multistream/client.go:23 +0x95
Error: unexpected EOF

[Stebalien]

@vyzo going to cook up some patches to help debug this.

[vyzo]

https://github.com/libp2p/go-ws-transport/issues/56 adds instrumentation (and a lock) to catch concurrent writes and print stack traces.
Can you try it out and report?

[Stebalien]

@vyzo could you create a branch of go-ipfs that updates go-ws-transport?

[vyzo]

I guess I could do that.

[djdv]

Follow up:
(still go-ipfs 410e7e878, but with https://github.com/libp2p/go-ws-transport/commit/e2dded5874d31da646da3d4a4176d90e0df00fb8 vendored)
Started the daemon, tried accessing hashes of mine through the gateway, eventually this happened.

Edit:
gist with more logging after running for more time.

https://gist.github.com/djdv/1d6cd8c552d2d2983cfe11f7918058d2

[vyzo]

I just pushed a debug/issue-6197 branch which has the patch too.

[UltraNote]

go-ipfs version: 0.4.22 and go-ipfs version: 0.5.0-dev

[zquestz]

Also seeing this on my node:

panic: concurrent write to websocket connection goroutine 154743888 [running]: github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc02f8aeb40, 0x20613ed80877501, 0x0, 0x0, 0x0, 0xc01822a4d0, 0xdc2fc7) /go/pkg/mod/github.com/gorilla/[email protected]/conn.go:591 +0x77a github.com/gorilla/websocket.(*messageWriter).Close(0xc02f8aeb40, 0x40, 0xc01822a548) /go/pkg/mod/github.com/gorilla/[email protected]/conn.go:709 +0x56 github.com/gorilla/websocket.(*Conn).prepWrite(0xc00a9cdce0, 0x2, 0x0, 0x0) /go/pkg/mod/github.com/gorilla/[email protected]/conn.go:459 +0x20d github.com/gorilla/websocket.(*Conn).NextWriter(0xc00a9cdce0, 0x2, 0x7f0fdc9430e8, 0x20, 0x200, 0x100) /go/pkg/mod/github.com/gorilla/[email protected]/conn.go:494 +0x39 github.com/gorilla/websocket.(*Conn).WriteMessage(0xc00a9cdce0, 0x2, 0xc030ee7900, 0x22, 0x40, 0x10, 0xc01adfb0d0) /go/pkg/mod/github.com/gorilla/[email protected]/conn.go:755 +0x73 github.com/libp2p/go-ws-transport.(*Conn).Write(0xc02f8ae840, 0xc030ee7900, 0x22, 0x40, 0x163dfa0, 0x1795ea0, 0x7f10199f5cc0) /go/pkg/mod/github.com/libp2p/[email protected]/conn.go:74 +0x59 bytes.(*Buffer).WriteTo(0xc02c3e8150, 0x7f101805ca88, 0xc02f8aeab0, 0x7f10199f5cc0, 0xc02c3e8150, 0xe25e01) /usr/local/go/src/bytes/buffer.go:242 +0xb8 io.copyBuffer(0x7f101805ca88, 0xc02f8aeab0, 0x1a98d00, 0xc02c3e8150, 0x0, 0x0, 0x0, 0xc01ad51500, 0x9a59d6, 0x1946a18) /usr/local/go/src/io/io.go:384 +0x34e io.Copy(...) /usr/local/go/src/io/io.go:364 github.com/multiformats/go-multistream.SelectProtoOrFail.func1(0x17c705d, 0xc, 0x7f101805c9f0, 0xc02f8aeab0, 0xc001bb30e0) /go/pkg/mod/github.com/multiformats/[email protected]/client.go:27 +0x164 created by github.com/multiformats/go-multistream.SelectProtoOrFail /go/pkg/mod/github.com/multiformats/[email protected]/client.go:23 +0x95

[vyzo]

Can you try running with the debug/issue-6197 branch? It should eliminate the panic and also print stack traces to help us debug the issue.

[zquestz]

Can you post an image on docker hub? If so I would happily test it. Right now I am just using 0.5.0-dev on master.On Nov 19, 2019 12:28 PM, vyzo [e-mail] wrote:Can you try running with the debug/issue-6197 branch? It should eliminate the panic and also print stack traces to help us debug the issue.

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe.

[jpeletier]

@zquestz Here you go: epiclabs/go-ipfs:issue-6197 https://hub.docker.com/repository/docker/epiclabs/go-ipfs

[jpeletier]

@vyzo Here is a stack trace of issue-6197 that printed out after some time running:

WebUI: http://127.0.0.1:5001/webui
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
goroutine 159924 [running]:
runtime/debug.Stack(0xc0007e9d08, 0x9e3b61, 0x1a60e88)
    /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
    /usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/libp2p/go-ws-transport.(*Conn).Write.func1(0xc0019091c0)
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/conn_native.go:82 +0x3d
github.com/libp2p/go-ws-transport.(*Conn).Write(0xc0019091c0, 0xc001fb5500, 0x34, 0x40, 0x34, 0x0, 0x0)
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/conn_native.go:90 +0x18b
github.com/libp2p/go-libp2p-secio.(*etmWriter).WriteMsg(0xc0019093c0, 0xc00349af50, 0x10, 0x10, 0x0, 0x0)
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/rw.go:64 +0x2ee
github.com/libp2p/go-libp2p-secio.readWriteMsg.func1(0xc000e33620, 0x7f8991ec8418, 0xc001cb54a0, 0xc00349af50, 0x10, 0x10)
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/rw.go:252 +0x4f
created by github.com/libp2p/go-libp2p-secio.readWriteMsg
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/rw.go:251 +0xab
goroutine 159804 [running]:
runtime/debug.Stack(0x9f6391, 0xc0004c98d8, 0x2)
    /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
    /usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/libp2p/go-ws-transport.(*Conn).Close(0xc0019091c0, 0x0, 0x0)
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/conn_native.go:98 +0x1cd
github.com/libp2p/go-libp2p-secio.(*secureSession).runHandshake(0xc0007ed800, 0x1beb560, 0xc0012f1620, 0x0, 0x0)
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/protocol.go:131 +0x29c
github.com/libp2p/go-libp2p-secio.newSecureSession(0x1beb560, 0xc0012f0c00, 0xc000042210, 0x22, 0x1bf3240, 0xc00093ed20, 0x7f8991e6c508, 0xc002663da0, 0xc002127950, 0x22, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/protocol.go:95 +0x1c5
github.com/libp2p/go-libp2p-secio.(*Transport).SecureOutbound(0xc0001f0360, 0x1beb560, 0xc0012f0c00, 0x7f8991e6c508, 0xc002663da0, 0xc002127950, 0x22, 0xc0001f0360, 0x0, 0x0, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/transport.go:40 +0x9b
github.com/libp2p/go-conn-security-multistream.(*SSMuxer).SecureOutbound(0xc0001f2190, 0x1beb560, 0xc0012f0c00, 0x7f8991e6c508, 0xc002663da0, 0xc002127950, 0x22, 0x17a5100, 0x1818560, 0xc0004c9a00, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/ssms.go:56 +0xf2
github.com/libp2p/go-libp2p-transport-upgrader.(*Upgrader).setupSecurity(0xc000214040, 0x1beb560, 0xc0012f0c00, 0x7f8991e6c508, 0xc002663da0, 0xc002127950, 0x22, 0x1bcf500, 0xc00093c6e0, 0x1c011e0, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/upgrader.go:108 +0x10c
github.com/libp2p/go-libp2p-transport-upgrader.(*Upgrader).upgrade(0xc000214040, 0x1beb560, 0xc0012f0c00, 0x1bee720, 0xc0002d2038, 0x1bfd8e0, 0xc002663da0, 0xc002127950, 0x22, 0x0, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/upgrader.go:86 +0x29e
github.com/libp2p/go-libp2p-transport-upgrader.(*Upgrader).UpgradeOutbound(0xc000214040, 0x1beb560, 0xc0012f0c00, 0x1bee720, 0xc0002d2038, 0x1bfd8e0, 0xc002663da0, 0xc002127950, 0x22, 0x0, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/upgrader.go:57 +0xcf
github.com/libp2p/go-ws-transport.(*WebsocketTransport).Dial(0xc0002d2038, 0x1beb560, 0xc0012f0c00, 0x1c011e0, 0xc0020cb280, 0xc002127950, 0x22, 0xc002262580, 0xc0028f9450, 0xdf84755bf, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/websocket.go:63 +0xed
github.com/libp2p/go-libp2p-swarm.(*Swarm).dialAddr(0xc00091f080, 0x1beb560, 0xc0012f0c00, 0xc002127950, 0x22, 0x1c011e0, 0xc0020cb280, 0x2790e60, 0x9e330a, 0xc0012f0c00, ...)
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/swarm_dial.go:462 +0x219
github.com/libp2p/go-libp2p-swarm.(*dialLimiter).executeDial(0xc000167e50, 0xc002414a80)
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/limiter.go:218 +0x1a9
created by github.com/libp2p/go-libp2p-swarm.(*dialLimiter).addCheckFdLimit
    /home/jm/go/pkg/mod/github.com/libp2p/[email protected]/limiter.go:168 +0x5fc

[lazyweirdo]

Same issue here, go-ipfs 0.4.22 , swarm, api and read only gateway open ...

[asiniscalchi]

same here ...
gp-ipfs 0.4.22 and 0.4.18

[Stebalien]

This should be fixed by https://github.com/ipfs/go-ipfs/pull/6804.