Discord.py: [rewrite] Websocket issue: asyncio.base_futures.InvalidStateError: invalid state

Created on 18 Mar 2019 · 17Comments · Source: Rapptz/discord.py

Summary

There seems to be an issue with the websockets library which is causing this error. Even though this is a problem in a different library, I feel like discord.py should gracefully handle it and either retry the request or something, as it's completely crashing my bot.

Basically, my bot runs on clusters which are Docker containers. My bot uses about 10 clusters each responsible for 3 shards. On 1 cluster out of the 10, this problem constantly happens. It happens about once every 30 minutes, which takes down the process, and then the cluster respawns.

Something interesting to note: I edited the number of clusters and shards to spawn so it would force the distribution to change, and every time I change it, this problem always happens on the cluster with the shard that is responsible for the support server of my bot. Now, a few days ago, I bought the $25 "game developer fee" on my bot and picked my support server as the target server to receive the perks, so maybe this is related, but I'm not sure. I just know that when I change the cluster distribution, the 1 cluster which is affected happens to be where the game developer perks are on.

Reproduction Steps

Explained above.

Expected Results

No error

Actual Results

Checklist

[x] I have searched the open issues for duplicates.
[x] I have shown the entire traceback, if possible.
[x] I have removed my token from display, if visible.

System Information

discord.py version: Latest rewrite version at the time of this issue; i'm not sure what the version number it is, but it's installed on a fresh docker instance with git+https://github.com/Rapptz/discord.py@rewrite#egg=discord.py[voice] in the requirements.txt file.
Python version: 3.6
Operating system: Ubuntu 16.04.5 LTS

dep bug

Source

xomien

All 17 comments

Hello, so currently my entire bot with 33k+ servers has been offline for days because of this issue as it keeps making 1 cluster out of 10 crash, so then my token gets reset daily. I cannot run the bot with this issue and I don't know what I can do to fix it. I tried editing d.py source locally to suppress the error, but it leads to more problems, so I reverted it. I can supply all the information needed to help debug this issue for Danny, but I'm just asking for some help and opinions on what I should do as this is pretty urgent for my bot.

xomien on 20 Mar 2019

Do you have anything that may be changing how your event loop operates, like uvloop, in use?

Gorialis on 20 Mar 2019

I don't do anything to the event loop

xomien on 20 Mar 2019

Can you use pip show websockets to show me which version of websockets you're using?

Gorialis on 20 Mar 2019

Name: websockets
Version: 7.0
Summary: An implementation of the WebSocket Protocol (RFC 6455 & 7692)
Home-page: https://github.com/aaugustin/websockets
Author: Aymeric Augustin
Author-email: aymeric.[email protected]
License: BSD
Location: /usr/local/lib/python3.6/site-packages
Requires:
Required-by: discord.py

xomien on 20 Mar 2019

Can confirm that this is also breaking on my bot with 14,800 servers.
Had to disable alot of events to just let it run.

Waspyeh on 20 Mar 2019

This exception seems to be caused by an issue rooted in websockets and/or asyncio itself. The best suggestion I can give without further investigation is to patch your websockets/protocol.py at line 441 to look like this:

if self._put_message_waiter is not None:
    if not self._put_message_waiter.done():
        self._put_message_waiter.set_result(None)
    self._put_message_waiter = None

An issue should be raised on the websockets repo since I don't think there's anything we can do about this fundamentally.

Judging by the summary though, I'm guessing this is some bad data edge case that occurs from Discord sending some incomplete or broken data down the websocket for servers that have this "game developer perks" feature enabled.

Gorialis on 20 Mar 2019

That code snippet prevents the process from dying, but the websocket continuously closes, making the cluster responsible for those shards unresponsive.
websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason

I'm not sure how I'd word the issue on the websockets repo, as I don't know much about how the websocket stuff works myself; would it be possible for someone else to make the issue?

xomien on 20 Mar 2019

It seems to me that right now the solution right now is to downgrade websockets to 6.0, i.e. pip install -U 'websockets<7.0' and then file an issue with the websockets repository.

Rapptz on 20 Mar 2019

🎉1

Getting the same websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason errors on my bot leading to clusters/the entire bot reconnecting several times per day. 32k servers if it's relevant.

CarlGroth on 20 Mar 2019

I'm no longer having any issues with my bot. @CarlGroth I had the same issue and finally managed to look into it today; the issue was I was calling too many discord API requests for one bot feature which made Discord keep kicking off my bot.

Closing since the main issue was resolved by dropping the websocket version.

Thanks Danny for finding the solution to the issue (by dropping the websocket version).

xomien on 25 Mar 2019

Note, https://github.com/aaugustin/websockets/issues/551 is probably the relevant websockets issue.

Harmon758 on 27 Mar 2019

It seems to me that right now the solution right now is to downgrade websockets to 6.0, i.e. pip install -U 'websockets<7.0' and then file an issue with the websockets repository.

can't thank you enough for the solution

fuzunspm on 23 Apr 2019

I'm pretty sure it's a bug in websockets, but not the one suggested in an earlier comment: https://github.com/aaugustin/websockets/issues/634

aaugustin on 30 Jun 2019

@aaugustin Is this still an issue with websockets==8.0.1? I've been using latest without any problems so far. Perhaps the constraint in setup.py can be lifted?

discosultan on 28 Jul 2019

Yes, I believe it's fixed in websockets ≥ 8.0. It was a cancellation bug. The only known version with this bug is 7.0.

aaugustin on 28 Jul 2019

👍1

@discosultan The dependency can't be bumped since it conflicts with the current version requirements in the library (ws >= 8.0 requires python >= 3.6, we're on python >= 3.5).

Rapptz on 29 Jul 2019

👍1

Was this page helpful?

0 / 5 - 0 ratings