Krypto-trading-bot: Bitfinex Reconnect Order Update Issue

[MiningInMySleep]

So when trading on Bitfinex, sometimes there will be a random disconnect and the bot reconnects fine.
However, orders no longer properly register on the server side.

Web UI shows orders, but quotes stay at 0/0/0.
Some digging with --debug-orders shows this:
02/14 17:05:39.546863 GW BITFINEX Warrrrning: CURL recv Error: Failed sending data to the peer.
02/14 17:05:40.506855 GW BITFINEX Quoting state changed to DISCONNECTED.
02/14 17:05:40.507010 GW BITFINEX WS Disconnected, reconnecting in 7s..
02/14 17:05:47.769692 QE Warrrrning: Unable to calculate quote, missing market data.
02/14 17:05:47.810478 GW BITFINEX Quoting state changed to CONNECTED.
02/14 17:05:47.810556 GW BITFINEX WS Streaming wallet and orders OK 0:"auth".
02/14 17:05:47.810941 GW BITFINEX WS Streaming channel 188:trades.
02/14 17:05:48.393761 GW BITFINEX WS Streaming channel 157007:book.
02/14 17:07:59.507446 DEBUG OG replace 7981870066::39568697695 [1]: 24.00000000 XRP at price 0.00003121 BTC.
02/14 17:08:36.343216 DEBUG OG replace 7981870066::39568697695 [1]: 24.00000000 XRP at price 0.00003120 BTC.
02/14 17:08:55.992559 DEBUG OG replace 7981870066::39568697695 [1]: 24.00000000 XRP at price 0.00003121 BTC.
02/14 17:09:03.370281 DEBUG OG replace 7981870066::39568697695 [1]: 24.00000000 XRP at price 0.00003120 BTC.
02/14 17:09:08.507438 DEBUG OG replace 7981870066::39568697695 [1]: 24.00000000 XRP at price 0.00003122 BTC.

When usually it should be a stream like this:
02/14 17:05:01.622751 DEBUG OG memory 2.
02/14 17:05:01.622771 DEBUG OG purge 1834480321::39568697694 [2]: 24.00000000 XRP at price 0.00003120 BTC.
02/14 17:05:01.622788 DEBUG OG memory 1.
02/14 17:05:01.683819 DEBUG OG saved 7981870066:: [0]: 24.00000000 XRP at price 0.00003120 BTC.
02/14 17:05:01.684953 DEBUG OG memory 2.
02/14 17:05:01.791596 DEBUG OG reply 7981870066::39568697695 [1]: 24.00000000 XRP at price 0.00003120 BTC.
02/14 17:05:01.793376 DEBUG OG saved 7981870066::39568697695 [1]: 24.00000000 XRP at price 0.00003120 BTC.
02/14 17:05:01.793409 DEBUG OG memory 2.

[MiningInMySleep]

Update: This same issue happened without even being disconnected. This time it's only for bids. Constant replace attempts without response from server. Asks are going through and getting a response without issue. Confirmed only asks showing up on actual exchange, while K interface shows nothing wrong.

[MiningInMySleep]

Update 2: It seems that when the disconnect happens, the current order is cancelled, but the bot still thinks it's there so it tries to replace it constantly and never realizes that it needs to submit a new order.

[MiningInMySleep]

Update 3: Increasing lifetime seems to help not being disconnected, but still ends up happening. Would be best just to have the bot realize that the orders it is trying to update don't exist anymore.

[eniware-org]

I am experiencing the same issue. Do you consider the limits for API calls? I notice that if I perform more trades the disconnect happens less frequently. Reading https://docs.bitfinex.com/docs/getting-started#section-rate-limits (JSON response {"error": "ERR_RATE_LIMIT"}) and https://github.com/bitfinexcom/bitfinex-api-node#how-many-orders-can-i-send might help for calculation of the maximum allocated rate.

[MiningInMySleep]

I've tried changing the limits, but I honestly don't think it matters. The Bitfinex API just randomly disconnects and K doesn't realize that the order has been cancelled.

[cryp2owl]

Did you guys try it on other exchanges as well? I think Bittrex is far more open when it comes to bot trading. also Kucoin.

[eniware-org]

I've tried changing the limits, but I honestly don't think it matters. The Bitfinex API just randomly disconnects and K doesn't realize that the order has been cancelled.

This was addressed to @ctubio

[eniware-org]

@ctubio, Can we expect any reaction on this issue from you?

[MiningInMySleep]

Bump @ctubio
The issue is still here and I have a feeling it's not on the BitFinex side. Can we please have this looked at?

[eniware-org]

+1

[ctubio]

thank you for the report, let me try to move my lazy ass

[MiningInMySleep]

To add to this, restarting the quoting engine will allow orders to be sent properly if the disconnect order bug happens.

[eniware-org]

https://github.com/bitfinexcom/bitfinex-api-node#order-creation-limits IS the reason. Which is not to say that the bot should not handle the situation better and restart the quoting engine automatically after a (5 minutes?) delay or at least display it in the QuoteStatus. One shall take care to limit the frequency of sending orders in order to avoid the ERR_RATE_LIMIT response from the exchange, like for example via lifetime as @MiningInMySleep has done, but this is fairly crude.

[MiningInMySleep]

https://github.com/bitfinexcom/bitfinex-api-node#order-creation-limits IS the reason. Which is not to say that the bot should not handle the situation better and restart the quoting engine automatically after a (5 minutes?) delay or at least display it in the QuoteStatus. One shall take care to limit the frequency of sending orders in order to avoid the ERR_RATE_LIMIT response from the exchange, like for example via lifetime as @MiningInMySleep has done, but this is fairly crude.

Are you actually seeing the "ERR_RATE_LIMIT" response from the exchange? Do I need to enable one of the debug options to see it? I'm testing higher and higher lifetime settings (currently at 600) and still running into disconnects with error messages related to curl that end up with a broken quoting system upon reconnect.

[eniware-org]

--lifetime=600 is 0.6s! I run on 1000-1200 and have pretty high trading volume, so I don't get disconnected. However, very occasionally (less than once per week) I am hammered by Bitfinex (e.g. when high trading volume is present on the exchange) and the QE then is not restarted. Haven't bothered debugging. It is definitely a shortcoming of the bot.
--debug-secret is what you need to debug the gateway.

[MiningInMySleep]

Running the latest commit with the WS checksum stuff and got this error looping since 2+ hours ago.

02/05 14:26:45.209077 GW BITFINEX PING TRADE BUY 0.00020000 BTC at price 37009 USD (value 7 USD).
02/05 14:33:57.378914 GW BITFINEX Warrrrning: WS Error: book out of sync, checksums failed..
02/05 14:33:57.724107 GW BITFINEX Quoting state changed to DISCONNECTED.
02/05 14:33:57.724250 GW BITFINEX WS Disconnected, reconnecting in 7s..
02/05 14:34:04.788774 QE Warrrrning: Unable to calculate quote, missing market data.
02/05 14:34:04.788898 GW BITFINEX WS Info {"event":"info","platform":{"status":1},"serverId":"504ed088-3dfd-43bc-82a9-58f99b9e2791","version":2}.
02/05 14:34:04.817816 GW BITFINEX Quoting state changed to CONNECTED.
02/05 14:34:04.818713 GW BITFINEX WS Streaming wallet and orders OK 0:"auth".
02/05 14:34:04.821138 GW BITFINEX WS Streaming channel 42:trades.
02/05 14:34:05.724127 QE Warrrrning: Unable to calculate quote, missing market data.
02/05 14:34:05.764862 GW BITFINEX WS Streaming channel 278868:book.
02/05 14:53:37.319497 GW BITFINEX Warrrrning: WS Error: book out of sync, checksums failed..
02/05 14:53:37.724147 GW BITFINEX Quoting state changed to DISCONNECTED.
02/05 14:53:37.724254 GW BITFINEX WS Disconnected, reconnecting in 7s..
02/05 14:53:44.782687 QE Warrrrning: Unable to calculate quote, missing market data.
02/05 14:53:44.782849 GW BITFINEX WS Info {"event":"info","platform":{"status":1},"serverId":"8bcd723b-0986-4d12-bfcc-cf25cf7a64c0","version":2}.
02/05 14:53:44.815345 GW BITFINEX Quoting state changed to CONNECTED.
02/05 14:53:44.815413 GW BITFINEX WS Streaming wallet and orders OK 0:"auth".
02/05 14:53:44.823596 GW BITFINEX WS Streaming channel 71564:trades.
02/05 14:53:45.724110 QE Warrrrning: Unable to calculate quote, missing market data.
02/05 14:53:46.115041 GW BITFINEX WS Streaming channel 98971:book.
02/05 14:53:46.323113 GW BITFINEX Warrrrning: WS Error: book out of sync, checksums failed..
02/05 14:53:46.724094 GW BITFINEX Quoting state changed to DISCONNECTED.
02/05 14:53:46.724203 GW BITFINEX WS Disconnected, reconnecting in 7s..
02/05 14:53:53.787649 QE Warrrrning: Unable to calculate quote, missing market data.
02/05 14:53:53.787766 GW BITFINEX WS Info {"event":"info","platform":{"status":1},"serverId":"25dff4e7-aa0d-4318-9bcc-2abefc7b6342","version":2}.
02/05 14:53:53.816502 GW BITFINEX Quoting state changed to CONNECTED.
02/05 14:53:53.816568 GW BITFINEX WS Streaming wallet and orders OK 0:"auth".
02/05 14:53:53.820056 GW BITFINEX WS Streaming channel 436571:trades.
02/05 14:53:54.724116 QE Warrrrning: Unable to calculate quote, missing market data.
02/05 14:53:55.188563 GW BITFINEX WS Streaming channel 618450:book.
02/05 14:54:29.236475 GW BITFINEX Warrrrning: WS Error: book out of sync, checksums failed..
02/05 14:54:29.725321 GW BITFINEX Quoting state changed to DISCONNECTED.
02/05 14:54:29.725410 GW BITFINEX WS Disconnected, reconnecting in 7s..
02/05 14:54:36.853696 QE Warrrrning: Unable to calculate quote, missing market data.
02/05 14:54:36.853816 GW BITFINEX WS Info {"event":"info","platform":{"status":1},"serverId":"ee11be43-fa54-42f2-b7a2-e1365ed6d35f","version":2}.
02/05 14:54:36.880279 GW BITFINEX Quoting state changed to CONNECTED.
02/05 14:54:36.880353 GW BITFINEX WS Streaming wallet and orders OK 0:"auth".
02/05 14:54:36.905629 GW BITFINEX WS Streaming channel 1036:trades.
02/05 14:54:37.096445 GW BITFINEX WS Streaming channel 301649:book.
02/05 14:54:37.291780 GW BITFINEX Warrrrning: WS Error: book out of sync, checksums failed..
02/05 14:54:37.724126 GW BITFINEX Quoting state changed to DISCONNECTED.
02/05 14:54:37.724242 GW BITFINEX WS Disconnected, reconnecting in 7s..
02/05 14:54:44.823813 QE Warrrrning: Unable to calculate quote, missing market data.
02/05 14:54:44.824570 GW BITFINEX WS Info {"event":"info","platform":{"status":1},"serverId":"d3d88ff9-5013-41c0-9183-0601450a3c9a","version":2}.
02/05 14:54:44.851480 GW BITFINEX Quoting state changed to CONNECTED.
02/05 14:54:44.851559 GW BITFINEX WS Streaming wallet and orders OK 0:"auth".
02/05 14:54:44.855088 GW BITFINEX WS Streaming channel 33748:trades.
02/05 14:54:44.980688 GW BITFINEX WS Streaming channel 551878:book.
02/05 14:56:35.257562 GW BITFINEX Warrrrning: WS Error: book out of sync, checksums failed..
02/05 14:56:35.724126 GW BITFINEX Quoting state changed to DISCONNECTED.
02/05 14:56:35.724242 GW BITFINEX WS Disconnected, reconnecting in 7s..
02/05 14:56:42.787031 QE Warrrrning: Unable to calculate quote, missing market data.

Currently at --lifetime=1200 and still getting disconnected randomly. Increasing lifetime does not seem to help at all. Upon auto-reconnect the web UI shows orders, but they don't show up on exchange and no actual trades are made until restarting the quoting engine or restarting K.

[ctubio]

@MiningInMySleep are you using the linux-x86_64 build or another?

for the whole last day i had only 1 time the book out of sync, but yours seems out of sync right after the very first check on every new reconnection

let me rereview this checksums calcs while adding some debug lines when it gets out of sync; an also to ask for another snapshot of the book instead of disconnect for 7 seconds (on next commits)

btw --lifetime argument should help to avoid the api rate limits, but has nothing to do with the book getting out of sync

:koala: thank you for your report''

[ctubio]

maaaybe meaaanwhile can you paste a first run with --debug-secret enabled?
(just until the first book out of sync msg is enough)

so i can see all communication between you and your endpoint and calc the checksum manually from it?
(to see if you really are missing data or is my checksum calc that is the worst ever)

[MiningInMySleep]

Edit: Yes, I am currently using the latest linux-x86_64 that included the checksum checking.
What parts do you need? There's a lot of stuff here and you recommended against sharing --debug-secret output.

[ctubio]

okok what i am most interested to know now is what currency pairs are you using :D (so i can try with the same)

are you still on XRPBTC?

:koala: thank you0'!

[MiningInMySleep]

E-mailed you the logs. Somehow got the error as I was logging. You can see it happen near line 6169. Only on btc/usd at the moment.

[ctubio]

:koala: i dont see nothing else wrong on bitfinex; lemme close this issue

[eniware-org]

Running with the latest version and this is happening again!

I notice that the QE is connected BEFORE the wallet and orders WS is OK. Manual disconnect/connect fixes it.