Zenbot: Backfill poloniex.* no responding

[phongtt3]

Hello,
I can't backfill any pair from poloniex from zenbot.

When i try to open poloniex's public api from browser, i meet google's image captcha, so i think zenbot can't crawl data because this captcha.

Is there any work-around way to backfill data from Poloniex? if yes, can you pls tell me.

---

edit by @DeviaVir:

@pushkarnagle + @ivanmladek + @dmzpkts experience the same problem.

For more people experiencing this problem, please use the emoji reacts.

[krull]

@phongtt3 Confirmed that this bug is replicatable:

docker run --rm --link zenbot_mongodb_1:mongodb -it zenbot_server ./zenbot.sh backfill poloniex.LTC-BTC --days 90

skipping 184 hrs of previously collected data

poloniex.LTC-BTC saved 200 trades 83 days left
.
error: getTrades() returned duplicate results
{ product_id: 'LTC-BTC', to: 1501709244 }
{ product_id: 'LTC-BTC', to: 1501709244 }

Any thing can be done to make this succeed?

[ivanmladek]

Replicable.

sudo docker run --rm --link zenbot_mongodb_1:mongodb -it zenbot_server ./zenbot.sh backfill poloniex.GAME-BTC --days 100

poloniex.GAME-BTC saved 200 trades 100 days left
............
poloniex.GAME-BTC saved 14636 trades 99 days left
...........
poloniex.GAME-BTC saved 3036 trades 98 days left
...........
poloniex.GAME-BTC saved 6478 trades 97 days left
...........
poloniex.GAME-BTC saved 3938 trades 96 days left
...........
poloniex.GAME-BTC saved 2536 trades 95 days left
...........
poloniex.GAME-BTC saved 2596 trades 94 days left
...........
poloniex.GAME-BTC saved 3127 trades 93 days left
...........
error: getTrades() returned duplicate results
{ product_id: 'GAME-BTC', to: 1501709452 }
{ product_id: 'GAME-BTC', to: 1501709452 }

[466]

Same here:

./zenbot.sh backfill poloniex.LTC-XMR  --days 200

skipping 904 hrs of previously collected data

poloniex.LTC-XMR saved 200 trades 162 days left
.
error: getTrades() returned duplicate results
{ product_id: 'LTC-XMR', to: 1501940800 }
{ product_id: 'LTC-XMR', to: 1501940800 }

[BarnumD]

Same here:

docker run --link zenbot_mongodb_1:mongodb -it zenbot_server ./zenbot.sh backfill poloniex.BTC-USDT --days 365

skipping 1195 hrs of previously collected data

poloniex.BTC-USDT saved 200 trades 316 days left
.
error: getTrades() returned duplicate results
{ product_id: 'BTC-USDT', to: 1501709273 }
{ product_id: 'BTC-USDT', to: 1501709273 }

[DeviaVir]

I'd kindly like to ask the people that are experiencing this issue, to see if this PR solves it:
https://github.com/carlos8f/zenbot/pull/580
If it does or doesn't, please comment on the PR.

[ratajczak]

This is caused by exchange downtime:
On this example we can see that Poliniex was down for almost 3 hours:

{"globalTradeID":202975702,"tradeID":6692889,"date":"2017-08-02 21:27:53","type":"buy","rate":"2713.04276222","amount":"0.00071579","total":"1.94196887"},{"globalTradeID":202975565,"tradeID":6692888,"date":"2017-08-02 18:51:29","type":"sell","rate":"2706.00100000","amount":"0.08807832","total":"238.34002199"},

But getTrades attempts to fetch data in 2 hour windows:
https://github.com/carlos8f/zenbot/blob/master/extensions/exchanges/poloniex/exchange.js#L66
and stucks if the time difference between trades is longer than 2 hours.

So far I was able to avoid this problem by increasing this time window from 2 to 12 hours like shown in above PR.
This won't fix the problem if Poloniex goes down for longer that 12 hours so probably it would be good to rewrite the way how it fetches the data to handle these edge cases.
But in the mean time this PR solved the problem with all currency pairs I've tested.

[DeviaVir]

12hrs of downtime is insane, I think 2hrs is already cutting it. Merged your PR. I'll assume this ticket to be fixed, so will close. Feel free to comment or open a new ticket if the problem still occurs.