Gekko: Missing candle when importing poloniex btc usdt

[totomaze]

I can't get the same backtesting result from 2 different machine (different public IP each) and using the same config.js
I triggered the import on both machine almost at the same time using Poloniex (no error in log during import)

The problem could come from the import, because I then delete both databases he redone the test, and I saw different result again.
I also did another full import on the machine giving me wrong result without dropping the database, but nothing changed.
After I used the same database files in both machine abd the results are the same.

I also notice wilt BTC_USDT it will always miss a candle between 2016-06-01 and 2016-06-02. Any clue or more debug log ?

Machine 1

_2016-10-21 04:18:47 (WARN):     The plugin Pushbullet does not support the mode backtest. It has been disabled.
2016-10-21 04:18:47 (INFO):     Scanning local history for backtestable dateranges.
2016-10-21 04:18:47 (INFO):     The database has 1 candles missing, Figuring out which ones...
2016-10-21 04:18:47 (INFO):     Gekko detected multiple dateranges in the locally stored history. Please pick the daterange you are interested in testing:
2016-10-21 04:18:47 (INFO):                      OPTION 1:
2016-10-21 04:18:47 (INFO):              from: 2016-06-01 00:50:00
2016-10-21 04:18:47 (INFO):              to: 2016-06-01 19:50:00
2016-10-21 04:18:47 (INFO):                      OPTION 2:
2016-10-21 04:18:47 (INFO):              from: 2016-06-01 20:50:00
2016-10-21 04:18:47 (INFO):              to: 2016-10-21 02:50:00
prompt: option:  2

2016-10-21 04:19:30 (INFO):             WARNING: BACKTESTING FEATURE NEEDS PROPER TESTING
2016-10-21 04:19:30 (INFO):             WARNING: ACT ON THESE NUMBERS AT YOUR OWN RISK!

2016-10-21 04:19:30 (INFO):     2016-06-03 21:59:00: Profit simulator got advice to short       666.394 USDT <= 1.000 BTC at 570.1
2016-10-21 04:19:31 (INFO):     2016-06-14 17:59:00: Profit simulator got advice to long        666.394 USDT => 1.082 BTC at 612
2016-10-21 04:19:31 (INFO):     2016-06-16 05:59:00: Profit simulator got advice to short       784.325 USDT <= 1.082 BTC at 729.7603
2016-10-21 04:19:31 (INFO):     2016-06-21 01:59:00: Profit simulator got advice to long        784.325 USDT => 1.125 BTC at 692.51258201
2016-10-21 04:19:31 (INFO):     2016-06-30 15:59:00: Profit simulator got advice to short       748.684 USDT <= 1.125 BTC at 669.72199999
2016-10-21 04:19:32 (INFO):     2016-07-07 05:59:00: Profit simulator got advice to long        748.684 USDT => 1.164 BTC at 639
2016-10-21 04:19:32 (INFO):     2016-07-12 17:59:00: Profit simulator got advice to short       778.291 USDT <= 1.164 BTC at 672.99
2016-10-21 04:19:33 (INFO):     2016-07-31 05:59:00: Profit simulator got advice to long        778.291 USDT => 1.222 BTC at 632.64000519
2016-10-21 04:19:34 (INFO):     2016-08-07 15:59:00: Profit simulator got advice to short       741.181 USDT <= 1.222 BTC at 610.3844426
2016-10-21 04:19:34 (INFO):     2016-08-14 13:59:00: Profit simulator got advice to long        741.181 USDT => 1.292 BTC at 570.00000085
2016-10-21 04:19:35 (INFO):     2016-09-03 19:59:00: Profit simulator got advice to short       775.215 USDT <= 1.292 BTC at 604
2016-10-21 04:19:36 (INFO):     2016-09-11 19:59:00: Profit simulator got advice to long        775.215 USDT => 1.271 BTC at 605.89999988
2016-10-21 04:19:37 (INFO):     2016-10-01 01:59:00: Profit simulator got advice to short       775.018 USDT <= 1.271 BTC at 613.69763595
2016-10-21 04:19:38 (INFO):     2016-10-19 15:59:00: Profit simulator got advice to long        775.018 USDT => 1.228 BTC at 626.816209
2016-10-21 04:19:38 (INFO):
2016-10-21 04:19:38 (INFO):     (PROFIT REPORT) start time:                      2016-06-01 20:50:00
2016-10-21 04:19:38 (INFO):     (PROFIT REPORT) end time:                        2016-10-21 02:51:00
2016-10-21 04:19:38 (INFO):     (PROFIT REPORT) timespan:                        5 months
2016-10-21 04:19:38 (INFO):
2016-10-21 04:19:38 (INFO):     (PROFIT REPORT) start price:                     532.399
2016-10-21 04:19:38 (INFO):     (PROFIT REPORT) end price:                       633.25000002
2016-10-21 04:19:38 (INFO):     (PROFIT REPORT) Buy and Hold profit:             18.942750000000004%
2016-10-21 04:19:38 (INFO):
2016-10-21 04:19:38 (INFO):     (PROFIT REPORT) amount of trades:                14
2016-10-21 04:19:38 (INFO):     (PROFIT REPORT) original simulated balance:      632.39900 USDT
2016-10-21 04:19:38 (INFO):     (PROFIT REPORT) current simulated balance:       777.88329 USDT
2016-10-21 04:19:38 (INFO):     (PROFIT REPORT) simulated profit:                145.48429 USDT (23.00514%)
2016-10-21 04:19:38 (INFO):     (PROFIT REPORT) simulated yearly profit:         375.93987 USDT (59.44663%)_

Machine2

_2016-10-21 04:17:49 (INFO):  Scanning local history for backtestable dateranges.
2016-10-21 04:17:49 (INFO): The database has 1 candles missing, Figuring out which ones...
2016-10-21 04:17:49 (INFO): Gekko detected multiple dateranges in the locally stored history. Please pick the daterange you are interested in testing:
2016-10-21 04:17:49 (INFO):      OPTION 1:
2016-10-21 04:17:49 (INFO):    from: 2016-05-31 23:09:00
2016-10-21 04:17:49 (INFO):    to: 2016-06-01 20:09:00
2016-10-21 04:17:49 (INFO):      OPTION 2:
2016-10-21 04:17:49 (INFO):    from: 2016-06-01 21:09:00
2016-10-21 04:17:49 (INFO):    to: 2016-10-21 04:09:00
prompt: option:  2

2016-10-21 04:17:56 (INFO):   WARNING: BACKTESTING FEATURE NEEDS PROPER TESTING
2016-10-21 04:17:56 (INFO):   WARNING: ACT ON THESE NUMBERS AT YOUR OWN RISK!

2016-10-21 04:17:56 (INFO): 2016-06-03 22:18:00: Profit simulator got advice to short   666.884 USDT <= 1.000 BTC at 570.59328664
2016-10-21 04:17:58 (INFO): 2016-06-21 00:18:00: Profit simulator got advice to long  666.884 USDT => 0.922 BTC at 718.39485096
2016-10-21 04:17:58 (INFO): 2016-07-01 04:18:00: Profit simulator got advice to short   625.812 USDT <= 0.922 BTC at 683.00000004
2016-10-21 04:17:59 (INFO): 2016-07-07 06:18:00: Profit simulator got advice to long  625.812 USDT => 0.980 BTC at 634.19447743
2016-10-21 04:17:59 (INFO): 2016-07-12 16:18:00: Profit simulator got advice to short   656.473 USDT <= 0.980 BTC at 674
2016-10-21 04:18:00 (INFO): 2016-07-22 14:18:00: Profit simulator got advice to long  656.473 USDT => 1.000 BTC at 652.13
2016-10-21 04:18:03 (INFO): 2016-09-03 20:18:00: Profit simulator got advice to short   604.118 USDT <= 1.000 BTC at 607.9999999
2016-10-21 04:18:04 (INFO): 2016-09-11 20:18:00: Profit simulator got advice to long  604.118 USDT => 0.989 BTC at 607
2016-10-21 04:18:05 (INFO): 2016-10-01 00:18:00: Profit simulator got advice to short   601.440 USDT <= 0.989 BTC at 612.24210628
2016-10-21 04:18:07 (INFO): 2016-10-19 14:18:00: Profit simulator got advice to long  601.440 USDT => 0.954 BTC at 626.6099724
2016-10-21 04:18:07 (INFO): 
2016-10-21 04:18:07 (INFO): (PROFIT REPORT) start time:      2016-06-01 21:09:00
2016-10-21 04:18:07 (INFO): (PROFIT REPORT) end time:      2016-10-21 04:10:00
2016-10-21 04:18:07 (INFO): (PROFIT REPORT) timespan:      5 months
2016-10-21 04:18:07 (INFO): 
2016-10-21 04:18:07 (INFO): (PROFIT REPORT) start price:       534.3
2016-10-21 04:18:07 (INFO): (PROFIT REPORT) end price:       635.31824038
2016-10-21 04:18:07 (INFO): (PROFIT REPORT) Buy and Hold profit:     18.90665%
2016-10-21 04:18:07 (INFO): 
2016-10-21 04:18:07 (INFO): (PROFIT REPORT) amount of trades:    10
2016-10-21 04:18:07 (INFO): (PROFIT REPORT) original simulated balance:  634.30000 USDT
2016-10-21 04:18:07 (INFO): (PROFIT REPORT) current simulated balance:   605.83487 USDT
2016-10-21 04:18:07 (INFO): (PROFIT REPORT) simulated profit:    -28.46513 USDT (-4.48764%)
2016-10-21 04:18:07 (INFO): (PROFIT REPORT) simulated yearly profit:   -73.53385 USDT (-11.59291%)_

[askmike]

Judging by the way things are formatted it appears you are running different versions of Gekko.

There have been a few small bumps in how local data has been passed to the trading method. In order to fully figure out what is going on, could you let me know the output of the following (on both machines)?

node gekko --version
git log | head -n 6

Regards, Mike

[totomaze]

Machine 1:
Gekko version: v0.3.4
Nodejs version: v6.8.1
commit 27a9e2b3d24c18d5dcd7474265782b7de59d5af1
Author: Mike van Rossum [email protected]
Date: Tue Oct 18 21:38:31 2016 +0100

fix #483

Machine 2
Gekko version: v0.3.4
Nodejs version: v6.4.0

commit 27a9e2b3d24c18d5dcd7474265782b7de59d5af1
Author: Mike van Rossum [email protected]
Date: Tue Oct 18 21:38:31 2016 +0100

fix #483

I got different result in both. Maybe the problem is not coming from gekko but poloniex or network problems.
I added a waiting of 500ms time between each api call, thinking it could be a limit in the api call but still not the same results.

I will try on a third machine, because I don´t know which result is the correct one.

Machine 3
Gekko version: v0.3.4
Nodejs version: v6.4.0

commit 27a9e2b3d24c18d5dcd7474265782b7de59d5af1
Author: Mike van Rossum [email protected]
Date: Tue Oct 18 21:38:31 2016 +0100

fix #483

[niklasbuschmann]

The problem isn't that there are 2 different versions, it is the time when the bot trades. First one trades at 00:18 + n * candle_size, second one at 01:59 + n * candle_size.

In my experience this two hours can make a huge difference in such volatile markets.

Specify a concrete starting point in the backtesting section of the config and post your results again.

config.backtest = {
  adapter: 'sqlite',
  daterange: {from: "2016-06-01 22:00:00"}, // or whatever both machines have available
  batchSize: 50
}

[askmike]

@niklasbuschmann is right: you are not backtesting the same data, which might calculate completely different indicators. @totomaze could you verify that testing the same daterange yields the same results?

So that means that importing from Poloniex gives different results every time you do it. I will do some testing to see what Poloniex does different (specifically around 2016-06-01 and 2016-06-02).

[totomaze]

@niklasbuschmann: with the same date range I have the same results! The different results make sense with your explaination.

I thought it was calculated differently since I use another trading bot that use these parameters and can triggered action quicker than the candle size (zenbot):
rs.rsi_periods = 26 // RSI smoothing factor
rs.rsi_period = '1h' // RSI tick size
rs.rsi_up = 70 // upper RSI threshold
rs.rsi_down = 30 // lower RSI threshold
rs.check_period = '1m' // speed to trigger actions at

Instead of 120 for the candle size I put 1 and change the persistence value to achieve what I wanted.

I think we can close the problem of the backtesting returning different results but the problem about missing candle is stil there (2016-06-01)

Thanks for your help.

[niklasbuschmann]

I think you can add specific candles from poloniex, this should fix the problem if there was just a package loss or some other random bug.

config.importer = {
  daterange: {
    // NOTE: these dates are in UTC
    from: "2016-06-01 18:00:00",
    to: "2016-06-01 23:00:00"
  }
}

Instead of 120 for the candle size I put 1 and change the persistence value to achieve what I wanted.

Did this work out well? Have currently no instance running, but I think I got pretty bad results when backtesting this approach. Don't know how zenbot handles this, last time I checked they lacked poloniex support.

[totomaze]

I tried to import just this part but there is still a missing candle

2016-10-25 02:09:32 (DEBUG):    Requesting data from 2016-06-01 18:00:00, to 2016-06-01 20:00:00
2016-10-25 02:09:32 (DEBUG):    Processing 72 new trades. From 2016-06-01 18:02:28 UTC to 2016-06-01 19:52:42 UTC. (2 hours)
2016-10-25 02:09:33 (DEBUG):    Requesting data from 2016-06-01 19:50:00, to 2016-06-01 21:50:00
2016-10-25 02:09:33 (DEBUG):    Processing 162 new trades. From 2016-06-01 20:02:11 UTC to 2016-06-01 21:41:28 UTC. (2 hours)
2016-10-25 02:09:34 (DEBUG):    Requesting data from 2016-06-01 21:40:00, to 2016-06-01 23:40:00
2016-10-25 02:09:34 (DEBUG):    Processing 30 new trades. From 2016-06-01 21:55:24 UTC to 2016-06-01 22:59:58 UTC. (an hour)
2016-10-25 02:09:34 (INFO):     Done importing!

2016-10-25 02:10:08 (INFO):                      OPTION 30:
2016-10-25 02:10:08 (INFO):              from: 2016-05-17 18:50:00
2016-10-25 02:10:08 (INFO):              to: 2016-06-01 19:50:00
2016-10-25 02:10:08 (INFO):                      OPTION 31:
2016-10-25 02:10:08 (INFO):              from: 2016-06-01 20:50:00
2016-10-25 02:10:08 (INFO):              to: 2016-10-21 02:50:00

Did this work out well? Have currently no instance running, but I think I got pretty bad results when backtesting this approach. Don't know how zenbot handles this, last time I checked they lacked poloniex support.

Looks it works well with the last 3 months but I would like to try with a bigger backtest because it could be really bad for another period. Yes they still doens t fully support poloniex but someone did a pull request. I use zenbot with gdax.

[askmike]

Very strange.. Could you send me the database files so I can have a look at
the differences? They are located in gekko/history/poloniex.. you can email
them to [email protected]

On Oct 23, 2016 05:34, "totomaze" [email protected] wrote:

_Machine 1:_
Gekko version: v0.3.4
Nodejs version: v6.8.1
commit 27a9e2b
https://github.com/askmike/gekko/commit/27a9e2b3d24c18d5dcd7474265782b7de59d5af1
Author: Mike van Rossum [email protected]
Date: Tue Oct 18 21:38:31 2016 +0100

fix #483

_Machine 2_
Gekko version: v0.3.4
Nodejs version: v6.4.0

commit 27a9e2b
https://github.com/askmike/gekko/commit/27a9e2b3d24c18d5dcd7474265782b7de59d5af1
Author: Mike van Rossum [email protected]
Date: Tue Oct 18 21:38:31 2016 +0100

fix #483

I got different result in both. Maybe the problem is not coming from gekko
but poloniex or network problems.
I added a waiting of 500ms time between each api call, thinking it could be
a limit in the api call but still not the same results.

I will try on a third machine, because I don´t know which result is the
correct one.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/askmike/gekko/issues/486#issuecomment-255569223, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA7MD5aIMPbMR-Xlj43LFO9Wy74Ye4Fgks5q2uO7gaJpZM4Kc2WE
.

[totomaze]

I had a look at the database yesterday, And I can see the missing candle.
select * from candles_USDT_BTC where start >1464810600 and start <= 1464814200
missing candle : 1464814140

The Db (30MB) https://drive.google.com/open?id=0B3PTbH__lCVHUDZZVTF6QjlQdHc

Here is what poloniex return around the missing candle :
https://poloniex.com/public?command=returnTradeHistory&currencyPair=USDT_BTC&start=1464814020&end=1464814260

_[{globalTradeID: 33632688, tradeID: 324699, date: "2016-06-01 20:50:05", type: "buy",…},…]
0
:
{globalTradeID: 33632688, tradeID: 324699, date: "2016-06-01 20:50:05", type: "buy",…}
1
:
{globalTradeID: 33632675, tradeID: 324698, date: "2016-06-01 20:50:02", type: "buy",…}
2
:
{globalTradeID: 33632564, tradeID: 324697, date: "2016-06-01 20:49:28", type: "sell",…}
3
:
{globalTradeID: 33632522, tradeID: 324696, date: "2016-06-01 20:48:54", type: "sell",…}
4
:
{globalTradeID: 33632395, tradeID: 324695, date: "2016-06-01 20:48:12", type: "buy",…}
5
:
{globalTradeID: 33632200, tradeID: 324694, date: "2016-06-01 20:47:56", type: "buy",…}
6
:
{globalTradeID: 33632154, tradeID: 324693, date: "2016-06-01 20:47:43", type: "buy",…}
7
:
{globalTradeID: 33632140, tradeID: 324692, date: "2016-06-01 20:47:26", type: "sell",…}
8
:
{globalTradeID: 33632139, tradeID: 324691, date: "2016-06-01 20:47:26", type: "sell",…}
9
:
{globalTradeID: 33632126, tradeID: 324690, date: "2016-06-01 20:47:20", type: "buy",…}
10
:
{globalTradeID: 33632125, tradeID: 324689, date: "2016-06-01 20:47:20", type: "buy",…}
11
:
{globalTradeID: 33632124, tradeID: 324688, date: "2016-06-01 20:47:20", type: "buy",…}_

I think the importer is not saving the entry from 20:49:28 with an amount=0 and volume=0

[askmike]

I think the importer is not saving the entry from 20:49:28 with an amount=0 and volume=0

@totomaze that's it! Thanks for looking into it for me.

To be accurate: Gekko aggregates trades into candles (OHCLVWP), and was not able to deal with trades that have a volume of 0..

I am working on a web UI for Gekko and had to do some refactoring. In the branch web-wrapper you'll find a new daterange scanner where you can specify the amount of candles that are allowed to be missing (if you miss a minute it most likely won't impact your result. Whereas a month of missing data should be guarded for).

[carlosfvp]

I am sorry to reopen this, but poloniex it's actually not delivering all the transactions that you ask for in a time range?
Or there is there some kind of issue with the candle manager/backtest feature?

I am requesting data from 1 month but I get too much missing candles when I run the backtest. But everytime I play with the next parameters in /importers/exchanges/poloniex.js, the missing candles begin reduce:

var batchSize = 60 * 2; // 2 hour
var overlapSize = 5; // 10 minutes

As you can see, this is really annoying. So back to the first paragraph, it's polo or something is missing?

[askmike]

@carlosfvp could you let me know the market & gap dates, I'll try to debug.

[carlosfvp]

market Poloniex
currency BTC
asset ETH
from 2017-03-01
to 2017-03-09

This issue happens only when the importer runs in console mode.
Around the 5 first days are collected without gaps. Then the missing candles are on the next days.

Yesterday I tested on UI mode and it worked perfect.

[askmike]

That is very strange.. When you run in console mode, are you running the
latest version of gekko?

On Thu, Mar 30, 2017 at 6:22 PM, Carlos Villacorta <[email protected]

wrote:

market Poloniex
currency BTC
asset ETH
from: 2017-03-01
from: 2017-03-09

This issue happens only when the importer runs in console mode.
Around the 5 first days are collected without gaps. Then the missing
candles are on the next days.

Yesterday I tested on UI mode and it worked perfect.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/askmike/gekko/issues/486#issuecomment-290481229, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA7MD7UrGPrSGhBCj6TaWNfmn2IEuE-Dks5rq-TfgaJpZM4Kc2WE
.

--
PGP key at keybase.io/mikevanrossum
https://keybase.io/mikevanrossum/key.asc

[greenbigfrog]

@carlosfvp is this still a issue?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If you feel this is very a important issue please reach out the maintainer of this project directly via e-mail: gekko at mvr dot me.