Gekko: GDAX not importing data?

[outpoints]

• I'm submitting a ...
  [x] bug report

• Action taken (what you did)
  Nothing happens when I try to import gdax data?

• Expected result (what you hoped would happen)
  Import like any other exchange?

• Actual result (unexpected outcome)
  
  
  
  

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, etc)
  No idea

[holeyness]

Here is the cause, GDAX has a rate limit of 3 - 6 requests a second. The code in gdax.js, in the method getTrades ignores that and blasts threw that limit. When the api returns 429 (rate limit exceeded), the code retries the whole thing, thus it will never complete.

[outpoints]

@holeyness do you have a fix?

[holeyness]

@mew351 im working on one, ill see if i can make a pull req when it works

[outpoints]

Let me know if I can help with anything!

On Dec 13, 2017 5:49 PM, "Ian Luo" [e-mail] wrote:

@mew351 https://github.com/mew351 im working on one, ill see if i can
make a pull req when it works

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/askmike/gekko/issues/1473#issuecomment-351583198, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ALzdJTl_xZB8UOCz92zIHO4Ab1bgE8ZBks5tAH6cgaJpZM4RBW62
.

[tdsticks]

I simply added a sleep at the start of the process method around line 228. This is just a workaround, I'm sure there's a better solution.

var process = function(err, response, data) {
sleep.sleep(2)

You'll need to install the sleep npm
npm install sleep --save

[outpoints]

So something like this?

var process = function(err, response, data) { sleep.sleep(2) var process = function(err, response, data) { if (data && data.message) err = new Error(data.message);

[tdsticks]

This is what I have:

[outpoints]

@tdsticks thanks!
Will report back once I can install sleep without issues -_-

[holeyness]

Here's my Trader.prototype.getTrades:
`
var err_cache = {};
var response_cache = {};
var data_cache = {};

Trader.prototype.getTrades = function(since, callback, descending) {
var args = _.toArray(arguments);
var lastScan = 0;
var delay = 334;
var current_page = 100;
var gdax_client = this.gdax_public;

function cacheOrFetch(page_num, callback) {
  if (page_num in response_cache) {
    console.log("cached, page: ", page_num);
    callback(err_cache[page_num], response_cache[page_num], data_cache[page_num]);
  } else {
    console.log("requesting", page_num);
    // retrieve from api
    setTimeout(function(){
        gdax_client.getProductTrades({'after': page_num, limit: batchSize}, callback);
    }, delay);
  }
}

var process = function(err, response, data) {

    if (response.statusCode === 429){
      // rate limit blown, retrying current one
      delay *= 2;
      setTimeout(function() {
          gdax_client.getProductTrades({'after': current_page}, process);
      }, delay);
      return;

    } else if (response.statusCode !== 200 || !data || data.length < 1){
      console.log(response);
      console.log("retrying");
      return this.retry(this.getTrades, args);
    }

    // We are good
    if (delay > 434) {
      delay -= 100;
    }

    // Caching
    console.log("stored in cache: ", current_page);
    err_cache[current_page] = err;
    response_cache[current_page] = response;
    data_cache[current_page] = data;


    var result = _.map(data, function(trade) {
        return {
            tid: trade.trade_id,
            amount: parseFloat(trade.size),
            date: moment.utc(trade.time).format('X'),
            price: parseFloat(trade.price)
        };
    });

    if (this.scanback) {
        var last = _.last(data);
        var first = _.first(data);

        // Try to find trade id matching the since date
        if (!this.scanbackTid) {
            // either scan for new ones or we found it.
            if (moment.utc(last.time) < moment.utc(since)) {
                this.scanbackTid = last.trade_id;
            } else {
                log.debug('Scanning backwards...' + last.time);
                current_page = last.trade_id - (batchSize * lastScan);
                cacheOrFetch(current_page, process);
                lastScan++;
                if (lastScan > 100) {
                    lastScan = 10;
                }
            }
        }

        if (this.scanbackTid) {
        // if scanbackTid is set we need to move forward again
            log.debug('Backwards: ' + last.time + ' (' + last.trade_id + ') to ' + first.time + ' (' + first.trade_id + ')');

            if (this.import) {
                this.scanbackTid = first.trade_id;
                callback(null, result.reverse());
            } else {
                this.scanbackResults = this.scanbackResults.concat(result.reverse());

                if (this.scanbackTid !== first.trade_id) {
                    this.scanbackTid = first.trade_id;
                    current_page = this.scanbackTid + batchSize + 1;
                    cacheOrFetch(current_page, process);
                } else {
                    this.scanback = false;
                    this.scanbackTid = 0;
                    if (!this.import) {
                        log.debug('Scan finished: data found:' + this.scanbackResults.length);
                        callback(null, this.scanbackResults);
                    }
                    this.scanbackResults = [];
                }
            }
        }
    } else {
        callback(null, result.reverse());
    }
}.bind(this);

if (since || this.scanback) {
    this.scanback = true;
    if (this.scanbackTid) {
        current_page = this.scanbackTid + batchSize + 1;
        cacheOrFetch(current_page, process);
    } else {
        log.debug('Scanning back in the history needed...');
        log.debug(moment.utc(since).format());
        setTimeout(function(){
            gdax_client.getProductTrades({limit: batchSize}, process);
        }, delay);
    }
} else {
    setTimeout(function(){
        gdax_client.getProductTrades({limit: batchSize}, process);
    }, delay);
}

}`

Theres some oppurtunistic caching in there, plus a backoff algorithm for not busting the api.

[tdsticks]

Cool @holeyness, I'll check that out, thanks for writing that!

I wonder if we should utilize the websocket feed GDAX provides for live data. Not sure if that would apply here.

[outpoints]

This is an incredibly retarded question but @holeyness where do I put this file? Do I overwrite the gdax in exchanges or the gdax in the importers?

[holeyness]

exchanges/gdax.js
Replace the method getTrades, and then add the 3 lines above it.

[outpoints]

Is it working for you @tdsticks ? It's still not working for me, it starts but doesn't actually work

EDIT

I fixed it, it just wasn't saving :P

[outpoints]

So uh. Is there any way to do this faster? :/

[holeyness]

Thats the limitation with gdax itself. the api only returns 100 trades per call, and you can only make 3 calls a second.

[holeyness]

I think it would be a good idea to share datasets

[outpoints]

I wonder if we setup a bunch of different APIs and had them doing different calls per second if it'd do it faster? @holeyness

[outpoints]

How would we go about getting something like this to work?
https://gekkowarez.com/download/data-downloader-for-gekko/

[holeyness]

gdax at least is rate limited per ip address, so if you set up a buncha servers on different ips you can get around the rate limit. it takes about 8 hours to download around 3 months of trades from gdax

[greenbigfrog]

@holeyness there's a reason, why they have the ratelimits. Getting past the rate limits by spawning up multiple servers will make the experience worse for everyone

[holeyness]

@greenbigfrog yup you're right. I have been waiting patiently for this data :/

[pereztechseattle]

Thank you for this. I am running GDAX USD/BTC import of just two days; will report how it goes.

[zipzapp]

I think this project encourages a lot of unnecessary slamming of the exchange APIs to fetch essentially the same data, couldn't we zip up in like 1 month increments per exchange, and either commit them here... or perhaps there's an s3 server that already serves this kind of data...

[holeyness]

@billyp1980 mine finished importing an entire year of data in about 3 days

[mmeisel]

Hi @holeyness, super helpful, thanks! But, unless I'm missing something, I think you'll need to add some code to invalidate your caches. Would you be willing to submit your code as a pull request so others can comment on the code directly?

[mmeisel]

Actually, I think I found a simpler/safer way to do this, although I'm not doing any caching. See PR #1534.

[holeyness]

@mmeisel, actually in the end I took out the caching thing. Because, I increased the skip rate on rewind, so it jumps up to 500 pages back when searching backwards. (this makes loading an entire year of data faster)
However, which means the cache hit rate will be < 1/500, so not really a huge performance gain there. And I was thinking that it may tax the memory too much.

[davij]

holyness after editing gdax.js according to your post i get "Child process has died". Syntax error??

[werkkrew]

Another potential enhancement here would be to use the authenticated endpoint if its configured in gekko. It has higher rate limits:

PUBLIC ENDPOINTS
We throttle public endpoints by IP: 3 requests per second, up to 6 requests per second in bursts.

PRIVATE ENDPOINTS
We throttle private endpoints by user ID: 5 requests per second, up to 10 requests per second in bursts.

[holeyness]

@davij, here's my gdax.js file. I've used it to successfully import a year worth of data (took 3 days).

https://pastebin.com/Kdvya1hx

[jmbartho]

Is this an issue with other exchanges as well?

I know that Kraken also has a rate limit. They recommend keeping requests to one / second to avoid a 15 minute suspension.

[prometheusminer]

I had the same error message "Child process has died".
@holeyness Could you share your js version or modules?

The following is the full log of this error.

[holeyness]

@prometheusminer, its not an import issue. There's a bug in the code, see my changes above.

[joao-roberto]

i have this issue with Binance, we can't import from tradingview ?

[askmike]

we can't import from tradingview ?

nope.

I want to tackle the importer asap, see #2448. Anyone mind helping out? My TODO is already pilling over atm.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If you feel this is very a important issue please reach out the maintainer of this project directly via e-mail: gekko at mvr dot me.