Gekko: Historical data from bitcoincharts.com?

[Suko]

Hi, did everyone tried to import historical data from bitcoincharts.com, they have it from biggest markets and it is updated twice a day.

Data can be found here:
http://api.bitcoincharts.com/v1/csv/

[askmike]

we can add an importer for bitcoincharts (though some time ago the data was not considered to be 100% accurate, I am not sure what the current status is).

[thegamecat]

I've done just that for btce. Took a while but worth it. Take the data, put it into 1m candles then import it into sqlite.

[askmike]

I'd love to make that part of Gekko at some point, should be pretty easy to create a new market file that will import a csv export from bitcoincharts and pipes into a gekkostream (even simpler/faster would be a script that inserts it straight into sqlite, but you'd lose flexibility for all of gekkos supported adapters (postgre, mysql, etc).

[thegamecat]

Yes, really you just need the candle maker from raw data then spits into the 3 supported dbs.

[tuxitor]

I support the idea of a file importer. At least a couple of sites are offering historical data in JSON and/or CSV format. That would leave the responsibility of fetching data to the user and possibly not pose any licence issues for this project. Another example is cryptocompare.com that has history data. Here is an example:
https://min-api.cryptocompare.com/data/histominute?fsym=BTC&tsym=USD&limit=60&aggregate=1&e=Bitstamp

A file importer could specify a format or be configurable. The obvious choices would be "JSON import" and "CSV import". Mapping a JSON/CSV format to another is usually a trivial task.

A file importer may possibly be of use for filling holes in the history database.

The dropdown menu should only show supported exchanges.

[askmike]

@eigilb that looks great.

A file importer could specify a format or be configurable. The obvious choices would be "JSON import" and "CSV import".

I'd say we need importers based on providers. We already have a "native" one that doesn't use any providers but the exchanges itself (requires no trust besides the exchange, this is a service given only by some exchanges). We can add additional ones for bitcoincharts and cryptocompare.

This is also handy because the logic needed for each provider is different: bitcoincharts gives trades, which Gekko would need to aggregate into candles, while cryptocompare already gives candles (just pipe them into a database).

Mapping a JSON/CSV format to another is usually a trivial task.

It is, but dealing with inconstancies in output is not: Besides OHCL, gekko also expects a few other properties, which might not be present in the data. Gekko does not work on a dataset of candles that are not exactly 1 minute, etc.

See below.

@thegamecat

Yes, really you just need the candle maker from raw data then spits into the 3 supported dbs.

That would be ~20 lines of code with Gekko: You import the adapter you want, you import the candleCreator (transforms trades into candles), fetch trades & pipe candles into the adapter.

---

So why don't we add some importers for data providers (see above) and add a simple script (gekko-import or something you can use if you have CSV data, with some documentation on how to format your data etc)?

[tuxitor]

Cryptocompare already has candles that are exactly one minute, and it has OHCL plus it has volume (volumefrom), but it is missing vwp and trade count. Am I understanding correctly that vwp can be calculated? I don't know how important the trade count is.
They only keep minute candles for one week, and max items per request is 2000. Default is 1440. That means the need for 7 chained requests to fetch a week of data.

I have not any experience with node programming, but as a former programmer, now retired since many years, I am impressed with the programming work done on this project.

[askmike]

I am open for pull requests, won't have time for this myself anytime soon!