Zenbot: Consider replacing mongodb by sqlite

[DeviaVir]

Something standalone like this really does not need a mongodb.

[tuxitor]

Why not a plugin model with an API layer like exchanges?

[arpheno]

As long as it's contained in a docker-compose, why bother? Also schemas, also using relational databases for object storage...

[egorbenko]

We could potentially abstract the db layer and create a plugin for any db but I don't really see a downside of using mongodb. Might be a lot of work for no real benefit?

[carlos8f]

I like mongo for this app, it allows fast inserts for backfill, indexing is flexible, and adding fields is painless as adding a variable to a js object. Eventually I'd like an http server component, which would likely work as a separate command from the trader, and read/write the shared DB.

That said, I have been considering a "lite" mode where the trader can function without a DB at all, buffering the trades in memory only. That might be nice for people running on Raspberry Pi and whatnot.

[ColtonProvias]

I think there should be an abstracted DB layer. While mongo is good for basic installations, more serious application will use time-series databases like InfluxDB, Graphite, etc.

I think all that is really needed are an database initializer, a set(key, timestamp, valueHash) and a get(key, timestamps=[]) functions.

For experimenting with this, I'd recommend testing out InfluxDB with Grafana. I'm running them here for monitoring my accounts and it works extremely well for giving me a dashboard. To write data to influxdb:

const influxdb = new influx.InfluxDB({host: '38.263.112.24', port: 8086, database: 'topSecret'})

// Writing a measurement. There can also be tags, the timestamp, whatever fields, etc.
await influxdb.writeMeasurement('madeUpNumbers', [{fields: {
  btcPrice: 2835.183920,
  realBtcPrice: someVariable,
  usdBalance: 1.35,
  targetUsdBalance: 100000000
}}])

// A SQL-like query language is used to recall data
let rows = await influx.query('select * from madeUpNumbers order by time desc')
rows.forEach(row => console.log(row))

[bdfoster]

+1, I'd like to see this as well.

@carlos8f I'm currently writing a little library for this, and it will soon be on npm.
It will come with an adapter for memory and ArangoDB, but I can write one for MongoDB and just about any other database out there. I'll check in if you're interested when it's ready and I'd be happy to work up some PRs to get it working here. Follows a lot of the same principles that js-data uses but just a little more flexible. Two of my other projects will be using it in production.

[timstoop]

Just to chime in here, sqlite is not really usable if you're doing heavy backtesting and sim. Having mongodb is nice so you can have a single process that fills it with the latest data, create a cluster over multiple servers and have lots of clients running sims based on the data contained therein. So if you want this change, I hope you go for it being optional instead of getting rid of mongodb.

[DeviaVir]

Agreed, I'm thinking of something customizable.