Ccxt: Streaming (WS, Websocket support): Orderbooks, Trades, Balances...

[cklester]

I would like to see a new market property called balances, that would function the same way as symbols. When you call fetchBalances(), then the balances property would get filled with an associative array of currencies with balances (free, used, total, available, etc.).

Balances would also be updated via websocket (when available) or by repeated calls to the API, depending on rate limits.

Ultimately, we could do something like:

kraken.fetchBalances();
let myAvailableBTC = kraken.balance.available['BTC'];
let myTotalBTC = kraken.balance.total['BTC'];

[kroitor]

You will have it, soon ) Thanks for posting!

[kroitor]

We have unified the balance API, as written here, in issue #36 . Now your feature request can be implemented easily ) We will add a dirty balance method that would save the balance on a market instance, first. And when we are ready to release the Websocket implementations, we will make it auto-updating, of course. Keep in touch! )

[cklester]

So, the "dirty balance" is an adjustment of the local ledger to reflect any orders I place? What about orders that fill? Is that also being tracked for the balances?

[kroitor]

Yes, when we add a rate-limited poller and release methods for websocket apis, the balance will reflect everything in real time (as real as it can possibly be). Currently, balances are not reloaded and not cached automatically, you have to do it manually, as we've just finished unifying them (that was a lot of work in itself). But, having them unified, we will soon add auto-adjustment as well. As for order fills, I don't think I get your question entirely, but a balance api usually tells an amount (left) in orders, a free available balance and a total, and there's usually a separate interface for execution details (transactions, trades). It will also be stream-based, basically, all data, including the balance, open orders, partially filled orders and closed orders, all of that must be mutually consistent.

[Helmi]

As this issue is still open and I had a hard time finding anything else about it (except issue #145 which is linked here in closed): Is there anything new on the whole WebSocket topic?

[kroitor]

@Helmi hi! This issue is still open, we are working on it and hope to deliver it soon. We are closing in to v2, as you may see from our versioning, where we will be introducing a major refactoring and many new features ) Our WS implementation will not only stream balances, but also fills, orders, and all that stuff, where available. Stay tuned for more updates! )

[Helmi]

Thanks for the quick headsup. That sounds awesome. Go for it 👍

[alexmipego]

+1
Any word on the version 2.0 release date?

[kroitor]

@alexmipego we are currently working on it intensively to deliver the first version of it asap, hopefully, this month.

[nitzanav]

Thank you for the effort! I am a fan!

@kroitor you wrote:

We are closing in to v2, as you may see from our versioning, where we will be introducing a major refactoring and many new features )

Is there a way to take glimpse on the v2 and WebSocket code and major refactoring? I couldn't find it nowhere.

Where are the versioning is managed? I couldn't find it.

[kroitor]

@nitzanav

Is there a way to take glimpse on the v2 and WebSocket code and major refactoring? I couldn't find it nowhere.

The refactoring is happening right now in the main branch (there's been a lot of changes since then), and the work on websockets is also on its way, but there's no single place that holds all of it right now, the work is done in forks mostly... Here's is an example of it: https://github.com/ccxt/ccxt/pull/751

Where are the versioning is managed? I couldn't find it.

Version single-sourcing is performed upon each build on Travis CI from package.json with the vss.js script. More about it: https://github.com/ccxt/ccxt/blob/master/CONTRIBUTING.md.

[nitzanav]

Thank you for the information and for this amazing project!

[rubobaquero]

@kroitor Is it going to be available in the three languages as usual?

[kroitor]

@rubobaquero yes, that's the whole point, if we could throw a implementation in just one single language into it, adding support for ws wouldn't take so long ) The main difficulty is to unify them all.

[GoChartingAdmin]

Any timeline updates on websocket implementation.

[kroitor]

@GoChartingAdmin yes, we are in the process of merging this PR: https://github.com/ccxt/ccxt/pull/751, so, expect to have first implementations of a generic ws wrapper in the coming days.

[fredfortier]

@kroitor I haven't completed my EtherDelta implementation yet. Does this PR allow for websocket implementation with all basic operations? Is it complete enough for me to give a stab at it?

[kroitor]

@fredfortier unfortunately, not yet... It is a first approach, and it is

• not unified (exchange-specific)
• not portable (language-specific)

We are still working on the unification of WS (this is the hardest part of it). If you want to dive deeper into the history of websockets in ccxt, here's an overview of it (contains several other implementations in JS, that are closer to a unified state): https://gist.github.com/kroitor/7dce1d23a10937ab8c07a5451f17ccf2

[kornrunner]

For PHP we can create a wrapper around some library, eg: https://github.com/amphp/websocket
Thing is, it would be available only for PHP 7+ (we can do it as a suggestion / optional thing)

[TimNZ]

I don't want to seem ungrateful or entitled because this is an awesome project, but you have been promising a WS enabled release for a while.

Perhaps you should create a Github project so it can be tracked more easily so others can contribute, and do a raw write up of simple documentation of the proposed API so we can comment on it for feedback when it is 'cheaper' to change?

I know very well pain of over-architecting things and not getting feedback early.

From what I've read what you are doing sounds quite ambitious and maybe a v3 when we are waiting for a v1.

[kroitor]

I don't want to seem ungrateful or entitled because this is an awesome project, but you have been promising a WS enabled release for a while.

We are doing the core work on the base code to deliver it asap.

Perhaps you should create a Github project so it can be tracked more easily so others can contribute, and do a raw write up of simple documentation of the proposed API so we can comment on it for feedback when it is 'cheaper' to change?

That is exactly as we think, we are going to release a subset of exchanges first, so that the community could pick up from there, but we need to have it passing our current tests at the very least, before we can release it. Hope for your understanding.

I know very well pain of over-architecting things and not getting feedback early.

We would love to get the early feedback, that is the reason for doing the preparation work now in the first place.

[TimNZ]

Awesome!

Since you are talking about passing tests and I see some PR for WS, that must mean you have settled on an implementation, and the API for it and usage?

Perhaps it would be a good idea to quickly knock out a draft section in the Manual marked WIP so we can view and provide feedback on the API and usage (if needed)?

Better for us to tell you now if we think we see problems.

[kroitor]

Perhaps it would be a good idea to quickly knock out a draft section in the Manual marked WIP so we can view and provide feedback on the API and usage (if needed)?

This is one of our topmost priorities now, along with a draft of the implementation itself. We are going to document it as soon as we have a draft layout (wip). Our team is very small, and we are doing our best to keep up with the support and with adding new functionality as well. WS is on the way, we just need a little more of your patience. Thx!

[TimNZ]

Cool, thanks.

Still suggest you look at using Github Projects.
Good for basic Roadmap management and visibility.

[mmehrle]

Hey guys - I'm pretty new here but have immersed myself into ccxt over the past two weeks. If I may offer a suggestion regarding the WIP and towards satisfying some of us eager beavers here interested in getting our hands on some [untested/undocumented] functionality. Many of us think in code and in my humble experience throwing together a few simple test examples in an ipython notebook would do wonders to satisfy both sides: we get to play with a few examples and the senior contributors implicitly develop an outline of the content that'll go into the WIP and later into the documentation. Plus an often ignored or unknown benefit of github is that it actually runs Jupyter notebooks and thus allows everyone to benefit from a mini quantopian style workbench if you will.

Anyway, just some food for thought. Keep up the good work.

[pursehouse]

Is there any organized list of the websocket work that is in progress so that others can attempt to help with it?

[kroitor]

Hi!

@mmehrle

Hey guys - I'm pretty new here but have immersed myself into ccxt over the past two weeks. If I may offer a suggestion regarding the WIP and towards satisfying some of us eager beavers here interested in getting our hands on some [untested/undocumented] functionality.

The main difficulty in this task is to provide some in-development/undocumented functionality. When we have it in an experimental state, that would mean we have it all (the other work from there is trivial). Having it in experimental state is our topmost priority now. Basically, you're asking, if it is possible to do 90% of the thinking job now, so that users could do the other 10%. Well, yes, that is possible, and that is what we are occupied with exactly.

@pursehouse

Is there any organized list of the websocket work that is in progress so that others can attempt to help with it?

The main difficulty is not in doing the routine, but is in the proper portable and robust design. We are going to publish the draft implementation as soon as we have it. If you can come up with a proper portable and robust design right now – your thoughts on this are welcome.

[TimNZ]

@kroitor First let me say everyone appreciates the work the team is doing.

All of us are asking the same thing - just write up a draft of the API you are currently expecting to expose for websocket support, so we can see what you are planning to provide and we can provide feedback. That's it.
You've been working on this for months, surely the first thing you did was figure out the use cases and come up with an idea of the API additions to CCXT and pseudo code.

Let us, the users of this module, see that now and provide feedback.

This work has been in progress for many many months and is well behind the timeline you kept on offering up. Naturally some of us are increasingly suspicious about your unwillingness and reasons not to share, and the ongoing delay, and want to help.

[TimNZ]

@kroitor Mate, just add a 'Realtime' section to the manual, marked WIP, with pseudo code showing usage of the proposed API, with init/config, example events flow - even a link to a quick and dirty GIST, only in JS.
That'll do nicely for now so we can see what it will look like as a consumer of the CCXT module.

Then perhaps you should consider doing a quick update here with a realistic estimate on when you will publish something we can play with.
You don't owe anyone anything, but there a lot of people waiting.

Having been in a similar situation to you, promising and not delivering, over and over, I learnt that it is better to release early and often anything I had as people want to help, and getting feedback early was a lot better in terms of engagement of community, and making changes when they were cheaper to make in terms of effort.

[maxbaluev]

I am also really need this feature and I'm ready to spend my time to testing and feedback.

[mmehrle]

"_The main difficulty is not in doing the routine, but is in the proper portable and robust design._ We are going to publish the draft implementation as soon as we have it. If you can come up with a proper portable and robust design right now – your thoughts on this are welcome."

Yeah, as much as I want to start contributing I would tend to agree when it comes to the design aspects of this. It's not trivial and it'll be a pisser to redesign later down the line. Until you sort all that out we remain in eager anticipation ;-)

[jjhesk]

@kroitor any drafts that i can look at now?

[kroitor]

@jjhesk working on adding the spec to the Manual, as requested by @TimNZ . No implementation draft yet.

[pursehouse]

I think first phase would need to be just getting the websocket configurations to be manageable in each base JS config. Because the style to implement in each language is going to be totally different. But if it could universally be handled just for creating the right url, that would be helpful. Thoughts?

[kroitor]

@pursehouse our thoughts started off with this https://gist.github.com/kroitor/aabecc346a5bded6ead7a82205d8ffc1

This is just the intro to an intro, the entire Streaming chapter and all that follows will be published in the Manual, hopefully, soon. This will eventually turn into a spec for WS support as requested here.

It has to be done in a cross-compatible way in 3 languages: JS, Python (both 2 and 3) and PHP, with respect to these rules: https://github.com/ccxt/ccxt/blob/master/CONTRIBUTING.md

[jjhesk]

@kroitor ok, i think the draft is pretty complete here. One more thing need to address is the websocket versioning support and its management.

[jjhesk]

@kroitor hows the progress?

[kroitor]

@kroitor hows the progress?

Let's see... I posted the above answer to satisfy the crowd 48 hours ago exactly.

Since then, I took a sleep two times for approx 6 hours, that is 12 hours in total, 36 hours remaining.
I also took several meals and did some other things that took approx 4 hours in total, 32 hours remaining. I did some work to earn money for living, that took approx 8 hours of time, nobody, except a few people who donated (thanks and respect to them), actually pays for the opensource work, remember? ) So, in order to have something to eat, we have to work on other projects as well.

That gives us a remaining time of 32-8 = 24 hours, one full day to work on ccxt. Then I spent approximately 4 hours answering how is it going to different people here and there. That is 20 hours left for the actual work. Within that time I closed 20 new issues (answered them, some of them multiple times), and merged approximately 30 pull requests with various bugfixes, enhancements and edits to the existing codebase. That is roughly 20 minutes per one issue/pull request. Some of them require intense communication and take longer, some are closed faster.

Now, I can't imagine how you could possibly have any serious progress on this particular issue within that timeframe. And my answer and attitude towards this type of questions is fully expressed in this comment: https://gist.github.com/kroitor/aabecc346a5bded6ead7a82205d8ffc1#gistcomment-2335076

Basically, the progress of this open source project is also open, so whenever you wonder how is it going, take a look into commit history )

[dmitriz]

Any thoughts on this proposal?
https://gist.github.com/kroitor/aabecc346a5bded6ead7a82205d8ffc1#gistcomment-2401614

[kroitor]

@dmitriz thx! Answered there )

[dmitriz]

It seems there is no way to receive
notifications for the new comments in Gists,
so I would like to propose to move the discussion here,
where we do have this and other nice facilities.

In relation to @kvdveer comment in
https://gist.github.com/kroitor/aabecc346a5bded6ead7a82205d8ffc1#gistcomment-2403346
I agree that each language should use its own preferred idiom
and that my suggestions are mostly focused on the JS.

However, it would be sad to have the "subclassing" forced onto languages that don't need it.
I find the class based JS code often very hard to understand, because the encapsulation is too often broken with classes referring to each other inner details whose definitions are spread over huge number of files.

I recently came across a whole ecosystem of stream libraries around
https://github.com/pull-stream/pull-stream
maintained by @dominictarr (who is one of the most prolific contributors in the JS-Node community). Both code and the API are very elegant and simple and make no use of classes.

[lfern]

@kroitor I have been testing about websockets feature. I have used react/ratchet and evenement in php and ws and EventEmitter in javascript. I don't know if this could be a valid aproach to ccxt, but I have created php and js examples to connect cex websocket and get orderbook updates, and it works (many things to check, but I see updates in execution log). If you think it could be a good starting point I would like to test python too.

[lfern]

Sorry, I forgot the github link: https://github.com/lfern/ccxt/tree/feature/websockets

[pursehouse]

@lfern nice, did you work on a structure for the per-exchange websocket configs? I think that was the real hurdle

[lfern]

@pursehouse I have tried to put all code related to websockets out of code writen for every exchange. This way I have defined a base class AsyncConnection

https://github.com/lfern/ccxt/blob/feature/websockets/js/base/async/async_connection.js

that only defines this methods: connect, close, send and sendJson. This base class inherits from EventEmitter and will emit events related to websockets events. In this test I have defined thre eevents:

• 'message' when message received from connection,
• 'error' when error ocurred and
• 'close' when connection is closed by server.

I have defined a child class WebsocketConnection, from AsyncConnection,

https://github.com/lfern/ccxt/blob/feature/websockets/js/base/async/websocket_connection.js

that implements simple websocket connection (if any exchanges define a different connection type, you could define new child classes from AsyncConnection, such as pusher in bitstamp). In Exchange.js you can capture this events

https://github.com/lfern/ccxt/blob/feature/websockets/js/base/Exchange.js#L1176

and invoke custom functions defined into exchanges classes in *.js.

In cex.js I have defined _asyncOnMsg method

https://github.com/lfern/ccxt/blob/feature/websockets/js/cex.js#L539

that gets the message received from websocket, parses it and do whatever have to do. I have modified Exchange class to inherits from EventEmitter too, so you can send events from cex.js. In this testing I have only implemented order book process, so when _asyncOnMsg method detects that orderbook is received, it process and emits 'ob' event.

https://github.com/lfern/ccxt/blob/feature/websockets/js/cex.js#L563

This event could be captured from main program to implement any action on receipt. I have defined _asyncSubscribeOrderBook method on cex.js

https://github.com/lfern/ccxt/blob/feature/websockets/js/cex.js#L596

that is invoked from asyncSubscribeOrderBook method in Exchange.js. In this _asyncSubscribeOrderBook method you can generate the message to be sent to websocket to make orderbook subscription (you would define for each exchange).

Right now the configuration structure for asynchronous connections is like this:

'async': {
'type': 'ws',
'url': 'wss://ws.cex.io/ws/',
'wait4readyEvent': 'auth',
},

where type is the type of connection (Websockets in this case), url is the websocket url and wait4readyevent defines the event that you must wait until you can subscribe or send any command to the websocket (in cex yo must do authentication before send any subscription request). But this structure must be revised when new asynchronous connections would be added. I have tested in node and php, and it seems it could work.

https://github.com/lfern/ccxt/blob/feature/websockets/examples/js/async-orderbook-cex.js

https://github.com/lfern/ccxt/blob/feature/websockets/examples/php/async_orderbook_cex.php

[lfern]

I think that it could be a good starting point to implement websockets in exchanges, but I don't want to go on if you think that it isn't.

[pursehouse]

@lfern looks like a good start to me. I'd try to get those big if statement blocks out of the cex._asyncOnMsg method, if you don't want that method to end up giant in each exchange in the long run. _async_orderBookSubscribe or whatever...

cuz I think ideally... you'd have a list of methods and their websocket key so that a more generic single method could be running the data check. So the exchange config would have an array of the keys and their corresponding method to process that data. Then the Exchange class would be the only method needing extensive logic written into it for checking what to do with the result. Or even have them 1:1 named
instead of if( e === 'order-book-subscribe' ) .... have it call something like cex._async_eventOrderBookSubscribe similar to on the already existing method mapping concept

maybe try out a few more exchanges with very different web sockets systems to see what issues/refinements come up.

remember a key factor will be how much less code needs to exist per-exchange that can be handled generically

[dmitriz]

I'd like to share this websocket implementation for Binance with fully functional code:
https://github.com/zoeyg/binance/issues/57

An important part is function re-usability and separation of concerns.
For instance, the whole WebSocket related part is completely encapsulated in this function:

// general purpose websocket factory
const WebSocket = require('ws')
const createWS = path => cb => 
  new WebSocket(path).on('message', cb)

The rest are pure functions that can be composed and tested in isolation.

[lfern]

@dmitriz I don't use streams before, but do you think this aproach could be used in other kind of connections? Bitstamp use a Pusher connection for trades, orderbook and tickers, and in gemini you have to connect to a diferent ws url for each symbol. Do you know a similar library to implement it in pyhton & PHP?

[dmitriz]

@lfern
Different ws url should not make any difference, you just compose the createWS factory with another suitable function getUrl.

I have no experience with Pusher, is that some sort of wrapper around WS?
Then it would replace the createWS factory by factory of its own, with the same idea:

const createPusher = (channel, event, key) => cb => 
  new Pusher(....).on(event, cb)

or something of that kind
(I don't know the exact syntax but they are usually similar, so I am just making it up.)

I don't have as much experience with Python or PHP
but the principle should be the same - you create general use function
that subscribes your callback function to the stream.
If it is general and reusable, you can nicely isolate it from the rest of the code.

[pursehouse]

@dmitriz @lfern the socket code doesn't go inside the exchange class, it goes in the language specific classes. The exchange classes just have config information and maybe a little exchange specific logic.
don't get hung up on what nodejs library to use, because that part is not relevant to getting the overall ability able to function.

the key is getting the exchange class config structure able to handle anything that the variety of exchanges can throw at it.

Most likely it would make sense for the library to just support building/reading the data for the sockets, not make a decision on what socket library to be using

[dmitriz]

@pursehouse

the socket code doesn't go inside the exchange class, it goes in the language specific classes.

Could you explain which classes you are referring to?
Is the Exchange class under /js/base, from which exchanges inherit, the language specific one? Then what classes are not language specific?

don't get hung up on what nodejs library to use, because that part is not relevant to getting the overall ability able to function.

Agree that it is good to keep these libraries in one place.
Where would you put them?

the key is getting the exchange class config structure able to handle anything that the variety of exchanges can throw at it.

If we can assume the exchanges always return some JSON, that is what the receiver functions will be passed as arguments. Or do you mean some post-prossessing the data based on certain configs?

Most likely it would make sense for the library to just support building/reading the data for the sockets, not make a decision on what socket library to be using

Indeed, it would be good to use some generic interface. As in my example, where the function signature only has the url and callback, that could be fitted into any other socket library.
Perhaps some source object abstraction containing all the information needed to retrieve the stream?

But at the end, the library will still have make some choice which socket lib to use, won't it?

[pursehouse]

Only the classes like binance or gdax are not language specific. The others are language specific.

What is needed, is a configuration/method structure that can exist in each exchange class which would create the data needed to create a socket connection. Meaning the URL and potential post/get data included. Then another set of rules/methods to parse the data returning in each cycle from those websockets.

There are way too many concepts of which specific event based library someone wants/needs to use, so having those features to do the simple handling of populating the socket connection information, and processing the results of that connection makes it so that there could be a specific or barrage of different socket libraries implemented to use. Because you still need to implement code to process that data.

$exchange = new \ccxt\binance();
$socketInfo = $exchange->getSocket( $someIdeaOfParams );
$socketLibrary = \SomeSocketLib();

\Amp\Loop::run( function () use ( $streamsUrl, $exchange ) {
$connection = $socketLibrary->open( $socketInfo['url'], $socketInfo['post'] );
while( $message = yield $connection->receive() ) {
/// stuff..
}
}

etc...

it's way to project specific to try and handle the actual socket connection.

the main hurdle is a viable repeatable config concept that works across all exchanges. The rest is secondary and not even really important to be inside the library... unless people feel like committing examples for different socket libs

[pursehouse]

@dmitriz I think you need to understand the functionality better to weigh in on this... just let @lfern continue his work

[dmitriz]

@pursehouse

@dmitriz I think you need to understand the functionality better to weigh in on this... just let @lfern continue his work

I find this comment rather rude, especially from someone who never contributed to this project.

[pursehouse]

@dmitriz sorry man your questions are just far too confused of how the split up between what can be reused and what can't. frustrating to explain it, just wait.

[imhazige]

this is a amazing project!

what's the timeline of the websocket api support?

[pursehouse]

@imhazige it's pretty much in @lfern 's hands at the moment, you could jump in and help though if you have time

[imhazige]

@pursehouse

ok, thanks, I think it should be this feature https://github.com/lfern/ccxt/tree/feature/websockets

[jraicr]

I will reproduce my comment from #3023

Hi guys, I need some help regarding this library. I'm a bit new at programming in the javascript world and I'm tring this to build some backend data calculation.

I'm trying to go through all the exchanges and download their tickers. In the loop I'm building a json structure where I can go through all the data to do calculations. I would like to do this fetching more faster since I'm planning to build a web client with some services, starting with a realtime coin marketcap. Websocket fetching will be available for ccxt?

Is it viable to do it this way? You can check the source here.

[pursehouse]

@jraicr i'd recommend branching from the work that @lfern started and try to continue his work on getting a generic format for the per-exchange configs to handle url building and result processing. The code to actually create a socket should likely be kept separate because of how different each language uses sockets. But adding a functional optional library as an example would certainly help people after the base websocket config system logic is built out.

And also read the many notes we've added about details in this thread :)

[lfern]

I have been testing in this feature https://github.com/lfern/ccxt/tree/feature/websockets . I have tested getting orderbook with gateio, binance, cex and coincheck, but I think its design is not enough robust yet. I would like to test it with other websockets exchanges. Do you know where to find most important exchanges with websocket support (not push protocol for now)?

Thanks

[kroitor]

@lfern you might want to look into gdax, bitmex, bitfinex as well... we hope to compile the complete list in ccxt... but until then, I think, the procedure is to go from CoinMarketCap ranking → CCXT exchanges' list → Exchanges' API docs.

[pursehouse]

@lfern yeah like he said, test those ones and from the top rated exchanges and work your way down the list till you feel you've tweaked things enough

[lfern]

I tested lbank & zb websocket orderbook successfully, but I have some questions:

1. I have created a context object/dictionary to put all temporal info you need about the websocket management. This dictionary could be accessed from any exchange code (eg binance.js) so you could get/set any data from/to it. If you need to store a value in this context, you could do this way:

let ethBtcSymbol = this.context['ob']['ETH/BTC'];
ethBtcSynbol['ob'] = obReceived;

But when this is transpiled to PHP, you could not get a reference to this.context['ob']['ETH/BTC'], you really get a copy of this dictionary, so you don't set this value in this.context. I think the only way to do this is to code this way in .js:

this.context['ob']['ETH/BTC']['ob'] = obReceived;

Do you know hoy we could resolve this?

1. I have defined all new methods with "async" prefix, Do you think this is fine or its better another prefix or not to use prefixes in this functions?

2. How could we merge this code on ccxt? Do you think this could be merged in a separte branch, or it is better to keep on this branch?

https://github.com/lfern/ccxt/tree/feature/websockets-multiple

I think we need other people tests this code and get feedback from them. Its first time I use asyncio and reacphp and I am sure I made many mistakes in the implementation.

[pursehouse]

@lfern if you're best with JS then just stick with solving all the JS stuff. I'll try and checking the PHP issues asap. But l would suggest not abbreviating things like _asyncHandleOb. took my a while to figure out what "ob" meant :) might as well keep things as verbose as possible.
As far as the naming question for async or whatever... I'd think that going with socket or websocket might be better because there are more async concepts than just websockets... so that'd prevent future problems. And less confusion against the js : async setting already

[kroitor]

But when this is transpiled to PHP, you could not get a reference to this.context['ob']['ETH/BTC'], you really get a copy of this dictionary, so you don't set this value in this.context.

That's right, PHP arrays are copied by value, not by reference (contrary to JS and Python).

I think the only way to do this is to code this way in .js:

this.context['ob']['ETH/BTC']['ob'] = obReceived;

Do you know hoy we could resolve this?

I'm afraid, the only way to resolve this in a transpileable way is to either create a set of base-class methods for working with object references, or to use the syntax from your line above.

But l would suggest not abbreviating things like _asyncHandleOb.

I would agree with @pursehouse here, "OrderBook" or "orderbook" would be a much more friendly name for "ob".

I have defined all new methods with "async" prefix, Do you think this is fine or its better another prefix or not to use prefixes in this functions?

I'd rather not use prefixes at all. I'd use the method naming that would be explicit in what it's doing without duplicate prefixes. Usually, it's in the form of "verb-noun(s)", like "updateData", "saveFile", "fetchOrderBook", "createOrder", "handleOrderBookDeltaUpdate"...

How could we merge this code on ccxt? Do you think this could be merged in a separte branch, or it is better to keep on this branch?

https://github.com/lfern/ccxt/tree/feature/websockets-multiple

We will start putting this into a separate branch of CCXT. However, we will need to rework the WS code itself somewhat to make it compatible with SignalR, pure/custom WS and socket.io protocols (those are three most popular streaming flavours used commonly across many exchanges). And for that we will have to make sure that we are not using any global variables or any instance variables. Otherwise this might produce tons of conflicts in the async concurrency (we're already having them in Python due to instance variables like self.last_http_response and alike).

Namely, if you can avoid using this.context['ob'], and use the function argument instead to pass it, this would help resolve the above "copy-by-value" problem somewhat, and would also make the code more robust. So, we would trade the globals and instance variables for local scope variables and for async generators.This way we would prevent possible conflicts in future. This, however, would be a significant change to your code. Generally, we prefer pure stateless functions (that are not altering a global state or instance state). Dirty stateful methods usually conflict with each other in a concurrent environment (racing conditions, etc...).

[kroitor]

How could we merge this code on ccxt? Do you think this could be merged in a separte branch, or it is better to keep on this branch?

A little more on these questions... We will start by collecting a list of ws-enabled exchanges, and will add a WS column to our exchanges' table, then will start filling it out. Hopefully, we'll have a dozen of exchanges with WS support in the first stable version of it. Once this is done, we are going to merge it to the master branch. And we will keep adding the rest of them from there.

[lfern]

We will start putting this into a separate branch of CCXT. However, we will need to rework the WS code itself somewhat to make it compatible with SignalR, pure/custom WS and socket.io protocols (those are three most popular streaming flavours used commonly across many exchanges). And for that we will have to make sure that we are not using any global variables or any instance variables. Otherwise this might produce tons of conflicts in the async concurrency (we're already having them in Python due to instance variables like self.last_http_response and alike).

I don't know how to do this without instance variables, because you must keep the live orderbook and modify it on every delta update, and process it in the same order as you receive them from websocket. So I think you can´t process incoming websockets messages using differents threads. (but I don't know so much about asyncio). Maybe this approach is not good to achieve this requirement.

Anyway, I think It is not a good idea to test more exchanges before these changes in WS code (to make it compatible with those protocols: SignalR, ...) would be ready. But I don't know how to collaborate in this job. I thought this could be achieved developing a wrapper over these protocols and, using EventEmitter, emitting 'on-open', 'on-message', 'on-error' and 'on-close' events. But if this is not a good aproach, maybe it would be better to start again and define a new class/interface that works fine with all of these protocols.

[kushn1ruk]

Maybe introduce some hint for JS to PHP transpiler to use the reference & instead of copy?
http://php.net/manual/en/language.references.whatdo.php

[kroitor]

Maybe introduce some hint for JS to PHP transpiler to use the reference & instead of copy?

Yeah, this could be a workaround, however, the goal is to keep the codebase clean in pure target languages, otherwise those hints would be there all over this repo already. We would rather add a base method like updateArrayByReference (&array).

[kroitor]

@lfern

I don't know how to do this without instance variables, because you must keep the live orderbook and modify it on every delta update, and process it in the same order as you receive them from websocket.

The usual way of doing it is not to use instance variables, instead, you pass a reference to the orderbook into the update handling method, along with the update data itself. So, if you pass the proper references through the entire call chain, then we won't need instance variables at all, right?

The main problem with instance variables (those are almost alike "global" variables), is when async responses get out of order, you might have a lot of side-effects, conflicts and inconsistencies with methods that do mutation of those instance variables. As I said, we had some of those earlier, so we'd better not repeat the mistake again.

[lfern]

ok, @kroitor . Maybe someone with a great experience in python asyncio could take a look into exchange.py. I couldn't find where I should have to place this local variable (live orderbook and other status variables) to be visible in any exnchange method that handle async messages from websocket. If this is not posible with actual code written in python, we will have to adapt node & PHP code to needed changes for asyncio in multithread mode.

[kroitor]

@lfern

Maybe someone with a great experience in python asyncio could take a look into exchange.py.

There are two exchange.py files, one for the sync python version, and one for the async python version, derived from the sync one. So, you're probably looking for https://github.com/ccxt/ccxt/blob/master/python/ccxt/async/base/exchange.py

Or I may have misunderstood the question, if so, please let me know.

I couldn't find where I should have to place this local variable (live orderbook and other status variables) to be visible in any exnchange method that handle async messages from websocket.

No worries on that, basically, what I mean is, instead of doing this:

function initializeOrderBook (snapshot) {
    this.context['ob'] = { ... snapshot ... }; // new orderbook state, for example
    this.ws.handler = this.updateOrderBook
}

function updateOrderBook (deltas) {
    const oldOrderBook = this.context['ob'];
    const newOrderBook = parseDeltas (deltas, oldOrderBook);
    this.context['ob'] = newOrderBook;
}

...

↓ instead of the above it is better to do something like this (pseudocode):

// a base method
function initializeOrderBook (snapshot) { 
    const newOrderBook = new ccxt.IncrementalOrderBook ({ ... snapshot ... }) // initialize it
    // should pass by ref in PHP in the base code, class instances are passed by reference
    this.ws.handler = (deltas) => {
        const updatedOrderBook = this.updateOrderBook (newOrderBook, deltas) 
        this.orderBookUpdated (updatedOrderBook)
    }
    return newOrderBook; // return it, don't store it in an instance variable
}

function updateOrderBook (orderBookInstance, deltas) {
    // mutate orderBookInstance
    orderBookInstance.update (deltas)
    return orderBookInstance;
}

In the second example the orderbook itself is an object "passed by reference", and returned from the calls, so no calls interfere with each other by writing to a shared state. The sync primitives should be in the orderbook itself. Not sure if I'm explaining it with this example...

Also, it is not recommended to mix asyncio + threads. We better either use just the threaded concurrency or just the async concurrency (preferred), and you can't really use both concurrencies at the same time, as they begin to... uhm... compete.

What I meant in my previous post is that we should just pass it along with the arguments to all calls interested in it, and return the updated value from those calls, instead of writing to the instance variable. This should the PHP issues, even when the value is passed by copy.

One of the key-points of PHP5 OOP that is often mentioned is that "objects are passed by references by default". This is not completely true.
A PHP reference is an alias, which allows two different variables to write to the same value. As of PHP5, an object variable doesn't contain the object itself as value anymore. It only contains an object identifier which allows object accessors to find the actual object. When an object is sent by argument, returned or assigned to another variable, the different variables are not aliases: they hold a copy of the identifier, which points to the same object.

Hope this helps, will be happy to answer if you have more questions on this.

[lfern]

Thanks @kroitor I am sure it's implict here but, sorry, I still can´t realize it. Please let me follow it from your code example:

// a base method in Exhange.js
function initializeOrderBook (snapshot) { 
    const newOrderBook = new ccxt.IncrementalOrderBook ({ ... snapshot ... }) // initialize it
    // should pass by ref in PHP in the base code, class instances are passed by reference
    this.ws.handler = (deltas) => {
        const updatedOrderBook = this.updateOrderBook (newOrderBook, deltas) 
        this.orderBookUpdated (updatedOrderBook)
    }
    return newOrderBook; // return it, don't store it in an instance variable
}
// a base methid in Exchange.js
function updateOrderBook (orderBookInstance, deltas) {
    // mutate orderBookInstance
    orderBookInstance.update (deltas)
    return orderBookInstance;
}

In cex.js I have define an entry method to parse any message received from websocket.

/* ...*/
processWebsocketIncomingMessage (msg) {
  // parse msg
  if (orderbook snapshot) {
    let orderBook = this.initializeOrderBook (msg.snapshot);
    return orderbook;
  } else if (orderbook delta) {
    /* here I need to recover incremental orderbook and then*/
  } else if (orderbook subscription) {
    /* here I need to recover a promise set when I send a subscription message (in other cex.js method) to tell the user subscription succeded*/
   }
}

This method is invoked from Exchange.js:

/* receive msg from websocket */
let orderbook = this.processWebsocketIncommingMessage(msg);

With this model I think I have a problem because I don't know if incoming mesage is an orderbook mesage in Exchange.js (this is don in cex.js), but in case I could where I could save this orderbook variable and then recover it in Exchange.js, so I could inject it (when delta message is received) in this processWebsocketIncommingMessage method without saving it in an instance variable?

Sorry, sure it is trivial, but now I can't see how to do it.

[kroitor]

@lfern sorry, i'm probably confusing you more than clarifying anything ) But anyways, let's not focus on instance variables, in fact we can get away with instance variables as well. Basically, there is no problem with your implementation of processWebsocketIncomingMessage (msg) in the async environment – this code won't trigger a conflict, however, accessing an instance variable may be a problem in a threaded environment (with PHP). More to it, PHP passes all arguments by copy, unless those are instances of a class.

So, we have the following choices:

• either design a standalone OrderBook class to avoid using the instance variables and make it a unique instance per thread:

// in main thread:
let exchangeInstanceInQuestion = new ccxt.cex ({ ... })

// in each worker thread:
let orderbook = new ccxt.OrderBook (exchangeInstanceInQuestion)

the orderbook will then encapsulate processIncomingOrderBookDelta method as well as other methods needed, and the user will have separate orderbook instance in each thread...

• or add primitive synching base methods to actually make the code thread-safe:

processWebsocketIncomingMessage (msg) {
  // parse msg
  if (orderbook snapshot) {
    let orderBook = this.initializeOrderBook (msg.snapshot);
    return orderbook;
  } else if (orderbook delta) {

    // lock // ←-------------- added

    /* here I need to recover incremental orderbook and then*/

    // unlock // ←-------------- added

  } else if (orderbook subscription) {
    /* here I need to recover a promise set when I send a subscription message (in other cex.js method) to tell the user subscription succeded*/
   }
}

So, you're right, there's no easy way to avoid the instance variables and make this thread-safe, without taking thread-safety into consideration from the very beginning of code design. We are not thread-safe at this moment, so to workaround this entire issue a separate exchange instance per thread is required in order to avoid possible racing. This might be a good resolution to the issue – basically we should discourage everyone from sharing the instances across threads in the first place.

Ultimately, it has to be one of:

• one orderbook per thread
• or one exchange per thread
• or has to contain syncing mutexes

Hope it makes sense.

P.S. To me there's actually very little point in threading the WS connection handling – this isn't something that should be threaded, instead a thread pool should manage separate WS re/connections.

UPD. sorry for the typos

[imhazige]

@lfern which branch should I look into if I want to test the latest websocket feature?
this one https://github.com/lfern/ccxt/tree/feature/websockets
or this one https://github.com/lfern/ccxt/tree/feature/websockets-multiple

[pursehouse]

I think it's less important to solve the concept of how to use sockets for the users, as much as it is to isolate the call creation and packet parsing so that they could be used by a potential premade socket calling method, or by their own cycles than an example could be distributed of for each language

[imhazige]

I really eager to see be able to use the socket feature with the latest ccxt version, in the browser by using javascript.

[lfern]

@imhazige the current branch is https://github.com/lfern/ccxt/tree/feature/websockets-multiple You can find an example in async_orderbook.js (same for ,php and .py). This example expects parameters:

node examples/js/async_orderbook.js [market2 ...]

Orderbook is implented in cex, gateio, binance, bitmex, lbank and zb. Orderbook delta updates need to be revised, but if you set verbose mode you can see incoming messages and hoy orderbook event is emited.

[imhazige]

@lfern

Thank you.

I just test the examples/js/async_orderbook.js, found the proxy did not work.

The verbose option is open, I can see the request is through the proxy. but it did not work, each time throw a request timeout error.

I then test other example without websocket, like live-ticker.js, still did not work. while at the latest ccxt repository mater branch, the example do works.

I think that's because the code base is far behind the mater branch of the ccxt repository.

Is it possible to merge the latest code of ccxt?

And I have a question, will the websocket support proxy?

[lfern]

@imhazige
merged with ccxt master. I didn't test it through a proxy. I am just doing some tests and I think we need another config variable for websocket proxy, because:

prepended to URL, like https://proxy.com/https://exchange.com/api...

does not work.

[imhazige]

@lfern

Thank you, I see the code have merged the ccxt master v1.14.240. now the http proxy works.

but when I try the websocket proxy, it did not work.

I tested with the proxy url in the example code http://185.93.3.123:8080/, it did not work

Did it work in your case?

[kroitor]

I tested with the proxy url in the example code http://185.93.3.123:8080/, it did not work

• To use it with HTTP(S) proxies, see this: https://github.com/ccxt/ccxt/wiki/Install#proxy
• To use it with CORS proxies, read this: https://github.com/ccxt/ccxt/wiki/Install#cors-access-control-allow-origin

Sorry, if this is not what you were asking for.

[imhazige]

@kroitor

Thank you for the examples.

for http, use CORS proxies is enough.

but when using websocket, I was thinking use this way https://github.com/imhazige/node-http-connect-proxy, after trying, it did not work in some country. :(, I have no idea now.

[lfern]

htto-proxy-agents seems to work fine with node websocket.

example.js

    const proxy = process.env.http_proxy || 'http://185.93.3.123:8080' // HTTP/HTTPS proxy to connect to
    const agent = new HttpsProxyAgent (proxy)
    exchange = new ccxt[id] ({
        apiKey: apiKey,
        secret: secret,
        enableRateLimit: true,
        verbose: true,
        agent: agent,
    });

But Autobanh/asyncio(python) is not working when try to use proxy:

2018-06-22T11:32:03 connection to tcp:104.20.34.190:443 established
2018-06-22T11:32:03 CONNECT ws.cex.io:443 HTTP/1.1
Host: ws.cex.io:443


2018-06-22T11:32:03 received HTTP response:

b'HTTP/1.1 400 Bad Request\r\nServer: cloudflare\r\nDate: Fri, 22 Jun 2018 10:32:01 GMT\r\nContent-Type: text/html\r\nContent-Length: 171\r\nConnection: close\r\nCF-RAY: -\r\n\r\n'


2018-06-22T11:32:03 received HTTP status line for proxy connect request : HTTP/1.1 400 Bad Request
2018-06-22T11:32:03 received HTTP headers for proxy connect request : {'server': 'cloudflare', 'date': 'Fri, 22 Jun 2018 10:32:01 GMT', 'content-type': 'text/html', 'content-length': '171', 'connection': 'close', 'cf-ray': '-'}
2018-06-22T11:32:03 failing proxy connect ('HTTP proxy connect failed (400 - BadRequest)')
2018-06-22T11:32:03 dropping connection to peer tcp:104.20.34.190:443 with abort=True: None
2018-06-22T11:32:03 _connectionLost: None
Traceback (most recent call last):
  File "src/py/async-orderbook.py", line 84, in <module>
    loop.run_until_complete(main())
  File "C:\Users\Luis\AppData\Local\Programs\Python\Python36\lib\asyncio\base_events.py", line 468, in run_until_complete
    return future.result()
  File "src/py/async-orderbook.py", line 64, in main
    await exchange.async_subscribe('ob', symbol)
  File "c:\users\luis\documents\projects\ccxt\python\ccxt\async\base\exchange.py", line 669, in async_subscribe
    await self._async_ensure_conx_active(event, symbol, True)
  File "c:\users\luis\documents\projects\ccxt\python\ccxt\async\base\exchange.py", line 480, in _async_ensure_conx_active
    await self.async_connect()
  File "c:\users\luis\documents\projects\ccxt\python\ccxt\async\base\exchange.py", line 490, in async_connect
    await async_connection.connect()
  File "c:\users\luis\documents\projects\ccxt\python\ccxt\async\async\websocket_connection.py", line 90, in connect
    await future
Exception: connection was closed uncleanly (None)

[imhazige]

@lfern

I tested the examples with agent option

const proxy = process.env.http_proxy || 'http://185.93.3.123:8080'; // HTTP/HTTPS proxy to connect to
  const agent = new HttpsProxyAgent(proxy);
  exchange = new ccxt[id]({
    apiKey: apiKey,
    secret: secret,
    enableRateLimit: true,
    verbose: true,
    agent: agent
  });
  exchange.on('err', (err, conxid) => {
    try {
      console.log(err);
      exchange.asyncClose(conxid);
    } catch (ex) {
      console.log(ex);
    }
  });

get a error

fetch:
 binance GET https://api.binance.com/api/v1/exchangeInfo
Request:
 {}
 undefined

Error: [RequestTimeout] binance GET https://api.binance.com/api/v1/exchangeInfo request timed out (10000 ms)

           at e  js/base/Exchange.js:388  throw new RequestTimeout (this.id + ' ' + method + ' ' + url + ' request timed …

is the proxy server http://185.93.3.123:8080 running? what's kind of proxy server it is?

[lfern]

Sorry, I don't have a proxy server in my network, and I didn't want to waste time installing it for testing, so I try a free proxy from free-proxy-list.net . I think these proxies aren't always available.

[imhazige]

@lfern

ok, thanks, I meant the proxy will not work in "some country".

[imhazige]

so how can I use this feature via npm? or I have to import the compiled code directly?

[lfern]

In a development environment I clone from lfern/ccxt in a local folder
(./ccxt) and create a new project (./test). Then exec 'npm init', 'npm
install ccxt -save' and 'npm link ../ccxt'. Or if you prefer, you can
import directly.

[imhazige]

@lfern

good to know, thank you.

[imhazige]

@lfern

I want to add the okex websocket feature, is there any develop guide for the implementation?

[lfern]

@imhazige Sorry, there aren't any doc ready yet. I have added some tips in
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/WEBSOCKETS.md
but you can look at cex.js, because I think orderbook subscription is similar to okex. Anyway, send me
any question about it and I would help you to enable it.

[lfern]

@imhazige maybe you can use this skeleton as starting point:

...
,
            'asyncconf': {
                'conx-tpls': {
                    'default': {
                        'type': 'ws',
                        'baseurl': 'wss://real.okex.com:10441/websocket',
                    },
                },
                'methodmap': {
                },
                'events': {
                    'ob': {
                        'conx-tpl': 'default',
                        'generators': {
                            'url': '{baseurl}',
                            'id': '{id}',
                        },
                    },
                },
            }
...
  _asyncOnMsg (data, conxid = 'default') {
    let msg = this.asyncParseJson (data);
    console.log(msg);
  }
  _asyncSubscribe (event, symbol, nonce) {
        if (event !== 'ob') {
            throw new NotSupported ('subscribe ' + event + '(' + symbol + ') not supported for exchange ' + this.id);
        }
        this.asyncSendJson ({
          event: 'addChannel',
          channel: 'ok_sub_spot_' + pairId + '_depth',
        });
    }

[imhazige]

Thank you so much, I will have a look today.

On 2018/6/28 00:43, lfern wrote:
>

@imhazige https://github.com/imhazige maybe you can use this
skeleton as starting point:

...
,
'asyncconf': {
'conx-tpls': {
'default': {
'type': 'ws',
'baseurl': 'wss://real.okex.com:10441/websocket',
},
},
'methodmap': {
},
'events': {
'ob': {
'conx-tpl': 'default',
'generators': {
'url': '{baseurl}',
'id': '{id}',
},
},
},
}
...
_asyncOnMsg (data,conxid = 'default') {
let msg= this.asyncParseJson (data);
console.log(msg);
}
_asyncSubscribe (event,symbol,nonce) {
if (event !== 'ob') {
throw new NotSupported ('subscribe ' + event + '(' + symbol+ ') not supported for exchange ' + this.id);
}
this.asyncSendJson ({
event: 'addChannel',
channel: 'ok_sub_spot_' + pairId+ '_depth',
});
}

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ccxt/ccxt/issues/56#issuecomment-400748176, or
mute the thread
https://github.com/notifications/unsubscribe-auth/AD7pFvG0YcY0lhX9tlQ_YdNpyrb0mkeaks5uA7YlgaJpZM4OjakB.

[imhazige]

@lfern
For this method asyncSubscribe (event, symbol), could we add a third parameter params, so we can add customer params like depth, and for future orderbook, it need a parameter of contract_type

[lfern]

@imhazige Of course! I think it is better to get full orderbook to manage a local order book correctly, but this depth parameter could be usefull to notify a fixed length orderbook. Do any change you need, because the current code is only a proposal and a few websocket exchanges has been tested.

[GoChartingAdmin]

Do we have a list of all exchanges that offer a websocket (of any sort) connection. I have collected a total of ~25-30 such exchanges and could add to the list that you may be maintaining at your end

[lfern]

@kroitor what do you think about adding these columns to the ccxt exchange list (ws ver, wsDoc)?
https://github.com/lfern/ccxt/tree/feature/websockets-multiple

[kroitor]

@lfern sure, I'll do that shortly. However, if you don't mind, I'd put that column into the wiki under the Streaming section (does not exist there yet). I would not put that on the main list in the README.md just yet, because if I do, noob people won't read anything and will think that those columns indicate websocket support in CCXT and will just flood us with questions on ws and all that, which is not what we need right now ) So, if you don't mind, I'd open a new section on streaming in our Wiki for now. Will do that shortly. That would be available for editing and for submitting urls and other things to our knowledge-base.

[imhazige]

I just found that the websocket proxy will not work in the browser for okex, it need allow cross origin header too.

[anymos]

Hi @lfern

Was trying to use the binance ws.

It sounds like the function _asyncHandleObRestSnapshot that manage the binance logic about the lastUpdateId u and U, is only called one time the snapshot retrieval.

Beside, I believe that there is an issue with mergeOrderBookDelta method and searchIndexToInsertOrUpdate. This last one looks like the OB have to be sorted to find the right element and then ultimatly delete it in mergeOrderBookDelta, which never happens at least using binance websocket :(

[lfern]

I am working on refactoring functions in base classes (removing 'async' prefix in methods), and adding some class methods to access context data. Hope to be ready this week.

[anymos]

@lfern , happy to hear about the refactoring it is going to be usefull, in the mean time could you kindly have a look to the market data flow for binance which is not returning the right data ? Trying to fix it on my side but you migh find the solution quicker. You can check against binance web site to see that there are quite a number of discrepancy.

[lfern]

Removed _async string from methods and added context methods to access websocket context data. Maybe it was better to do it in a separate class, but I think it requires to modify transpile script. This context methods aren't thread safe, but I think this way it could be implemented when required. Anyway, each websocket connection has its own context data, and I think every websocket connection has to be managed by a single thread. These are the new context methods to get/save data:

    _contextGetEvents(conxid) // get events dictionary ('ob', 'tickers', ...)
    _contextGetSymbols(conxid, event) // get symbols from event 
    _contextResetEvent(conxid, event) // reset event dictionary
    _contextResetSymbol(conxid, event, symbol) // reset symbol dictionary ({'subscribed': false, 'subscribing': false, 'data':{}})
    _contextSetSubscribed (conxid, event, symbol, subscribed) // return subscribed status for event/symbol
    _contextIsSubscribed (conxid, event, symbol) // set subscribed status for event/symbol
    _contextSetSubscribing (conxid, event, symbol, subscribing) // return subscribing status for event/symbol
    _contextIsSubscribing (conxid, event, symbol) // set subscribing status for event/symbol
    _contextGetSymbolData(conxid, event, symbol) // get symbol user defined dictionary
    _contextSetSymbolData(conxid, event, symbol, data) // set symbol user defined dictionary
    _contextSet (conxid, key, data) // set data to user defined dictionary (to store any user defined data)
    _contextGet (conxid, key) // get from user defined dictionary

[lfern]

Next step: to make some tests on merge orderbook methods. Sure there are something wrong!!!

[lfern]

@anymos I have just added a test for updateBidAsk and now I think it is working fine.

[lfern]

@imhazige I have made changes in your okex code to make it transpilable and to use context methods.

[imhazige]

@lfern

Great!

[wardbradt]

Is https://github.com/lfern/ccxt/tree/feature/websockets-multiple the official branch for WebSocket development? I am interested in contributing.

[pursehouse]

@wardbradt it's a significant branch for it yes. Nothing is really "official" yet I think. But help is greatly appreciated, this is highly likely to be the merged fork I think

[skeller88]

What help is needed on that branch? Is there a doc somewhere with remaining steps and who's working on what? I'd like to make time to help with the python implementations if I can.

[lfern]

I think it would be nice if you could try to add websocket support for any exchange and write here some feeadback about problems you find or if this aproach is not really a good way to add websockets in ccxt. With this feedback, owners of ccxt project accept/reject to merge this branch.

There is an initial document with some tips about how you can add websocket support:
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/WEBSOCKETS.md

But any question about it, please write a comment in this issue and I will answer it.

[pursehouse]

Hey, I don't have time at the moment to integrate, but I have this example for a base class and extended class for converting a specific exchange kline packet payload into a generic cross-exchange format to use by the system. For PHP it's a great benefit to have a structured class to hold these things rather than a method converting. I assume node code is the same, I just don't deal with that as much

class SocketCandleKLineBinance extends SocketCandleKLine {
    public function __construct( $payload )   {
        $this->interval             = $payload->k->i;
        $this->start_time           = $payload->k->t;
        $this->trades_count         = $payload->k->n;
        $this->price_open           = $payload->k->o;
        $this->price_high           = $payload->k->h;
        $this->price_low            = $payload->k->l;
        $this->price_close          = $payload->k->c;
        $this->volume               = $payload->k->v;
        $this->taker_base_volume    = $payload->k->V;
        $this->taker_quote_volume   = $payload->k->Q;
    }
}

and..

class SocketCandleKLine {
    public $interval;
    public $start_time;
    public $trades_count;
    public $price_open;
    public $price_high;
    public $price_low;
    public $price_close;
    public $volume;
    public $taker_base_volume;
    public $taker_quote_volume;
    public $lowest_ask;
    public $highest_bid;
}

[lfern]

I would also prefer to use classes to store the data, but when doing transpilation of .js code I get error accesing attributes of any class

let x = $payload.k;

I get this transpiled code in PHP

$x = $payload.k;

Sure it could be done modifying the transpile script, but it isn't easy for me.

Maybe we could define some wrapper utils out of transpilable .js code so you can access any returned dictionary from ccxt functions to classes like this.

[pursehouse]

oh that's weird... yeah I dunno how the transpiler works.. maybe @kroitor ?

[kroitor]

@lfern the problem with generic transpiling of interpreted classes (without any strict typing and with closures) is that it's very hard to guess which variable is which type without a full-scale AST traverse in run-time and without introspection/without going into the v8/python/php "bytecode" so to speak... I'll figure something as a quick solution to this.

I get this transpiled code in PHP

Actually, this particular problem is PHP-specific and it should be easily solvable, let me work a bit on the transpile.js, will get back to you on this shortly. Just need to add a few exceptions for the dot operator.

[lfern]

Thanks @kroitor, I see that you don't create classes into .js transpilable code of exchanges, so I supposed that it could be hard to implement. I have tried to avoid it in this branch. Anyway, I think it is more important now that other contributors try to implement other websocket orderbook exchange and get feedback from them. If they like the way we are implementing websockets, then we could add classes proposed by @pursehouse

[kroitor]

I see that you don't create classes into .js transpilable code of exchanges, so I supposed that it could be hard to implement

Yep, you're absolutely right, I thought the same initially... I've just reviewed it and tried to add a quickfix, but... it didn't work well ) I'm now looking for a quick workaround to properly transpile the dot without escaping it and without tweaking the code, leaving all string-literals intact, purely by means of the transpiler... I think i'm going to need a bit more time, but I'll figure it out and will let you know asap.

[lfern]

Add websocket api doc link in table list for some exchanges:
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/README.md

[skeller88]

Lfern are you on telegram or discord? I have some questions.

EDIT: A ccxt-dev slack group would be nice. Does that exist?

On Thu, Jul 12, 2018 at 9:01 AM lfern [e-mail] wrote:

Add websocket api doc link in table list for some exchanges:
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/README.md

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ccxt/ccxt/issues/56#issuecomment-404563014, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFQFAllW7QM1RwVI_U-3BTGoV3LqumQOks5uF3LRgaJpZM4OjakB
.

[lfern]

sorry @skeller88 . I don't have telegram/discord account for now. If you don't mind, write them here and I try to answer quickly.

[lfern]

@GoChartingAdmin, would you mind to post your websocket exchange list to add to this list?

[skeller88]

@lfern I'll try and add Binance support on binance-websocket-support, a
branch that I forked off lfern/feature/websockets-multiple
https://github.com/skeller88/ccxt/tree/feature/binance-websocket-support.
Then I'll submit a PR against websockets-multiple. Does that work?

I'm trying to run examples/py/websocket-orderbook.py using "binance" and
seeing the below error. The self.wsconf Dict is empty at runtime. There's
already some code in binance.js so was the websocket-orderbook.py ever
working for you using binance? I'll assume it wasn't.

Traceback (most recent call last):
  File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py",
line 1664, in <module>
    main()
  File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py",
line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py",
line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm
CE.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in
execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File
"/Users/shanekeller/Documents/lfern_ccxt/examples/py/websocket-orderbook.py",
line 81, in <module>
    loop.run_until_complete(main())
  File "/Users/shanekeller/anaconda/lib/python3.6/asyncio/base_events.py",
line 467, in run_until_complete
    return future.result()
  File
"/Users/shanekeller/Documents/lfern_ccxt/examples/py/websocket-orderbook.py",
line 61, in main
    await exchange.websocket_subscribe('ob', symbol, {'limit': limit})
  File
"/Users/shanekeller/Documents/lfern_ccxt/python/ccxt/async/base/exchange.py",
line 712, in websocket_subscribe
    raise ExchangeError('Not valid event ' + event + ' for exchange ' +
self.id)
ccxt.base.errors.ExchangeError: Not valid event ob for exchange binance
binance requires to release all resources with an explicit call to the
.close() coroutine.
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x10c483dd8>

On Fri, Jul 13, 2018 at 9:08 AM lfern [e-mail] wrote:

@GoChartingAdmin https://github.com/GoChartingAdmin, would you mind to
post your websocket exchange list to add to this list?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ccxt/ccxt/issues/56#issuecomment-404878798, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFQFApUvfY_oNwHXIIOjxPXdP1Bel4lYks5uGMYBgaJpZM4OjakB
.

[lfern]

Sorry @skeller88 . I am not commiting changes of .py and .php transpiled files. Maybe I would have to commit these files too. You have to do npm install and then npm run-script transpile to get transpiled binance.py file. Then websocket-orderbook.py would work

[skeller88]

After running transpile, websocket-orderbook.py works. I see binance
already has websocket order book support, so I'll try and add websocket
orderbook support for bittrex. Their websocket is in beta so if I can't get
that working, I'll add it for gdax. Sound good?

Have any tests been written? I'm not finding any yet.

Are you waiting to add the python and php code until lfern/ccxt is approved
for merge into ccxt/ccxt?

On Fri, Jul 13, 2018 at 2:23 PM lfern [e-mail] wrote:

Sorry @skeller88 https://github.com/skeller88 . I am not commiting
changes of .py and .php transpiled files. Maybe I would have to commit
these files too. You have to do npm install and then npm run-script
transpile to get transpiled binance.py file. Then websocket-orderbook.py
would work

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ccxt/ccxt/issues/56#issuecomment-404957677, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFQFAhc61obJP9u6Y9Yu5iqWP6-s-0Htks5uGQ_XgaJpZM4OjakB
.

[lfern]

I think Gdax websocket doc is better than bittrex, but we are still testing, so any exchange could be interesting te check if its easy to implement. The only restriction it not to implement exchanges that uses pusher protocol (like bitstamp), or pubnut.com, Only "raw" websocket connections is supported in this branch.

Have any tests been written? I'm not finding any yet.

Sorry, I think it is not easy to write tests that simluates a websocket connection. I don't think about it, but I suppose we can create a Websocket subclass that emulates a websocket connection, but this class would be created for each exchange... Maybe someone in this thread could give us some ideas about it,.

Are you waiting to add the python and php code until lfern/ccxt is approved
for merge into ccxt/ccxt?

Yes, I thought that this could be commited when this branch would be merged. But if you think that commiting the transpiled code makes life easier anyone who wants to test it in php or python I will commit this code too! Maybe @kroitor could tell us if this would be a good idea, or it is better not to commit transpiled code for now.

[skeller88]

I tried to hook up the gdax websocket. See this branch
https://github.com/lfern/ccxt/compare/feature/websockets-multiple...skeller88:feature/websockets-multiple-gdax?expand=1.
I found the wsconfg parsing and configuration to be a bit confusing. It
seems like the whole reason for that parsing is to construct websocket urls
that are connected to, as in the case of binance. But for gdax, only one
url is connected to, and multiple channels are subscribed to via
subscription requests over that websocket connection. For gdax, it doesn't
seem like the per event level url definition and variable interpolation is
necessary right? So that configuration could be optional.

I'm getting a timeout error when I try to connect to the gdax websocket.
I'm out of time to work on this for a few days, but I'll try and help out
as much as I can.

On Fri, Jul 13, 2018 at 4:15 PM lfern [e-mail] wrote:

I think Gdax websocket doc is better than bittrex, but we are still
testing, so any exchange could be interesting te check if its easy to
implement. The only restriction it not to implement exchanges that uses
pusher protocol (like bitstamp), or pubnut.com, Only "raw" websocket
connections is supported in this branch.

Have any tests been written? I'm not finding any yet.

Sorry, I think it is not easy to write tests that simluates a websocket
connection. I don't think about it, but I suppose we can create a Websocket
subclass that emulates a websocket connection, but this class would be
created for each exchange... Maybe someone in this thread could give us
some ideas about it,.

Are you waiting to add the python and php code until lfern/ccxt is approved
for merge into ccxt/ccxt?

Yes, I thought that this could be commited when this branch would be
merged. But if you think that commiting the transpiled code makes life
easier anyone who wants to test it in php or python I will commit this code
too! Maybe @kroitor https://github.com/kroitor could tell us if this
would be a good idea, or it is better not to commit transpiled code for now.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ccxt/ccxt/issues/56#issuecomment-404976377, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFQFAiQDi3R9BmZDxnTbVYsgAeXjJFRuks5uGSoGgaJpZM4OjakB
.

[pursehouse]

yeah so there should be a generic concept for building the opening url, which needs ability to accept objects to build the url. as well as needs the socket calls for adding subscriptions as a generic concept that accepts objects.
then separately those all those object types need their own central structure that each exchange can extend to format to and from the ccxt format

[lfern]

Yes, you are right @skeller88 . Variable interpolation is not really usefull for now, because current websocket implemented exchanges except binance have only one websocket for all events. I think we need something that Exchange.js can use to know what to do when it receices a subscription/unsubscription command on event/symbol. Exchange.js manages websocket connections, so we need something in each exchange that let us know what to do on every subscription command:

• when multiple channels in only one websocket: on subscripion and if websocket is not opened then open websocket, reset context. If still opened do nothing, on unsubscription close websocket (or not) if no more subscription are active, and reset websocket context.
• when multiple channels on one websocket, but differents url connections for each channel (binance): on subscription/unsubscription you have to reconnect to websocket using a different URL (generated with previous subscriptions/unsubscriptions), and reset websocket context. On unsubscription if is the last stream active, close websocket.
• when pusher: every event/symbol has its own websocket connection info, so we need something that maps event/symbol to pusher configuration.
• when every event or event/symbol corresponds to a different websocket: on subscription open websocket for event/symbol, reset context; on unsubscription, close websocket for event/symbol.

I don't still find a exchange for this last case, but variable interpolarion would be useful to get source websocket for that event/symbol info.

Maybe it would be better to take out this from config and create methods, invoked from Exchange.js, that provides this information to manage websocket connection pool.

[pursehouse]

@lfern that's why I would suggest focusing on the individual features needed for processing data that is sent or received with the socket; rather than solving how to use sockets in general.
That way the requirements can be split into phases, and potentially releasing the core functionality features with examples on how to use them, can be followed with a future ( and much more complicated problem ) of a generic way to call the sockets. Which will get much more community support because the building blocks will exist.
Cuz whatever socket style you create for JS, now there is also a new socket style needed for PHP, and Python. So forcing the release to include a full socket solution means those also need solving. and for PHP, that would be complex to say the least.. because it's not event driven like JS is, so in order for me to send the responses into my database or whatever after CCXT processes it, I need to be along the lines of...

// $streamsUrl would be populated by the CCXT Binance socket url generator based on objects passed to it
\Amp\Loop::run( function () use ( $streamsUrl, $exchange ) {
    /** @var \Amp\Websocket\Connection $connection */
    $connection = yield Websocket\connect( $streamsUrl );
    while( $message = yield $connection->receive() ) {
        $payload = yield $message->buffer();
// this decode line would be replaced with the CCXT decoder for Binance
        $decode = json_decode( $payload );
// this switch would use the CCXT generic format for a socket packet
        switch( $decode->data->e ) {
            case "kline":
// methods do save my data
            break;
        }
        yield new Delayed( 1000 );
    }
} );

so if you solve the building blocks, then people can implement their systems to use it. and perhaps a generic socket solution that can pass packets to the developers system will emerge over time.

[kroitor]

@pursehouse to me the above just boils down to a proper and flexible method decomposition. However, I'd agree that doing async things with PHP can be tricky, might even require a threading solution or a PHP7-only-wrapper... But anyways, doing it in a language-specific or exchange-specific way would be even harder to maintain or transpile. I think our ultimate goal is to move as much as possible to the base class for unification purposes.

[pursehouse]

@kroitor right so don't try to solve the socket abilities at all right now, just focus on building blocks:

• url generation
• packet creation
• packet processing

cuz right now it seems a lot of time is going into how to do sockets in general, which isn't going to solve across all languages. And seems to be the most difficult aspect as far as making it a generic solution, and even if the generic sockets are solved for JS, then all those building blocks are still required to be able to function across all exchanges. So why not do building blocks first, and maybe include some example code on how to use them. Then afterwards a generic socket solution can be worked on and most likely have a larger scale of contributors.

[GoChartingAdmin]

@lfern Will do by tomorrow EOD

[GoChartingAdmin]

@lfern Was going through list - you have added Bistamp. Bitstamp currently uses Pusher. I thought Pusher was out of scope at this stage

[lfern]

@GoChartingAdmin , yes, you are right. I think we could add all exchanges with websocket and maybe add a column to point out that pusher protocol is not available. What do you think?

[GoChartingAdmin]

@lfern I agree. That will take out Bitstamp and Quionex

[backnight]

Thanks guys for working on WS! Do you need any help to make it happens faster? Also do you have an ETA when this will be ok? My company really needs this feature and we can allocate 1 or 2 developers if necessary.

[lfern]

@GoChartingAdmin Added "Ws Protocol" clumn in readme table.

[lfern]

@backnight in recent posts some changes has been proposed. But I think it would be better other people could write code for other exchanges so we could realize current ws implementation is really a good implementation. I don't know if anyone is trying to do this, or modifying this implemention in some way, but I am waiting for some feedback about it. If this implementation is accepted (with any changes proposed) I think that new exchanges could be added really fast.

[GoChartingAdmin]

@lfern Only FYI. I recently was able to integrate non websocket APIs (signalR or Pusher) in our app. We currently support WS for 21+ exchanges in our app GoCharting. See live production for a signalR client below

https://gocharting.com/terminal/IN/CRYPTO/ADT%7CBTC/BitTrex

I will chip in with some of those pieces when we have covered the normal websockets. I realized they are all mere wrappers around the native websocket implementation and hence can be implemented without any other third party libraries

[imhazige]

@lfern

a posiible bug? - socket not open error when invoke websocketSubscribe

at branch https://github.com/lfern/ccxt/tree/feature/websockets-multiple

this is like to happen in a slow network when connect to okex.

it is strange, that the socket is still not open when invoke websocketSubscribe.

Error Stack:

Error: not opened I20180905-18:02:28.011(8)? at WebSocket.send (node_modules/ws/lib/WebSocket.js:360:18) I20180905-18:02:28.011(8)? at WebsocketConnection.send (ccxt-websocket/js/base/websocket/websocket_connection.js:77:28) I20180905-18:02:28.012(8)? at WebsocketConnection.sendJson (cxt-websocket/js/base/websocket/websocket_base_connection.js:37:14) I20180905-18:02:28.012(8)? at okex.websocketSendJson (ccxt-websocket/js/base/Exchange.js:1866:31) I20180905-18:02:28.012(8)? at okex._websocketSubscribe (ccxt-websocket/js/okex.js:295:14) I20180905-18:02:28.012(8)? at Promise.asyncApply (ccxt-websocket/js/base/Exchange.js:2092:14)

this is not going to be easily reproduced.

[amit2103]

@kroitor Thanks for the awesome project. Is there any example showing how to get trades via websocket?
And which exchanges are supported till date

[AlexeyZelenin]

@amit2103

Is there any example showing how to get trades via websocket?

Not sure if it's possible at all via websocket, can't you use rest api?
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/examples/js/bitfinex-fetch-trades.js

And which exchanges are supported till date

https://github.com/lfern/ccxt/tree/feature/websockets-multiple#supported-cryptocurrency-exchange-markets

[djoffrey]

@lfern hi，could you please add an example of binance websocket fetch orderbook ?

[lfern]

Hi @djoffrey , in examples/js/websocket-orderbook.js you can find an example to fetch orderbook. Execute it with these parameters and you could fetch and get events from orderbook info:

node examples/js/websocket-orderbook.js binance 'no-apikey-required' 'no-key-required' 10 ETH/BTC

First parameter is the exchange id, second/third are api-key/secret (not required by binance), fourth is the number of bids/asks to get and fifth and following are the symbol to get.

[TechupBusiness]

Hey guys, and how about FIX API ? CoinbasePro supports it and it would open the doors to classical (stock/...) trading exchanges. http://en.wikipedia.org/wiki/Financial_Information_eXchange Dont know which other Crypto-Exchanges support this standard.

UPDATE: Found a good article... https://algosforcryptos.com/trading-apis-top-crypto-exchanges/

FIX API is available for:

• Bitfinex
• Bitstamp
• BTCC
• CEX.io
• CoinbasePrime
• CoinbasePro
• GEMINI
• HitBTC

Means one implementation of FIX API (which is also streaming like websocket as far as I understand) would cover instantly 8 (ok 7) crypto exchanges plus a lot of traditional exchanges.

There are already nodeJS implementations for the FIX API that ccxt could rely on with a unified access to FIX and Websocket streams. Just an idea... :)

[bramvbilsen]

I hate to be that guy, but any progress on this? Do not get me wrong, I think it is amazing that you are giving away such an amazing project for free, but the websocket implementation was supposingly almost done over a year ago.
Simply using a HTTP request in a loop as in some examples is not a good solution for my project.
So is there any ETA for this?
Once again, thanks for the great project :)

[TechupBusiness]

@bramvbilsen Maybe you need to start implementing it in ccxt. I guess @kroitor and all other developers would be happy if you can draft how you would like it to implement with ccxt (allowing callback function instead of returning a promise I guess). As soon as I get to the FixAPI (have also lots of other tasks on my list) I will probably start on this, as I really appreciate the work of the ccxt community (especially @kroitor these days). I want to give something back to the community. But like the "one year ago" issue, it will also take some time. ;-)

[bramvbilsen]

@TechupBusiness Do not get me wrong, I would not mind doing it. But the reason I was looking at ccxt is because I do not have a lot of time at all unfortunately.

[TechupBusiness]

@bramvbilsen Thats probably the reason why most of us landed here: Saving time as its the most scarce resource. :)

PS I would also love to see a kind of ccxt for blockchain/DLT interactions one day.

[bramvbilsen]

@TechupBusiness Exactly!

[soliveira-vouga]

If your project needs this and the exchange(s) have websocket support, then why not write your own wrapper. It's a lot easier than writing something that fits within a unified abstraction layer which the folks at ccxt are trying to do. There are plenty of implementations already for popular exchanges like binance, bitfinex, bitstamp, etc. There's no reason why you should be looping over HTTP requests if you have a websocket api available to you

[bramvbilsen]

@soliveira-vouga I am not doing that, I am currently using the websocket implementations of different exchanges. But I thought I saw that that is what the examples in this repo do to get "live" data.

[adrianog]

Can the feature/websockets-multiple branch be used from python?

[lfern]

Exchanges with ws support (js/py/php) in feature/websockets-multiple branch
@quazzuk, @tommuhm, @yevsev, @imhazige, @beijingkaka, @firepol, thanks for your help!!

| exchange | ws type | events |
|---------------|-----------|-------------------------------|
|binance | ws | orderbook ticker trade ohlcv |
|bitmex | ws | orderbook |
|bitstamp | pusher | orderbook |
|cex | ws (auth) | orderbook |
|coincheck | ws | orderbook |
|coinbaseprime | ws | orderbook |
|coinbasepro | ws | orderbook |
|gateio | ws | orderbook |
|gemini | ws | orderbook |
|hadax | ws | orderbook |
|hitbtc2 | ws | orderbook |
|huobipro | ws | orderbook |
|lbank | ws | orderbook |
|liquid | pusher | orderbook |
|okex | ws | orderbook |
|poloniex | ws | orderbook |
|zb | ws | orderbook |

[lfern]

Does anyone know about another url (https://algosforcryptos.com/trading-apis-top-crypto-exchanges/) to get more exchanges that support websocket or pusher connections?

[rg687]

@lfern I am trying to figure out how the python part works. Where is the config for websocket url for each exchange? and the events available?

ask exception was never retrieved
future: exception=ExchangeError('Not valid event orderbook for exchange binance',)>
Traceback (most recent call last):
File "websocket-playground.py", line 59, in main
await exchange.websocket_subscribe(event, symbol, params)
File "/home/user/crypto/ccxt/python/ccxt/async_support/base/exchange.py", line 773, in websocket_subscribe
raise ExchangeError('Not valid event ' + event + ' for exchange ' + self.id)
ccxt.base.errors.ExchangeError: Not valid event orderbook for exchange binance

thanks

[lfern]

@rg687 I am not commiting PHP and Python generated files from transpilation process yet. You must execute npm run transpile in base ccxt folder to get them. You will find a `wsconf' configuration structure with valid events ('ob' for orderbook)

[lfern]

@rg687 maybe you must execute npm run export-exchanges before transpilation to ensure exchange.json is generated.

@rg687 I am not commiting PHP and Python generated files from transpilation process yet. You must execute npm run transpile in base ccxt folder to get them. You will find a `wsconf' configuration structure with valid events ('ob' for orderbook)

[firepol]

@lfern I just tested with python and it worked nice with a few exchanges I tested. Good job man!

I see many major exchanges have been implemented.

Ideas when bitfinex will be implemented?

[lfern]

added websocket orderbook for therock, theocean, bitfinex2

[AlexeyZelenin]

@lfern May I ask when do you plan to merge 'websockets-multiple' to the master?

[lfern]

@AlexeyZelenin I don't really know if this development is mature enough to make a push request to master. If ccxt owners think so, I would add a push request to master. But I think they have to check if code developed and dependencies added could be merged on master branch.

[firepol]

@lfern great job about bitfinex2, I just tested it and it works great.
I also tested cex (authenticated WS).

May I report a few issues I've found? I've tested a few websocket examples in examples/py:

(btw I think it may be a good idea to put a few examples with usage (feel free to copy/adapt the following) in the WEBSOCKETS.md file, noob friendly.) Anyway here my findings:

• websocket-poloniex-orderbook.py, (at line 11 replace import ccxt.async with import ccxt.async_support) usage ./websocket-poloniex-orderbook.py works
• websocket-playground.py, usage ./websocket-playground.py bitstamp ob BTC/EUR works like a charm
• websocket-orderbook.py, (at line 11 replace import ccxt.async with import ccxt.async_support), usage:
  
  
  
  • one symbol: ./websocket-orderbook.py bitstamp None None 5 BTC/EUR
  
  
  • multiple symbols ./websocket-orderbook.py bitstamp None None 5 BTC/EUR XRP/EUR
  
  

After around 5 few seconds I get this error:

Traceback (most recent call last):
  File "/opt/projects-python/ccxtws/examples/websocket-orderbook.py", line 81, in <module>
    loop.run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 473, in run_until_complete
    return future.result()
  File "/opt/projects-python/ccxtws/examples/websocket-orderbook.py", line 74, in main
    await exchange.websocket_unsubscribe('ob', symbol)
  File "/opt/projects-javascript/ccxt/python/ccxt/async_support/base/exchange.py", line 825, in websocket_unsubscribe
    self._websocket_unsubscribe(conxid, event, symbol, oid, params)
  File "/opt/projects-javascript/ccxt/python/ccxt/async_support/bitstamp.py", line 941, in _websocket_unsubscribe
    self.websocketSendJson(payload)
  File "/opt/projects-javascript/ccxt/python/ccxt/async_support/base/exchange.py", line 669, in websocketSendJson
    websocket_conx_info['conx'].sendJson(data)
  File "/opt/projects-javascript/ccxt/python/ccxt/async_support/websocket/websocket_base_connection.py", line 31, in sendJson
    self.send(json.dumps(data))
  File "/opt/projects-javascript/ccxt/python/ccxt/async_support/websocket/pusher_light_connection.py", line 172, in send
    if json_data['event'] == 'subscribe':
KeyError: 'event'

./websocket-orderbook.py binance None None 5 NANO/BTC BTC/USDT: seems to work well, but after around 30s it fails with KeyError:'limit'.

I think the lines await asyncio.sleep(5) are responsible for this. If I increase e.g. to 20, it will work 20 seconds. Is it possible to modify the example so that it works forever?

Also I was expecting multiple websockets to work simultaneously, which is not really the case.

Please fix all the ccxt.async with ccxt.async_support in the examples if there are other leftovers.

Thanks again for the great work. I was really looking forward to have a good library supporting many exchanges and this does the job.

[lfern]

Thanks! @firepol, your traceback helps me to find two bugs:

• one symbol: ./websocket-orderbook.py bitstamp None None 5 BTC/EUR
• multiple symbols ./websocket-orderbook.py bitstamp None None 5 BTC/EUR XRP/EUR

After around 5 few seconds I get this error:

Traceback (most recent call last):
  File "/opt/projects-python/ccxtws/examples/websocket-orderbook.py", line 81, in <module>
    loop.run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 473, in run_until_complete
    return future.result()
  File "/opt/projects-python/ccxtws/examples/websocket-orderbook.py", line 74, in main
    await exchange.websocket_unsubscribe('ob', symbol)
  File "/opt/projects-javascript/ccxt/python/ccxt/async_support/base/exchange.py", line 825, in websocket_unsubscribe
    self._websocket_unsubscribe(conxid, event, symbol, oid, params)
  File "/opt/projects-javascript/ccxt/python/ccxt/async_support/bitstamp.py", line 941, in _websocket_unsubscribe
    self.websocketSendJson(payload)
  File "/opt/projects-javascript/ccxt/python/ccxt/async_support/base/exchange.py", line 669, in websocketSendJson
    websocket_conx_info['conx'].sendJson(data)
  File "/opt/projects-javascript/ccxt/python/ccxt/async_support/websocket/websocket_base_connection.py", line 31, in sendJson
    self.send(json.dumps(data))
  File "/opt/projects-javascript/ccxt/python/ccxt/async_support/websocket/pusher_light_connection.py", line 172, in send
    if json_data['event'] == 'subscribe':
KeyError: 'event'

This is a caused by an unsubscription timeout error. Pusher subscription returns an ack eventm but unsubscription doesn't return any event and binance code was waiting for it.
./websocket-orderbook.py binance None None 5 NANO/BTC BTC/USDT: seems to work well, but after around 30s it fails with KeyError:'limit'.

if you want to subscribe to binance orderbook updates on a second market symbol, you need to reconnect to the websocket with additional url parameters. When websocket is reconnected its context variables are reset, so "limit" parameter saved on context are lost and causes this "keyerror:limit" error. In this type of exchanges (streams) we have to store this config parameters in context persistent variables.

Thanks again for your help!

[firepol]

Thanks to you @lfern for the quick fixes, I appreciate.

I noticed another thing, testing multiple websockets on the same exchange, e.g:

./websocket-orderbook.py bitfinex2 None None 5 BTC/USDT XRP/USDT

I start seeing BTC/USDT updates immediately, but before seeing anything about XRP/USDT I have to wait around 5 seconds.

I think this is caused by the await asyncio.sleep(5) is this really needed?

If I try with 3 pairs:

./websocket-orderbook.py bitfinex2 None None 5 BTC/USDT XRP/USDT

Same: BTC/USDT shows up immediately, after 5 seconds I see XRP/EUR updates, then I have to wait another 5 seconds to see ETH/EUR updates. Like this, imagine you want to monitor 11 pairs, you'll see updates of the last pair only after 50 seconds. Is this a sort of rate limiting for websockets?

Also, in the websocker-orderbook example, it seems that the websocket is unbsubscrbed and re-subscribed each 5 seconds. Something doesn't look correct in this example.

I tried to comment the first await asyncio.sleep(5), but then it seems to only subscribe the first pair.
I tried to put a lower value, 1 second: await asyncio.sleep(1) so the delay now is very low.
I did put a high value for the second sleep: await asyncio.sleep(2000000) like this it's unsubscribed only later. Has this an implication? What happens if the websocket gets disconnected earlier? 5 seconds was a way to make it resubscribe in such an event? Is there an event fired if a subscription to a websocket doesn't work anymore? Would be nice to have an example to handle a websocket disconnection properly, e.g. try to re-subscribe after 2 minutes when the websocket connection is lost.

Thanks for your feedback

[firepol]

Hi there, after some more testing I actually developer my little own project and open sourced it here: https://github.com/firepol/ccxt-websockets-db-updater

@lfern I have trouble connecting to more than one pair at binance, maybe you can help to fix that? From what I've seen in my other tool I coded (also based on another library, so I'm totally no expert at all in websockets), binance and GDAX need a "product registration" before subscribing to each order book. I don't know what that means, but that may be why binance doesn't work if I try to connect in parallel as I did in my program.

See: https://github.com/firepol/ccxt-websockets-db-updater/issues/1

[lfern]

added live orderbook market table example:
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/examples/js/websocket-orderbook.js
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/examples/py/websocket-orderbook.py

[lfern]

@firepol, binance uses streams in websocket connection. You have to open websocket with all symbols you want to observe: wss://stream.binance.com:9443/stream?streams=//

First time you try to subscribe websocket library open websocket with only one stream:wss://stream.binance.com:9443/stream?streams=NANOBTC@depth

while waiting for first subscription succeeds, if you try to subscribe to second symbol, binance ws driver need to close previous connection and open websocket with a new URL: wss://stream.binance.com:9443/stream?streams=NANOBTC@depth/ETHBTC@depth, and the first websocket open action is closed before it succeeds.

I don't know how to handle it correctly, maybe before closing binance driver would have to wait to first subscription succeeds. but close function is not async, and we need to be sure previous ws connection is closed before connecting with new URL. For the time being, you must not subscribe in paralell on same exchange

Hi there, after some more testing I actually developer my little own project and open sourced it here: https://github.com/firepol/ccxt-websockets-db-updater

@lfern I have trouble connecting to more than one pair at binance, maybe you can help to fix that? From what I've seen in my other tool I coded (also based on another library, so I'm totally no expert at all in websockets), binance and GDAX need a "product registration" before subscribing to each order book. I don't know what that means, but that may be why binance doesn't work if I try to connect in parallel as I did in my program.

See: firepol/ccxt-websockets-db-updater#1

[firepol]

UPDATE: thx a lot @lfern for fixing my little project, I appreciate!

TL;DR: @lfern helped me to make it work to subscribe multiple websockets from the same exchange, in parallel. Have a look at my script: https://github.com/firepol/ccxt-websockets-db-updater

@lfern thxfor your reply, btw I noticed a small thing with bitfinex2, the timestamp is missing.

For binance to allow multiple pairs, maybe you can get some inspiration by the xchange-stream library, even if it's java, maybe some ideas can be useful for CCXT too? in particular check ProductSubscribtion. See e.g. how I use it (I'm no expert, somebody helped me to make it work) here:

https://github.com/firepol/crypto-websockets/blob/dev/src/main/java/com/github/firepol/cryptows/ExchangeManager.java

See in processWebsockets:

        pairsByExchange.forEach((exchangeName, pairsCollection)->{
            if (Arrays.asList(NEED_PRODUCT_REGISTRATION).contains(exchangeName)) {
                pairsCollection.pairs.forEach(pair->{
                    StreamingExchange exchange = getStreamingExchange(exchangeName);
                    ProductSubscription productSubscription = ProductSubscription.create()
                        .addOrderbook(pair)
                        .build();
                    exchange.connect(productSubscription).blockingAwait();
                    subscribeOrderBook(exchange, pair);
                });
            } else {
                StreamingExchange exchange = getStreamingExchange(exchangeName);
                exchange.connect().blockingAwait();
                pairsCollection.pairs.forEach(pair->subscribeOrderBook(exchange, pair));
            }
        });

Basically the first if checks if the exchange name is in the NEED_PRODUCT_REGISTRATION array (which contains gdax, binance, maybe other exchanges will show up) and if so connects to the websocket passing the product subscription. How this is handled I don't know exactly, but let's have a look at the source code here:

Here the ProductSubscription class

Here how it is used e.g. in BinanceStreamingExchange.java

    private BinanceStreamingService createStreamingService(ProductSubscription subscription) {
        String path = API_BASE_URI + "stream?streams=" + buildSubscriptionStreams(subscription);
        return new BinanceStreamingService(path, subscription);
    }

So far so good as you already mentioned, how this is handled in xchange-stream I'm not sure, I see that the message is handled in the super class, maybe you can have a look and have better luck in understanding it?

[firepol]

Hi @lfern during more testing (trying to keep many connections alive for a longer period) I found more issues:

• bitstamp, ws for btc/usd seems to subscribe, but no values are returned. See bitstamp websocket documentation, for any other pair than btc/usd, the channel name is order_book_{currency_pair}, but for btc/usd it's not order_book_btcusd, but simply order_book.
• bitstamp: datetime is always: 1970-01-18T20:58:45.806Z
• bitfinex: datetime & timestamp are both null
• bitfinex: volume values for asks are returned with negative values, e.g. -240, expected: 240
• cex seems to be quite unstable: I tested many connections and cex is the one that disconnect the most. Cex websockets need to be authenticated. If you want to test this and don't have an account, you can register at cex.io I know it may not be an issue of your implementation, but it would be nice to have an example showing how to reconnect. I'm working on it on python and could not manage yet to find a solution for this.

In general, double check timestamp & datetime, so far really good work!
Thanks again and cheers

[lfern]

Thanks @firepol for feedback.

• bitstamp datetime and btc/usd subscription problems are fixed
• I can't find timestamp info in bitfinex orderbook messages. So I prefer not to fill these values with local time.
• bitfinex negative values is fixed too.
• I will add a new example with reconnection example. Maybe we have to change error reporting with more detailed error messages.

[firepol]

Hi @lfern ,

I checked your last commit but the commit message is misleading as it states "bitfinex", but the file changed it bitstamp.js. I think you forgot to include bitfinex changes or confused bitstamp with bitfinex changes you were about to commit next?

About handling disconnections, I tried to attempt something in my project (which btw saves data in a database already, I use Postgres, but you can test also with SQLite). I created a custom exception (WsError), which I raise in: https://github.com/firepol/ccxt-websockets-db-updater/blob/master/utils.py:

    @exchange.on('err')
    async def websocket_error(err, conxid):  # pylint: disable=W0612
        error_message = type(err).__name__ + ":" + str(err)
        error_stack = traceback.extract_stack()
        logging.error(f'{exchange.id}: {error_message}')
        logging.error(error_stack)
        print(f'{exchange.id}, {datetime.datetime.now()}, {error_stack}')
        await exchange.close()

And catch it here: https://github.com/firepol/ccxt-websockets-db-updater/blob/master/ob_updater.py

    except WsError as wse:
        print(f'Canceling: {wse}')
        ob_subscriptions[wse].cancel()

I'm sure I'm doing something wrong, I'm still a noob in async programming in python.

Ideally I'd like in the ob_updater.py program to catch the exception per exchange (that's why I added the tasks in a dictionary, as follows:

            # make a list of tasks by exchange id
            ob_subscriptions[exchange.id] = asyncio.ensure_future(utils.subscribe_ws('ob', exchange, symbols, limit,
                                                     pp, args.debug, args.verbose, session))

The idea was then to properly cancel the task (which I think is not being canceled correctly now as it complains it had to be awaited) and then create a new one of the same exchange. That would be the best.

If you can include that in your repo examples then I'll try to learn from there and apply to my separate project. Cheers

[lfern]

added new examples for recovering from websocket error.

• websocket-recover-connection.js
• websocket-recover-connection.py

@firepol, what do you think about recovering process? Maybe it could be easier.

[firepol]

Hi @lfern I quickly tried your new example in python: /websocket-recover-connection.py (exchange: cex, I passed my api key/secret, pairs: just 'BTC/USD'

subscribe: BTC/USD
KeyError:'ok'
  File "/home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/base/exchange.py", line 711, in websocket_connection_message
    self._websocket_on_message(conxid, msg)
  File "/home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/cex.py", line 563, in _websocket_on_message
    getattr(self, method)(contextId, msg, oid, resData)
unsubscribing all ...
  File "/home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/cex.py", line 570, in _websocket_handle_auth
unsubscribe: BTC/USD
    if msg['ok'] == 'ok':
ExchangeError:not recoverable error
unsubscribing all ...
unsubscribe: BTC/USD

As the error says, it seems that there is no 'ok' key in the msg. Did you try this in python, with cex?

[maayank]

Awesome work & branch @lfern, many thanks.
Regarding 'ob', is it possible to get only the first snapshot and then the updates as they come, not everything (e.g. with an option)? I see in the code (at least, when mid-way printing the received messages from the exchange) that the data patching is done by the ccxt code.
Many thanks

[lfern]

Sorry @firepol, I haven't tested with CEX, I need to complete CEX verify process to generate new api key. Tested with bitfinex2, gemini and bitstamp

[lfern]

Thanks @maayank, I think we need to keep a copy of orderbook inside ccxt, so you can get a live orderbook copy when invoking websocketFetchOrdebook method. It could be posible to add an adidtional attribute to 'ob' event object with the last changes received (like 'info' ccxt attribute with original response received from exchange in CCXT REST methods). But some exchanges always sends a snapshot (e.g. bitstamp). In these cases ccxt only replaces internal orderbook copy with new one.

It is not difficult to add this new parameter 'info' with last asks/bids received (price/amount), or we could define a new event 'rawob' that send only snapshot and updates, But before adding this I would like to know what other people think about it.

Regarding 'ob', is it possible to get only the first snapshot and then the updates as they come, not everything (e.g. with an option)? I see in the code (at least, when mid-way printing the received messages from the exchange) that the data patching is done by the ccxt code.
Many thanks

[mishaker]

@lfern is your script here (https://github.com/lfern/ccxt/blob/feature/websockets-multiple/examples/py/websocket-orderbook.py) can be executed standalone ?

[lfern]

added cobinhood websocket support: orderbook/trade/ticker/ohlcv

[lfern]

@mishaker Yes, you can execute this example to print simple live orderbook bids/asks table. It is a console script, and I have only tested in windows/cygwin, but I think it would have to work on linux console.

@lfern is your script here (https://github.com/lfern/ccxt/blob/feature/websockets-multiple/examples/py/websocket-orderbook.py) can be executed standalone ?

[mishaker]

@lfern maybe a noob question but I have this error while trying to execute your script:

Task exception was never retrieved
future: <Task finished coro=<subscribe() done, defined at example.py:67> exception=AttributeError("'binance' object has no attribute 'on'",) created at example.py:64>
source_traceback: Object created at (most recent call last):
  File "example.py", line 194, in <module>
    main()
  File "example.py", line 64, in main
    asyncio.ensure_future(subscribe (ex, exchange['symbols'], config['symbolDefaults']), loop=loop)
Traceback (most recent call last):
  File "example.py", line 69, in subscribe
    @exchange.on('err')
AttributeError: 'binance' object has no attribute 'on'

Maybe I should include something before running the script ?

[lfern]

Sorry @mishaker. I am not commiting transpiled python code. So you must execute npm run trasnspile before execute any websocket python script.

[lfern]

added upbit websocket support: orderbook/trade/ticker

[gl-man]

Hello, How can I get orders by time for binance exchange house?
How can I get orders with timestamp?
Is there a ccxt method?

[firepol]

@lfern some news from my side: I updated my repo and now it works quite well. I left it running an entire night on a VM with 2 CPUs and 2 GB ram and I had no (apparent) websocket disconnection ... the hard disk was spinning quite a bit I must say, so I'll consider a setup where the database is in memoy, SQLlite has such a feature.

Sharing a little issue: I noticed some conflicts if binance and cexio run at the same time. Basically no issue running many exchanges and binance. But if I run just cexio + binance, binance gets some errors. I had to disable error catching because of that. Also not catching those errors has no effect and everything is updated, so I don't know why do I get such error events. maybe you can have a look if you can reproduce it from your side as well? I created an issue in my repo: https://github.com/firepol/ccxt-websockets-db-updater/issues/4 I will add the stack error, but if you try my repo using the --verbose and --debug options you will notice it immediately.

To change topic: I was wondering if you could also add websockets support for bittrex? I'll be happy to do further testing and more exchanges there are, the better.

Awesome job man, I love the websockets feature, looking forward that it's merged in master!

Btw since you do not commit python transpiled code in the repo, I created a repo that takes only the (transpiled) python files, for easier use in my repo: https://github.com/firepol/ccxt-websockets
(maybe a side project would be to automate this, so on each update from your side, a Travis pipeline will transpile the code and update my repo without me doing it manually...).

[maayank]

@lfern , regarding bitfinex and timestamps, they do send timestamps if you send a "conf" event on subscription with the appropriate flag (grep for "Timestamp in milliseconds." in
https://docs.bitfinex.com/v2/docs/ws-general
).
(also @firepol , as I see he raised it)
I can write a patch and send it to your branch, or if it's easier for you to change just FYI.

[lfern]

@maayank , added orderbook timestamp handling right now.

@firepol , sorry!!!, I am trying to get some time this week to check your issue and add bitrex support

[lfern]

@firepol I have just added websocket support to bittrex. This exchange uses signalr and it is using our internal websocket implementation (maybe it would be necesary to create a new websocket class for signalr later). I have been testing it this afternoon and it is working right.

I have found a PHP transpilation problem:
This javascript code

'connectionData': '[{"name":"c2"}]',

is transpiled to PHP like this

'connectionData' => [array ("name":"c2")]'

I will merge ccxt/master to this branch to check if it fixed up.

[firepol]

Hi @lfern thx for implementing bittrex! I tested it and it works!

Maybe the next big exchange could be kraken? I've read that they recently added websocket support: https://blog.kraken.com/post/2019/websockets-public-api-launching-soon/

It will be officially launched in a few days, but it's possible to test it already using the sandbox.

[lfern]

websocket support on kraken. Right now only sandbox websocket url is active. To test it now, you have to change baseurl configuration to sandboxurl.

[hfsam]

可以告诉我的货币总算有多少吗？谢谢！

[Fcl69]

Hey guys! First of all, @lfern @firepol - thank you for pushing forward this big initiative as WS, I think the whole ccxt community is very excited about it.

I was trying to dive into it. And I've got the following results. First, I've started with this script: https://github.com/lfern/ccxt/blob/feature/websockets-multiple/examples/py/websocket-orderbook.py
and it just exited with no output:

v:py fcl$ python3 websocket-orderbook.py
v:py fcl$

Then I've proceeded with this repo - https://github.com/firepol/crypto-websockets
set up APIs, settings, data folder, etc. So it runs, but no output neither, no activity, and DB is empty:

v:ccxt-websockets-db-updater fcl$ ./ob_tester.py -e binance -s ETH/BTC --debug
^CClosing Loop
After complete
v:ccxt-websockets-db-updater fcl$ cat cryptows.db
???N!!?gtableorder_bookorder_bookCREATE TABLE order_book (
id INTEGER NOT NULL,
exchange_name VARCHAR(20) NOT NULL,
base VARCHAR(10) NOT NULL,
quote VARCHAR(10) NOT NULL,
side VARCHAR(3) NOT NULL,
price DECIMAL NOT NULL,
volume DECIMAL NOT NULL,
sort INTEGER NOT NULL,
datetime DATETIME NOT NULL,
PRIMARY KEY (id)

I'd appreciate if you could point out for me the direction to debug it properly.

[lfern]

Hi @Fcl69 , in websocket-multiple branch python modules aren't transpiled from node code. If you try this example:

 python3 examples/py/websocket-playground.py kraken ob ETH/XTC limit:1

and you get something like this:

Task exception was never retrieved
future: <Task finished coro=<main() done, defined at examples/py/websocket-playground.py:19> exception=ExchangeError('Not valid event ob for exchange kraken',)>
Traceback (most recent call last):
  File "examples/py/websocket-playground.py", line 58, in main
    await exchange.websocket_subscribe(event, symbol, params)
  File "c:\users\luis\documents\projects\ccxt\python\ccxt\async_support\base\exchange.py", line 826, in websocket_subscribe
    raise ExchangeError('Not valid event ' + event + ' for exchange ' + self.id)
ccxt.base.errors.ExchangeError: Not valid event ob for exchange kraken
kraken requires to release all resources with an explicit call to the .close() coroutine. If you are creating the exchange instance from within your async coroutine, add exchange.close() to your code into a place when you're done with the exchange and don't need the exchange instance anymore (at the end of your async coroutine).
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x0000022292D4C860>

then you have to transpile code with:

npm run transpile

and then you get all python code ready to test.

websocket-orderbook.py script clean the console screen to update orderbook data as it is received from exchanges, maybe this cleaning have removed the printed stacktrace error.

[Fcl69]

it says:
v:ccxt fcl$ npm run transpile

[email protected] transpile /Users/v.golban/Documents/_personal/_work/ccwd/bin/ccwd/ccxt_lfern/ccxt
node transpile

internal/modules/cjs/loader.js:611
throw err;
^

Error: Cannot find module 'ololog'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:609:15)
at Function.Module._load (internal/modules/cjs/loader.js:535:25)
at Module.require (internal/modules/cjs/loader.js:663:17)
at require (internal/modules/cjs/helpers.js:20:18)
at Object. (/Users/v.golban/Documents/_personal/_work/ccwd/bin/ccwd/ccxt_lfern/ccxt/transpile.js:5:14)
at Module._compile (internal/modules/cjs/loader.js:734:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:745:10)
at Module.load (internal/modules/cjs/loader.js:626:32)
at tryModuleLoad (internal/modules/cjs/loader.js:566:12)
at Function.Module._load (internal/modules/cjs/loader.js:558:3)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] transpile: node transpile
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] transpile script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?

npm ERR! A complete log of this run can be found in:
npm ERR! .npm/_logs/2019-01-31T22_15_59_360Z-debug.log

and then:

v:ccxt fcl$ cat .npm/_logs/2019-01-31T21_11_25_875Z-debug.log
0 info it worked if it ends with ok
1 verbose cli [ '/usr/local/Cellar/node/11.9.0/bin/node',
1 verbose cli '/usr/local/bin/npm',
1 verbose cli 'run',
1 verbose cli 'transpile' ]
2 info using [email protected]
3 info using [email protected]
4 verbose run-script [ 'pretranspile', 'transpile', 'posttranspile' ]
5 info lifecycle [email protected]~pretranspile: [email protected]
6 info lifecycle [email protected]~transpile: [email protected]
7 verbose lifecycle [email protected]~transpile: unsafe-perm in lifecycle true
8 verbose lifecycle [email protected]~transpile: PATH: /usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/Users/Documents/_personal/_work/ccwd/bin/ccwd/ccxt_lfern/ccxt/node_modules/.bin:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools
9 verbose lifecycle [email protected]~transpile: CWD: ccxt_lfern/ccxt
10 silly lifecycle [email protected]~transpile: Args: [ '-c', 'node transpile' ]
11 silly lifecycle [email protected]~transpile: Returned: code: 1 signal: null
12 info lifecycle [email protected]~transpile: Failed to exec transpile script
13 verbose stack Error: [email protected] transpile: node transpile
13 verbose stack Exit status 1
13 verbose stack at EventEmitter. (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:301:16)
13 verbose stack at EventEmitter.emit (events.js:197:13)
13 verbose stack at ChildProcess. (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack at ChildProcess.emit (events.js:197:13)
13 verbose stack at maybeClose (internal/child_process.js:978:16)
13 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:265:5)
14 verbose pkgid [email protected]
15 verbose cwd ccxt_lfern/ccxt
16 verbose Darwin 16.7.0
17 verbose argv "/usr/local/Cellar/node/11.9.0/bin/node" "/usr/local/bin/npm" "run" "transpile"
18 verbose node v11.9.0
19 verbose npm v6.5.0
20 error code ELIFECYCLE
21 error errno 1
22 error [email protected] transpile: node transpile
22 error Exit status 1
23 error Failed at the [email protected] transpile script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

[lfern]

Sorry, I think you must to execute npm install first to install node modules.

[Fcl69]

my bad
but then:
v:ccxt fcl$ npm run transpile

[email protected] transpile /Users/Documents/_personal/_work/ccwd/bin/ccwd/ccxt_lfern/ccxt
node transpile

internal/modules/cjs/loader.js:611
throw err;
^

Error: Cannot find module './exchanges.json'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:609:15)
at Function.Module._load (internal/modules/cjs/loader.js:535:25)
at Module.require (internal/modules/cjs/loader.js:663:17)
at require (internal/modules/cjs/helpers.js:20:18)
at transpileDerivedExchangeFiles (/Users/Documents/_personal/_work/ccwd/bin/ccwd/ccxt_lfern/ccxt/transpile.js:717:17)
at Object. (/Users/Documents/_personal/_work/ccwd/bin/ccwd/ccxt_lfern/ccxt/transpile.js:958:17)
at Module._compile (internal/modules/cjs/loader.js:734:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:745:10)
at Module.load (internal/modules/cjs/loader.js:626:32)
at tryModuleLoad (internal/modules/cjs/loader.js:566:12)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] transpile: node transpile
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] transpile script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! .npm/_logs/2019-01-31T23_08_18_442Z-debug.log

[lfern]

Sorry again! execute
npm run export-exchanges
If it doesn't work execute full build:
npmrun build

[Fcl69]

hm... I'm sorry to poking you, but it doesn't help. building fails at

v:ccxt fcl$ npm run build
.....
Transpiling from ./js/test/base/functions/test.datetime.js (transpileDateTimeTests @ transpile.js:859)

→ ./python/test/test_exchange_datetime_functions.py (transpileDateTimeTests @ transpile.js:883)
→ ./php/test/test_exchange_datetime_functions.php (transpileDateTimeTests @ transpile.js:884)
Transpiling ./python/test/test_async.py → ./python/test/test.py (transpilePythonAsyncToSync @ transpile.js:778)
Transpiled successfully. ( @ transpile.js:975)

[email protected] qa /Users/Documents/_personal/_work/ccwd/bin/ccwd/ccxt_lfern/ccxt
npm run check-python-syntax && npm run check-php-syntax

[email protected] check-python-syntax /Users/Documents/_personal/_work/ccwd/bin/ccwd/ccxt_lfern/ccxt
cd python && tox -e qa && cd ..

sh: tox: command not found
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR! [email protected] check-python-syntax: cd python && tox -e qa && cd ..
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the [email protected] check-python-syntax script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /Users/.npm/_logs/2019-02-01T02_38_44_448Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] qa: npm run check-python-syntax && npm run check-php-syntax
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] qa script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /Users/.npm/_logs/2019-02-01T02_38_44_481Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build-without-docs: npm run export-exchanges && npm run vss && npm run check-js-syntax && npm run browserify && npm run transpile && npm run qa
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build-without-docs script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /Users/.npm/_logs/2019-02-01T02_38_44_510Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build: npm run build-without-docs && npm run pandoc-all && npm run update-badges
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!

and it brings me to the situation you've mentioned:

v:ccxt fcl$ python3 examples/py/websocket-playground.py kraken ob ETH/XTC limit:1
Task exception was never retrieved
future: exception=AttributeError("'kraken' object has no attribute 'on'",)>
Traceback (most recent call last):
File "examples/py/websocket-playground.py", line 43, in main
@exchange.on('err')
AttributeError: 'kraken' object has no attribute 'on'
kraken requires to release all resources with an explicit call to the .close() coroutine. If you are creating the exchange instance from within your async coroutine, add exchange.close() to your code into a place when you're done with the exchange and don't need the exchange instance anymore (at the end of your async coroutine).
Unclosed client session
client_session:

[lfern]

@Fcl69 please check if exchanges.json file has been generated in the ccxt folder. If so, try again

npm run transpile

[lfern]

These are current exchanges that supports orderbook websocket connection:

• binance
• bitfinex2
• bitmex
• bitstamp
• bittrex
• cex
• cobinhood
• coincheck
• coinbaseprime
• coinbasepro
• gateio
• gemini
• hadax
• hitbtc2
• huobipro
• kraken
• lbank
• liquid
• okex
• poloniex
• theocean
• therock
• upbit
• zb

Please, does anybody knows whose exchanges might be the next?

[Helmi]

It's totally awesome to see what you guys achieve here - thank you for that.

How do you all (especially @lfern and @kroitor) think about going bigger with that project? It's a bit exhausting to see this all handled in one issue thread. Maybe a wiki page with what's working already and what isn't and maybe a rough howto or something could be helpful.

Again thanks for your hard work!

[Fcl69]

@lfern

exchanges.json is there:

v:ccxt fcl$ ls exchanges.json
exchanges.json

"npm run transpile" goes well:

v:ccxt fcl$ npm run transpile

[email protected] transpile /Users/Documents/_personal/_work/ccwd/bin/ccwd/ccxt_lfern/ccxt
node transpile

Transpiling from _1btcxe.js (transpileDerivedExchangeFile @ transpile.js:696)
Transpiling from acx.js (transpileDerivedExchangeFile @ transpile.js:696)
Transpiling from allcoin.js (transpileDerivedExchangeFile @ transpile.js:696)
....

Transpiling from zaif.js (transpileDerivedExchangeFile @ transpile.js:696)
Transpiling from zb.js (transpileDerivedExchangeFile @ transpile.js:696)
Transpiling from ./js/test/base/functions/test.number.js (transpilePrecisionTests @ transpile.js:896)
→ ./python/test/test_decimal_to_precision.py (transpilePrecisionTests @ transpile.js:945)
→ ./php/test/decimal_to_precision.php (transpilePrecisionTests @ transpile.js:946)
Transpiling from ./js/test/base/functions/test.datetime.js (transpileDateTimeTests @ transpile.js:859)
→ ./python/test/test_exchange_datetime_functions.py (transpileDateTimeTests @ transpile.js:883)
→ ./php/test/test_exchange_datetime_functions.php (transpileDateTimeTests @ transpile.js:884)
Transpiling ./python/test/test_async.py → ./python/test/test.py (transpilePythonAsyncToSync @ transpile.js:778)
Transpiled successfully. ( @ transpile.js:975)

but then it doesn't work anyways:

v:ccxt fcl$ python3 examples/py/websocket-playground.py kraken ob ETH/XTC limit:1
Task exception was never retrieved
future: exception=AttributeError("'kraken' object has no attribute 'on'",)>
Traceback (most recent call last):
File "examples/py/websocket-playground.py", line 43, in main
@exchange.on('err')
AttributeError: 'kraken' object has no attribute 'on'
kraken requires to release all resources with an explicit call to the .close() coroutine. If you are creating the exchange instance from within your async coroutine, add exchange.close() to your code into a place when you're done with the exchange and don't need the exchange instance anymore (at the end of your async coroutine).
Unclosed client session
client_session:
^CClosing Loop
after complete
v:ccxt fcl$

[lfern]

OK, @Fcl69 I think now you need to install python dependencies:

pip3 install -e python

[Fcl69]

I did, plus transpile again, but same stuff:

v:ccxt fcl$ python3 examples/py/websocket-playground.py kraken ob ETH/XTC limit:1
Task exception was never retrieved
future: exception=AttributeError("'kraken' object has no attribute 'on'",)>
Traceback (most recent call last):
File "examples/py/websocket-playground.py", line 43, in main
@exchange.on('err')
AttributeError: 'kraken' object has no attribute 'on'
kraken requires to release all resources with an explicit call to the .close() coroutine. If you are creating the exchange instance from within your async coroutine, add exchange.close() to your code into a place when you're done with the exchange and don't need the exchange instance anymore (at the end of your async coroutine).
Unclosed client session
client_session:

=(

[lfern]

Sorry @Fcl69 . I have downloaded this branch on a clean linux distro and with these steps I have configured my development enviroment:

git clone https://github.com/lfern/ccxt.git
cd ccxt/
git checkout -t origin/feature/websockets-multiple
npm install
pip3 install -e python/
npm run transpile
npm run export-exchanges
npm run transpile
python3 examples/py/websocket-playground.py kraken ob ETH/XBT 

I have had to upgrade from python3.5 to python3.6, and install pip3 to this python version, but if python3.6 or above is installed on your system, it should work

[firepol]

@Fcl69 I just checked my project (you linked my other project in java btw, the correct link is https://github.com/firepol/ccxt-websockets-db-updater ), fixed a few issues affecting ob_tester.py and made a demo video that shows it in action https://www.youtube.com/watch?v=0UIvIBUiqQQ

Maybe in the readme it was not clear, but if you specify no --verbose and no --debug option, there is no output. Sorry if that was not clear. So, just try to set it up as in the video, from scratch. If you use linux as I do, you can just copy/paste the installation code block.

Also note, ob_tester.py doesn't populate the DB: it just connects to the websockets and does nothing.
ob_updater.py populates the DB.

Btw, in case you have trouble using my ccxt-websockets-db-updater project, feel free to write an issue in my repository https://github.com/firepol/ccxt-websockets-db-updater and I'll try to help you to get things up and running.

Also, I keep seeing the same issues, that people forget to transpile etc. and in fact I created already a transpiled repository a long time ago, to make it easier to get started (I use it as dependency in my ccxt-websockets-db-updater program): https://github.com/firepol/ccxt-websockets note that this is not suitable if you plan modifying the CCXT project, it's meant "read only" just to "consume" lfern's work ;)

[firepol]

@lfern I tried huobipro using the latest version of your branch, in my project (feel free to try it, maybe you can use it to test your changes in python in a more advanced scenario), but it doesn't seem to work. I get no output.

See instructions at https://github.com/firepol/ccxt-websockets-db-updater (see also my video)

Working connection (with output):

./ob_tester.py -e binance -s eth/btc --verbose --debug

Huobipro (it doesn't need api keys, or does it?), not working, maybe can you have a look at it?

./ob_tester.py -e huobipro -s eth/btc --verbose --debug

Kraken, same, no error, but no output:

./ob_tester.py -e kraken -s btc/usd --verbose --debug

Expected: see websockets output, as in binance. Result: nothing...

Ideas?

[Fcl69]

@lfern @firepol I was able to launch script using a different setup.

Thank you very much for your help. I'll keep digging.

[lfern]

@firepol Huobipro has some transpilation erros and exchange.py gunzip method was wrong. It seems now it is working well.

[maayank]

Hey @lfern , two issues with poloniex:

1. In poloniex.js:1191, there's an error that goes over to python where channelId isn't casted to string and contextId isn't passed to emit(). So, it should be:
   1191 this.emit ('err', new ExchangeError (this.id + '._websocketOnMessage() failed to get symbol for channelId: ' + channelId.toString ()), contextId);

2. I've encountered the above as part of another issue where I can't subscribe to BTC/USD or BTC/USDT. I get, for example:
   ExchangeError: poloniex._websocketOnMessage() failed to get symbol for channelId: 121

Let me know if there's anything else I can check.
Cheers

[npomfret]

Thanks so much for the effort that's gone into this branch! Is there a readme for the origin/feature/websockets-multiple branch?

I have got it working, but was wondering what the API was? Specifically,

• what are the valid values for the event param for websocketSubscribe? _(looks like 'ob', is anything else supported?_
• how to use websocketFetchOrderBook method?
• the data in the orderbook callback for bitfinex has the entire order book. Is this a common format for all exchanges?

Also, I noticed the updateBidAsk method looks a problematic in that the bids/asks are stored in an array. For busy orderbooks it's going to be doing a lot of searching to make sure the items are stored in the right order. I think replacing the array with a binary tree would be a much more efficient solution.

[firepol]

Hi @npomfret all credits go do @lfern and other contributors, I'm just a "consumer" of this branch too. But let me help you since I have some minutes to use ;)

Yes there is a readme and I believe you can find some answers by reading it:

https://github.com/lfern/ccxt/blob/feature/websockets-multiple/WEBSOCKETS.md

Ws Features:

• ob: orderbook
• ti: tickers
• tr: trades
• oh: ohlcv

Other questions I can't help, but until then you have something to read. Hope this helps, cheers

[lfern]

Thanks @maayank , let me check out

Hey @lfern , two issues with poloniex:

1. In poloniex.js:1191, there's an error that goes over to python where channelId isn't casted to string and contextId isn't passed to emit(). So, it should be:
   1191 this.emit ('err', new ExchangeError (this.id + '._websocketOnMessage() failed to get symbol for channelId: ' + channelId.toString ()), contextId);
2. I've encountered the above as part of another issue where I can't subscribe to BTC/USD or BTC/USDT. I get, for example:
   ExchangeError: poloniex._websocketOnMessage() failed to get symbol for channelId: 121

Let me know if there's anything else I can check.
Cheers

[lfern]

@npomfret, websocketFetchOrderBook function returns a copy of the current internal orderbook managed by exchange. I think this function is not the best way to get orderbook info, it is better to listen 'ob' events emited as it receives updates from websocket connection. You can find some examples in examples folder in python, js and php language (files with "websocket-" prefix).

About orderbook format, I think that all of them has a similar format. Some of them let you to choose the depth so you receive updates up to this depth. In some cases you have to get a snapshot using REST API and receive update from websocket connections ... Anyway, each exchange implementation try to manage an internal copy of exchange orderbook and emit 'ob' events to subscriber when any update is received.

And yes, You are right about how to update askd/bids in the ordered array. From the begining this was a task that we had to improve, but we wanted to check if the way we were implementing websockets would work fine.

Thanks so much for the effort that's gone into this branch! Is there a readme for the origin/feature/websockets-multiple branch?

I have got it working, but was wondering what the API was? Specifically,

• what are the valid values for the event param for websocketSubscribe? _(looks like 'ob', is anything else supported?_
• how to use websocketFetchOrderBook method?
• the data in the orderbook callback for bitfinex has the entire order book. Is this a common format for all exchanges?

Also, I noticed the updateBidAsk method looks a problematic in that the bids/asks are stored in an array. For busy orderbooks it's going to be doing a lot of searching to make sure the items are stored in the right order. I think replacing the array with a binary tree would be a much more efficient solution.

[lfern]

@maayank I think poloniex is fixed. Please check it out

Hey @lfern , two issues with poloniex:

1. In poloniex.js:1191, there's an error that goes over to python where channelId isn't casted to string and contextId isn't passed to emit(). So, it should be:
   1191 this.emit ('err', new ExchangeError (this.id + '._websocketOnMessage() failed to get symbol for channelId: ' + channelId.toString ()), contextId);
2. I've encountered the above as part of another issue where I can't subscribe to BTC/USD or BTC/USDT. I get, for example:
   ExchangeError: poloniex._websocketOnMessage() failed to get symbol for channelId: 121

Let me know if there's anything else I can check.
Cheers

[npomfret]

Thanks @lfern and @firepol

I've used binary trees to implement orderbooks before, its very simple. I think the package I mentioned is the one that coinbasepro uses on their site.

[maayank]

Works well. As always, many thanks @lfern :)

[furryboffin]

hi, I'm trying to find some examples of how to connect to the websocket for bitmex in python. I'm happy to help add functionality for exchanges if needed. which branch should I start from for the most up-to-date python examples?

[firepol]

Hi @furryboffin check this out: https://github.com/lfern/ccxt/tree/feature/websockets-multiple

Examples etc. read here: https://github.com/lfern/ccxt/blob/feature/websockets-multiple/WEBSOCKETS.md

Read "Before you test in python or PHP" (you need to transpile to generate the python files).

For your convenience I created a git repository containing (already transpiled) the latest code from lfern's branch: https://github.com/firepol/ccxt-websockets

Enjoy

[aviad21]

Hi,

Encountering 'CloudFlare' error, when letting the code run for several hours. This was reproduced multiple times. I'm running python's example code:

examples/py/websocket-playground.py kraken ob ETH/XBT

1. Kindly asking if there is an implemented (or planned) mechanism intended to prevent this? (I've generally read about a requirement for some keepalive, e.g. ping?)
2. When it does occur - What should be the recommended approach to handle this exception? Should an unsubscribe and subscribe again be good enough?

NetworkError:CloudFlare WebSocket proxy restarting
File "examples/py/websocket-playground.py", line 64, in loop.run_forever()
File "/usr/lib/python3.6/asyncio/base_events.py", line 427, in run_forever self._run_once()
File "/usr/lib/python3.6/asyncio/base_events.py", line 1440, in _run_once handle._run()
File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run self._callback(self._args)
File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/autobahn/asyncio/websocket.py", line 100, in connection_lost self._connectionLost(exc)
File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/autobahn/websocket/protocol.py", line 3365, in _connectionLost WebSocketProtocol._connectionLost(self, reason)
File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/autobahn/websocket/protocol.py", line 1105, in _connectionLost self._onClose(self.wasClean, self.remoteCloseCode, self.remoteCloseReason)
File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/autobahn/asyncio/websocket.py", line 181, in _onClose res = self.onClose(wasClean, code, reason)
File "/home/user/Projects/ccxt/python/ccxt/async_support/websocket/websocket_connection.py", line 43, in onClose self.event_emitter.emit('err', Exception(reason))
File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/pyee/__init__.py", line 151, in emit
result = f(args, *kwargs)
File "/home/user/Projects/ccxt/python/ccxt/async_support/base/exchange.py", line 782, in websocket_connection_error self.emit('err', NetworkError(error), conxid)
File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/pyee/__init__.py", line 151, in emit result = f(args, **kwargs)
File "examples/py/websocket-playground.py", line 47, in websocket_error traceback.print_stack()
Closing Loop
after complete

I would like to contribute, either through debugging or if you can hint me towards the general direction of investigating this...

Cheers!

[lfern]

Hi @aviad21 . In theory Kraken sends a message every second (https://www.kraken.com/features/websocket-api#message-heartbeat) and it seems that CloudFlare could close sockets after 100 or 1000 seconds of inactivity, so this error should not take place. Maybe we can include a ping request if we don't receive any packet in a short period of time.

Anyway, you can catch exceptions emited from exchange and decide what to do then:

• always try to reconnect inmmediately
• wait some period of time if previous error was raised a short time ago.

Take a look at examples/py/websocket-recover-connection.py and websocket-recover-connection2.py. These scripts simulates a NetworkError and wait 5 seconds before recovering websocket connection. Basically:

 @exchange.on('err')
    async def websocket_error(err, conxid):
        # log error
...
        # close websocket if necesary
        exchange.websocketClose(conxid)
       # analyze what to do now
        if isinstance(err, ccxt.NetworkError):
            # wait 5 seconds
            await asyncio.sleep(5)
            try:
                # clean current context data
                exchange.websocketCleanContext(conxid)
                # subscribe all symbols again
                await doSubscribe(exchange, symbols, {
                    'limit': limit,
                })
            except Exception as ex:
                print(ex)
                sys.stdout.flush()
        else:
...

Or you can try a new exchange method that try to reconnect to websocket ans subscribe again to subscribed symbol (it still is not fully tested)

async def websocket_error(err, conxid):
        #log error
        if isinstance(err, ccxt.NetworkError):
            # close if necesary
            exchange.websocketClose(conxid)
            #wait 5 seconds
            await asyncio.sleep(5)
            try:
                await exchange.websocketRecoverConxid(conxid)
            except Exception as ex:
                print(ex)
                sys.stdout.flush()
        else:

[aviad21]

@lfern, I appreciate the reply, thank you.
Following several hours of trying to get to the roots of the above mentioned issue, I can summarize the following observations:

• After some duration at which all is running well, websocket_ob() events stop occurring all together
• websocket_error() is not received either (i.e. except for the above 'CloudFlare' one, which appears as a symptom of the actual issue...)
• I'm subscribed to multiple symbols of a specific exchange - when the issue occurs, all of the symbols stop receiving websocket_ob() updates
• This reproduces repeatedly on: poloniex (highest reproduce rate), huobipro, kraken.
• This was not yet reproduced over the last 24 hours on: binance, bitstamp, coinbasepro.
• (i.e. each exchange is running separately, on a dedicated python process. Processes run simultaneously on same computer. No RAM or high CPU issues...)
• This reproduces both on my pc, and on GCP instance, python 3.6.7
• I'm running asyincio logging in debug mode, but do not see anything alarming

A further hint of things going wrong, is observed when I try to recover from this situation.
Since there is no websocket_error() - After of timeout of 60seconds that nothing is received, I try to run: websocketClose(), then websocketCleanContext(), asyncio.sleep(5), and finally doSubscribe(), as you've mentioned above.

A first exception is raised when issuing websocketClose():

2019-03-15 13:09:54,345 - INFO: handle_log: websocketClose conxid: default
2019-03-15 13:09:54,345 - ERROR: handle_log: GENERIC (non ccxt.BaseError) exception while websocketClose: 'NoneType' object has no attribute 'close'
Traceback (most recent call last):
  File "examples/py/websocket-generic-orderbook-poloniex.py", line 107, in handle_log
    exchange_ctx.websocketClose(conxid)
  File "/home/user/Projects/ccxt/python/ccxt/async_support/base/exchange.py", line 728, in websocketClose
    websocket_conx_info['conx'].close()
  File "/home/user/Projects/ccxt/python/ccxt/async_support/websocket/websocket_connection.py", line 107, in close
    self.client._closeConnection(True)
  File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/autobahn/asyncio/websocket.py", line 128, in _closeConnection
    self.transport.close()
AttributeError: 'NoneType' object has no attribute 'close'

And then when reaching doSubscribe() execution, another exception occurs. It is due to exchange.py's call _websocket_ensure_conx_active(), trying again to close the connection:

2019-03-15 13:09:59,388 - ERROR: exception close location 002: 'NoneType' object has no attribute 'close'
Traceback (most recent call last):
  File "/home/user/Projects/ccxt/python/ccxt/async_support/base/exchange.py", line 652, in _websocket_ensure_conx_active
    conx.close()
  File "/home/user/Projects/ccxt/python/ccxt/async_support/websocket/websocket_connection.py", line 107, in close
    self.client._closeConnection(True)
  File "/home/user/Projects/ccxt/venv/lib/python3.6/site-packages/autobahn/asyncio/websocket.py", line 128, in _closeConnection
    self.transport.close()
AttributeError: 'NoneType' object has no attribute 'close'

When I add a try-except on the close() attempts under _websocket_ensure_conx_active(), I am able to recover from the state, and resubscribe.
Alas... the issue keeps occurring once in a while (e.g. on poloniex it could be once in 30 minutes or so, on huobi and kraken observed once in several hours...) - so I need to understand the actual cause and resolve it.
Any hint on how I can further debug this?

[firepol]

Hi @aviad21 when you say you run the processes simultaneously, can I ask you if you do that similarly as I do in my program (based on lfern branch) https://github.com/firepol/ccxt-websockets-db-updater (see ob_updater.py I use one .py file and asyncio to loop all exchanges and all pairs)? Would be interesting to know if you can reproduce the issue also using my ob_updater.py...

[aviad21]

Hi @firepol

when you say you run the processes simultaneously, can I ask you if you do that similarly as I do in my
program...

I meant to say that I'm executing multiple python scripts concurrently, each launched independently:

python examples/py/websocket-generic-orderbook-krken.py &
python examples/py/websocket-generic-orderbook-poloniex.py &
python examples/py/websocket-generic-orderbook-huobipro.py &

(each of these is based on the ccxt\examples\py\websocket-playground.py example)

[maayank]

Hey @lfern , I believe there's a bug with poloniex or the python base/exchange class. It currently doesn't update the orderbook upon delete updates, i.e. when there's a (Price = p, Amount = 0) update.

This can be evidenced in 3 ways:

1. A slight modification of the original poloniex orderbook example: https://github.com/maayank/ccxt_polo_bug/blob/master/websocket-poloniex-orderbook.py
   In essence, it subscribes to BTC/USDT with a limit of 100 and quits with a message if one of the top 10 bids/asks prices is deleted (as opposed to have its amount set).
   It never quits (e.g. I've tested for ~3 hours)
2. Putting a print message on the execution branch that deletes the row in the internal orderbook data structure and grepping for it (or similarly, putting a breakpoint) shows the code of that branch is never called. See this copy of async_support/base/exchange.py at https://github.com/maayank/ccxt_polo_bug/blob/fe43543f88cd4079d167700119f910ddfb359d6d/exchange.py#L344
3. As far as I can tell, the issue is that:
   a. _websocket_handle_ob calls (non-async-)exchange.parse_order_book
   b. Which calls self.parse_bids_asks which filters items with amount == 0
   c. Later in the poloniex class _websocket_handle_ob_delta_cache calls (async-)exchange.mergeOrderBookDelta which calls updateBidAsk, which in turn expects amount==0 items to exist for signalling deletion
   d. git blaming the 'js' and 'python' non-async exchange classes shows the former becoming a simple map (without the filtering side-effect) in March 2018.

Unsure how to fix it myself, as it is potentially foundational.
FYI, let me know if I can further assist.

[furryboffin]

Hi @furryboffin check this out: https://github.com/lfern/ccxt/tree/feature/websockets-multiple

Examples etc. read here: https://github.com/lfern/ccxt/blob/feature/websockets-multiple/WEBSOCKETS.md

Read "Before you test in python or PHP" (you need to transpile to generate the python files).

For your convenience I created a git repository containing (already transpiled) the latest code from lfern's branch: https://github.com/firepol/ccxt-websockets

Enjoy

THANKS!

[aviad21]

Hi @kroitor,
Following @maayank's comment, It seems that zero-value-amount orders are not updating the cached orderbook, and thus related prices are not removed from the orderbook.

Possibly, the python origin of this was introduced in the following added condition:
https://github.com/ccxt/ccxt/blob/45febed9b792ab33c9454626e468e8e9672dad37/python/ccxt/base/exchange.py#L1233 ...wheres the intention was filtering out 'none' values?

If so, to avoid filtering out valid 'amount==0' entries, kindly consider using:

if bidask[price_key] is not None and bidask[amount_key] is not None:
...
if (price_key in bidask) and (amount_key in bidask) and (bidask[price_key] is not None and bidask[amount_key] is not None):

[npomfret]

Where does the timestamp come from in the gdax/coinbase pro order book feed? I ask because I'm getting updates with very old (more than 10 seconds) timestamps from time to time.

[317459062]

how can i use websocket in huobipro and zb for 'ob' and trade?
can u take a detail steps?
thx

[firepol]

how can i use websocket in huobipro and zb for 'ob' and trade?
can u take a detail steps?
thx

Hi @317459062, as I answered in the other issue you opened in the wrong repository (which is just lfern's branch, transpiled in python for convenience), here a link to my little project, where I tried to create a practical example on how to use lfern's work to save the latest order books in a database: https://github.com/firepol/ccxt-websockets-db-updater

You can easily see it in action. Follow the instructions in the readme to get started, then run this command:

python ob_tester.py -e huobipro -s BTC/USDT --debug --verbose

I just tested it and the tester works. To save the data in a database, see ob_updater.py usage. The error handling is missing, so it's still work in progress but hopefully it can get you started. Cheers.

[aviad21]

Hi @lfern, there appears to be a bug in Kraken's _transpiled_ python code: Orderbook 'bids' updates are not always handled.

From time to time, Kraken will send an orderbook update that contains both 'asks' and 'bids', in a single message e.g.:

[0,{"a":[["5035.90000","7.19104284","1554559551.265576"]]},{"b":[["5029.00000","0.00000000","1554559551.278368"],["3601.00000","0.03400000","1554235030.583914"]]}]

The relevant _transpiled_ code in ccxt\async_support\kraken.py seems to handle data only from first item of msg:

    def _websocket_on_message(self, contextId, data):
        msg = json.loads(data)
        event = self.safe_string(msg, 'event')
        status = self.safe_string(msg, 'status')
        if event is None:
            # channel data
            chanId = msg[0]
            data = msg[1]       ### <-- need to take into account a possible msg[2] as well

Please let me know if I should provide any further information about this. Thank you

Edit: The key issue is that 'a' and 'b' might arrive as two separated dictionaries, within a single list. As appose to 'as' and 'bs' which arrive as two keys of a single dictionary. Seen in Kraken's example payloads. This example python code will fix it:

if len(msg) is 3:
    data.update(msg[2])

(Should probably be placed under the if event == 'ob': line)

[bugs181]

Hey @lfern, wanted to let you know that I've added a couple events to the Coinbase/GDAX exchange (ticker, trades, and heartbeat). Willing to make a PR once I have it cleaned up. Running into a small issue though and can't seem to figure out the wait4readyEvent.

For the wait4ready:

(node:1558) UnhandledPromiseRejectionWarning: Error: WebSocket is not open: readyState 0 (CONNECTING)
    at WebSocket.send (.../node_modules/ws/lib/websocket.js:314:19)
    at WebsocketConnection.send (.../node_modules/ccxt/js/base/websocket/websocket_connection.js:81:28)
    at WebsocketConnection.sendJson (.../node_modules/ccxt/js/base/websocket/websocket_base_connection.js:37:14)
    at coinbasepro.websocketSendJson (.../node_modules/ccxt/js/base/Exchange.js:2106:35)
    at coinbasepro._websocketSubscribe (.../node_modules/ccxt/js/gdax.js:1032:16)
    at Promise (.../node_modules/ccxt/js/base/Exchange.js:2323:26)
    at process.internalTickCallback (internal/process/next_tick.js:77:7)

Although the exchange seems to work fine otherwise, would really like to solve the connection riddle.

[xmatthias]

@lfern from what i understand from the comments here, the websocket api is rather stable.

Are there plans to merge this to the main ccxt version soonish?

[lfern]

@lfern from what i understand from the comments here, the websocket api is rather stable.

Are there plans to merge this to the main ccxt version soonish?

Don't know if it is stable enough, see ccxt/ccxt#4335

[lfern]

Hi @bugs181, Someone reported some problems on slow connections, and I think there is a bug that throws this exception when you try to do a second call to subcription method before a first call had returned. If you have a public testing branch for this feature, I could check it out.

Hey @lfern, wanted to let you know that I've added a couple events to the Coinbase/GDAX exchange (ticker, trades, and heartbeat). Willing to make a PR once I have it cleaned up. Running into a small issue though and can't seem to figure out the wait4readyEvent.

For the wait4ready:

(node:1558) UnhandledPromiseRejectionWarning: Error: WebSocket is not open: readyState 0 (CONNECTING)
    at WebSocket.send (.../node_modules/ws/lib/websocket.js:314:19)
    at WebsocketConnection.send (.../node_modules/ccxt/js/base/websocket/websocket_connection.js:81:28)
    at WebsocketConnection.sendJson (.../node_modules/ccxt/js/base/websocket/websocket_base_connection.js:37:14)
    at coinbasepro.websocketSendJson (.../node_modules/ccxt/js/base/Exchange.js:2106:35)
    at coinbasepro._websocketSubscribe (.../node_modules/ccxt/js/gdax.js:1032:16)
    at Promise (.../node_modules/ccxt/js/base/Exchange.js:2323:26)
    at process.internalTickCallback (internal/process/next_tick.js:77:7)

Although the exchange seems to work fine otherwise, would really like to solve the connection riddle.

[QuantBits]

I’ve tried using the websocket branch, but I haven’t been able to understand the documentation and for some reason the code isn’t self explanatory for me. Would anyone mind just nudging me in the right direction getting by show an example on how I would get e.g the ws ticker for BitMEX up and running in PHP? Pusesdo code would probably work, as long as it use the correct metods. Hope some one can push me in the right direction. Thanks!

[maayank]

@btc-zz , just a nudge (especially when I never used the php code), but once you git clone the websockets branch you need to transpile (i.e. compile) the js code to php using npm. So:

git clone https://github.com/lfern/ccxt.git
cd ccxt
git checkout remotes/origin/feature/websockets-multiple
npm install opencollective
npm install ws
npm run postinstall
npm run export-exchanges
npm run transpile

It might ask for additional node.js modules - YMMV.

[bugs181]

@lfern

I think there is a bug that throws this exception when you try to do a second call to subcription method before a first call had returned.

This sounds like what I'm doing. I've noticed that await exchange.websocketSubscribe just hangs however. The promise will never resolve. I had assumed the ws implementation would queue the subscriptions. Most times after the connect error, it will retry the subscription and work fine. There have been some times where subscribing to the heartbeat channel fails and node closes after some time.

[lfern]

@bugs181 If websocketSubscribe hangs forever then there is something wrong in js implementation, you should get a timeout exception. Maybe you have a slow connection or you have to wait for some message sent by server before sending any subscription to server. Maybe gdax websocket server change the protocol. I would try to make some tests, Are you using js version?

[lfern]

@bugs181 I have just tested current gdax js version and it seems it is connecting properly. Maybe we can commit to websocket branch and do some tests.

[bugs181]

@lfern Just tested the code I had and it's working fine today. Strange indeed. It's possible that it was from a slow connection (we've had storms for about 2 weeks straight here).

Edit:
Is there an linting ruleset you'd like me to follow for PR's? I've noticed a few things about your coding style and want to ensure that we can agree.

[lfern]

@bugs181 Not really, follow ccxt general rules, and take care that js code transpile right to python and PHP.

Edit:
Is there an linting ruleset you'd like me to follow for PR's? I've noticed a few things about your coding style and want to ensure that we can agree.

[xCuri0]

@lfern Seems like cobinhood has changed their WS API or something. It gives timed out half of the time

[lfern]

feature/websocket-multiple merged to current ccxt/master version.

@xCuri0 how much time to get timeout error? Testing ob event without error for 1 hour.

[xCuri0]

@lfern around 75% of the time it fails to subscribe with timeout error. Doesn't happen when I manually use wscat to connect

[guotie]

how to use websocket api now? and is there some docs?

[xCuri0]

@guotie WEBSOCKET.md on the feature/websocket-multiple branch. Also you can look at the examples. Afaik reconnecting on a network error in WS doesn't work so except it to disconnect after a few hours unless you have perfect internet

[xCuri0]

Dunno why but if I simulate a network error by turning off WiFi for a while it can reconnect. A real network error gives AttributeError: 'NoneType' object has no attribute 'close' when trying to reconnect

[xCuri0]

@kroitor what are the requirements for the websocket branch to be merged into master ?

[guotie]

I did not found feature/websocket-multiple branch. @xCuri0

[xCuri0]

@guotie on lfern's CCXT there is

[guotie]

Ok, thanks

[xCuri0]

Seems like the only problem with the ws branch is the php build now. Hopefully it will be fixed so it can be merged into master

[adi90x]

Any update on this ?

[martalson]

@kroitor What's missing to be able to integrate? Lots of people looking forward to the websockets

[qwasko1212]

@kroitor Could you please tell, when websockets will be avaliable in ccxt?

[kroitor]

@qwasko1212 I will post an update here within a week or so. Plz hold on. Thx for your patience!

[kroitor]

Hello everyone, and a big thank you to all CCXT users and contributors! We appreciate your involvement very much!

First off, we want to thank the community for enabling us to get this far. You are all awesome people, and without you this project would not be in the position that it stands today. Even as CCXT has concentrated on RESTful APIs, we have always appreciated and understood your enthusiasm for WebSockets. The constant stream of comments expressing interest in the status of WebSockets has not gone unnoticed, despite the fact that we have remained mostly silent on the matter.

It is clear that CCXT cannot remain REST-only any longer. The community has waited for a long time and our silence has created frustration and even doubt that we were working on adding this feature. I apologize for not replying to everyone, but I wanted to provide a meaningful answer and not just an empty promise. We're very focused on the actual work of maintaining and improving CCXT, so we hope for your understanding.

Streaming support within CCXT has been in development for some time, and we are very close to having an architecture that satisfies our criteria for ease of use, hot-swapping, unification and portability. Everything has been coded internally from scratch to the same high standards that CCXT is known for having, with extensibility being an additional goal.

Over the past two years CCXT has grown exponentially, and the amount of required maintenance work and support has grown proportionally with it. As a MIT licensed open source project, we have self-funded everything, but at this scale we cannot continue to do that. Without the necessary resources to maintain the library in the face of a rapidly-changing crypto universe, it would soon lose functionality and decline in usefulness. Without continued development, new integrations would stagnate. None of this is good for the library, the community, or the development of cryptocurrency infrastructure to compete with traditional finance.

Which brings us to CCXT Pro. CCXT Pro is a professional add-on to CCXT that will support WebSockets and much, much more. CCXT will continue as an open source project under the MIT license and will be supported by you, the community, in conjunction with a better-funded CCXT Dev Team. CCXT Pro will be hosted from a private repository on a paid basis, but with the opportunity for top CCXT contributors to get free access.

This balances our strongly-held belief in the value and power of community-oriented open source projects while also addressing the reality of limited resources and a growing demand for a higher level of service and responsiveness from professional traders, larger companies, and financial institutions that use CCXT for mission-critical work. These commercial users want instant access to support, compiled languages, low latencies, additional trading protocols, and, of course, WebSocket capability.

Creating a commercial license and charging these CCXT Pro users allows us to continue the important work on CCXT and, we believe, is an ethical way of supporting the community and its contributors. We expect that some people will object to making CCXT Pro a commercial project, and we are interested in your opinions, particularly if you have been an active contributor. We want to reward our contributors with CCXT Pro licenses, and will soon open a channel where inquiries can be made to do this.

CCXT Pro's release is planned for Q3 this year. We will publish more details on the roadmap to CCXT Pro soon.

With Heartfelt Thanks and Appreciation!

[fredfortier]

Good to hear @kroitor ! This is the right decision. Please make sure that those with a professional license can still build from sources. Personally, I'm not concerned with paying for software, but I can't afford the security risks of trading with proprietary software.

[kroitor]

Please make sure that those with a professional license can still build from sources.

The licensees will have the access to the source code repository of CCXT Pro on GitHub and will be able to build from sources. Contributors will have a chance to get a free license.

[zanoo]

Thanks @kroitor. Any idea how much CCXT Pro will cost?

[kroitor]

@zanoo we will offer a simple grid of prices for different types or tiers of customers. Considering the purpose of the lib which is to make money, the starting plan for individual developers will be affordable – comparable to your monthly cable/sat tv subscription.

With CCXT Pro the user will communicate with exchanges directly without intermediaries. Even if your license expires CCXT Pro will keep working. However, it's in your best interest to have a license that will give you the access to the protected repository – including the most recent updates, bugfixes, dedicated support, new exchanges, methods and enhancements.

[firepol]

@kroitor from CCXT Pro, sure enterprises that need features and quick adjustments to the library will benefit, so I think you should charge them accordingly. Medium companies that will almost never bother/ask you something but make money thanks to CCXT should pay, but much less.

If you want to make even small fishes (private persons) pay I think it would be unfair (people can potentially use CCXT to trade, so yes "to make money", but ultimately they invest their funds and it's with their risk that they make money, not directly with CCXT). To consider that there are bots like 3commas, cryptohopper, gekko, gunbot etc. that cost very cheap and actually have some AI / algo that make people gain money.

Also if you wanna leave this for free only for "top CCXT contributors" the question will arise: who are top ccxt contributors, sure the ones doing the dirty work. What about people testing this and submitting lot of bug reports (like me) and useful ideas?

Actually, top contributors (e.g. like lfern) not only should get a free license, but would rather deserve to get paid too. I mean, else how do you plan to incentive active contributions? If you don't pay me a cent, I keep my code for myself, why releasing it for free and you make money with it and maybe even ask me to pay a license for it, because I'd not be a "top" contributor but just an "occasional" contributor?

I think it would be fair to keep a personal tier for people not having a business and using this as "hobby" under 10 USD/month. Or even better, for free. I mean: for western countries 10 USD is still ok, but I can imagine that some developers in development countries would like to save 10 USD/month...

...ultimately I think you should allow a license for free for people not using it commercially. E.g. if a developer codes his own bot based on CCXT, this is not commercial. Should be free of charge "non-commercial" license.

I coded some stuff based on CCXT and it's open source, would you want to make people like me pay for it? Seriously, this would remind me Apple requiring developers to pay 99 USD/year, even if they plan publishing free apps. Ridiculous!

If a developer created a bot or a software based on CCXT and wants to sell his software, then I agree that he needs to pay a commercial license. Hopefully, there should be a tier for this too. Imagine again a little developer, trying to code a tool based on CCXT and trying to sell it, but nobody will buys it at the beginning, maybe a few copies per year. There are many developers who try to make some little money after investing many many hours on something and not investing in marketing much to make it a super commercial product... I sure don't wanna spend 50-70 usd/month for a "commercial license" (e.g. coinmarketcap charges 70 usd/month, ridiculously high price for a small developer able to sell maybe 2-3 copies of an app at 3 USD each) and rather remove the CCXT dependency and use native APIs instead. It would be a step back and very disappointing one.

I hope you will consider such aspects and make free licenses for non-commercial people or people tying to use it commercially, but also not making big gains out of it... and relying on people honesty for it. E.g. look at jetbrains, syntevo. Until you are non profit using their tools, you pay nothing, the day you start making money you pay the licenses. It's fair enough.

I really hope that CCXT will remain for free for occasional users. Let the enterprises & commercial users pay! Thx

[kroitor]

@kroitor from CCXT Pro, sure enterprises that need features and quick adjustments to the library will benefit, so I think you should charge them accordingly. Medium companies that will almost never bother/ask you something but make money thanks to CCXT should pay, but much less.

Yep!

If you want to make even small fishes (private persons) pay I think it would be unfair (people can potentially use CCXT to trade, so yes "to make money", but ultimately they invest their funds and it's with their risk that they make money, not directly with CCXT). To consider that there are bots like 3commas, cryptohopper, gekko, gunbot etc. that cost very cheap and actually have some AI / algo that make people gain money.

Don't forget that the original CCXT with all of its functionality will remain under MIT and will still remain free, so, people who use it are not affected.

The paid CCXT Pro part will be the professional part, which includes streaming, that means higher speeds to be more competitive. That part (which has not been released yet) will be paid, and it will be an optional add-on. You will be able to either build upon the free CCXT, or seamlessly switch to the commercial version and use CCXT Pro to be slightly faster and more competitive.

That paid part is not based on the work done by @lfern, but is coded internally by the CCXT Dev Team from scratch. But @lfern undoubtedly deserves a free license. And licensees will have the access to the source code.

Also if you wanna leave this for free only for "top CCXT contributors" the question will arise: who are top ccxt contributors, sure the ones doing the dirty work. What about people testing this and submitting lot of bug reports (like me) and useful ideas?

Top contributors are people who have contributed significantly to the project in this or that way. Either with code, or with testing and reporting (we'd love to incentivize testers as well, by the way ;)), or by other means. Plz, don't worry, we won't accidentally miss people who have put their effort into this project. Knowing that you have indeed helped with it a lot over the year, we will, of course, account for that.

Actually, top contributors (e.g. like lfern) not only should get a free license, but would rather deserve to get paid too. I mean, else how do you plan to incentive active contributions?

In fact, that is exactly what we'd want to do along with the other things – we are going to reward contributors and maybe even invite contributors to join the project. In other words, the whole purpose of making CCXT Pro a commercial project – is to sustain the living of both CCXT and CCXT Pro. Because... see the answer to the next question...

If you don't pay me a cent, I keep my code for myself, why releasing it for free and you make money with it and maybe even ask me to pay a license for it,

... why would I keep maintaining CCXT? Why the CCXT Dev Team would put the effort into releasing and maintaining the code? ) Why would we help others build their projects? I've released quite a lot of code done by myself so that you could use it for free and I did not demand a payment from you ) Why would I do that do you think? ))

To me a better attitude is the other way around – making it commercial will help us properly incentivize the contributors. And, again, the original CCXT will remain free. CCXT Pro will be paid though. And, ideally, they will be interoperable, means you could switch back and forth without even changing your existing userland code.

I think it would be fair to keep a personal tier for people not having a business and using this as "hobby" under 10 USD/month. Or even better, for free. I mean: for western countries 10 USD is still ok, but I can imagine that some developers in development countries would like to save 10 USD/month...

We will do our best to make it affordable for individuals. Our starting tier will be close to your estimate. However, please, note, that in order to provide the access to sources and to remain accessible to GitHub users (to be easy-to-use with the existing infrastructure) we might have to spend $9 per seat per month (https://github.com/pricing), not including other costs and not including the effort of the devs.

...ultimately I think you should allow a license for free for people not using it commercially. E.g. if a developer codes his own bot based on CCXT, this is not commercial. Should be free of charge "non-commercial" license.

If it is a non-commercial use, then (if you think of it, logically) it should fit perfectly into the category of purposes of CCXT which remains free and will be developed further under the MIT license. So, those who build non-commercial software and free tools, and even commercial bots based on CCXT – those can still enjoy everything for free and they are not affected. But there are also users who want to enjoy slightly more than others – it is ethical to charge them for extra speed.

If you think of any purpose of streaming – that is purely commercial. Can you name at least one non-commercial purpose of using a slightly faster tool when a free version of the same tool exits, all the rest being equal? Taking it further, if it's a bot, then this is a commercial usage for sure, in all senses. Because the pure purpose of a bot is to be commercial, "to make money", obviously. Why would you need faster speeds other than for building a faster bot? All other purposes including historical tracking – don't really have to be instantaneous, in other words, you would only need CCXT Pro if you want "to make slightly more money than average". If you don't need a faster bot, then you can continue using the CCXT under MIT. Absolutely for free ) And that's honest, open and fair.

I coded some stuff based on CCXT and it's open source, would you want to make people like me pay for it?

Of course, not. Because CCXT remains free.

Seriously, this would remind me Apple requiring developers to pay 99 USD/year, even if they plan publishing free apps. Ridiculous!

Totally agree with you on this example. That is why:

• We will have a "CCXT bRO" license which will be free for non-commercial use and academic use, and will be free for our contributors, who will be able to get a free license by contributing code to the CCXT Pro repo.
• Top contributors to the CCXT repository will also get their free bRO licenses for CCXT Pro. We will provide more details on that.
• CCXT will remain fully free under MIT and will be maintained even better if contributors get better incentivized.
• CCXT will be subsidized by CCXT Pro (otherwise it's not sustainable).
• CCXT Pro will charge for the advantage only, basically, that includes the maintenance, adding new integrations quickly, the development, support and the underlying costs.

If a developer created a bot or a software based on CCXT and wants to sell his software, then I agree that he needs to pay a commercial license. Hopefully, there should be a tier for this too. Imagine again a little developer, trying to code a tool based on CCXT and trying to sell it, but nobody will buys it at the beginning, maybe a few copies per year. There are many developers who try to make some little money after investing many many hours on something and not investing in marketing much to make it a super commercial product...

The license will be mostly addressing bigger fish. So, a poor developer will have the opportunity to struggle as long as he wants and our plan for beginners will be affordable. The CCXT Pro license will target those developing companies who don't struggle with marketing and who will use CCXT Pro to build successful products and services. Which is fair, again.

I sure don't wanna spend 50-70 usd/month for a "commercial license" (e.g. coinmarketcap charges 70 usd/month,

Our plan for beginners is far from that. No worries.

ridiculously high price for a small developer able to sell maybe 2-3 copies of an app at 3 USD each) and rather remove the CCXT dependency and use native APIs instead. It would be a step back and very disappointing one.

We understand that, therefore we don't have the intent to shoot the project in the leg by raising the entry threshold too high.

I hope you will consider such aspects and make free licenses for non-commercial people or people tying to use it commercially, but also not making big gains out of it... and relying on people honesty for it.

We think the same.

E.g. look at jetbrains, syntevo. Until you are non profit using their tools, you pay nothing, the day you start making money you pay the licenses. It's fair enough.

Totally with you on this.

I really hope that CCXT will remain for free for occasional users. Let the enterprises & commercial users pay! Thx

CCXT will remain free for occasional users, moreover, it will remain free even for commercial users and for the enterprises.

CCXT Pro will have a commercial license, but will be affordable to beginners, professional traders, businesses and whales.

We will post more details in a week or two. Stay tuned!

[TechupBusiness]

Wow this is huge news, I didn't continue on my project the last months so I didnt really participate in ccxt discussions or development... not sure yet if I like these changes or not ;-) There must be a startup pricing for little growing businesses (without becoming a huge cost block one day), otherwise I would think twice using CCXT. At least for the free version I needed to write tests (using an active exchange account and very small amounts of fiat to test all objects and responses), improvements etc. by myself as it was far away from stable and unified objects. I hope you used TypeScript for the rewriting ;-) And as soon as money is involved, good QM is very important. Better less supported exchanges but more stable lib API. There is nothing worse than loosing huge amounts of money because of a buggy library.

[firepol]

@kroitor thanks a lot for the answer. Actually I think you should put all these points in a FAQ and publish them in the ccxt website/blog/twitter as soon as possible, and update them over time, so to have a central place and not a thread like this to scroll to find scattered information here and there.

This thread yes it's about streaming and now I understand there will be 2 different versions of the project, normal rest API and Pro = websocket/streaming version... got it now.

To me, charging more than 9 USD/month to occasional users is a no-go. Especially if you think that you have to waste 9 USD/month per sit, to pay the github license. I mean come on (this github team license is more expensive than the "evil" Apple Developer license!). I understand that it's more convenient to stay on github, but as mentioned (and you seemed to agree on that), if you rely on people honesty you can just create a new repo, leave the source code publicly accessible (just the license has to be clear), no need to pay 9 USD for any members and let everybody download it. So you still stay on github, but no need to pay 9 USD/month for people that will use this occasionally.

If you are afraid that people, like this, will download the source and use it without paying for the license, then it means you don't trust the paying audience... (the premise was that you trust people's honesty).

You can create/use a separate system for the licenses. I'm sure there are many ways to achieve this. Probably even accept crypto as payment, since this is a crypto project, why not?

Eventually, you can create another repo, ccxt-enterprise, where VIP people (enterprises) will pay the big check and get their special github account to report bugs/issues where you will give higher priority (if you need this). Could be a solution, no? Like this all small fishes won't need to be taken in consideration and you can skip that 9 USD/month for them... actually this "Team" feature of github doesn't really seem necessary to me, and it only increases costs to a project that should stay as cheap as possible for occasional/non-commercial users.

To name a single case where websockets are needed for no-commercial purposes: notifications.
As trader I like to monitor prices using my own bot and I want to be notified when a certain price is reached. Yes, ultimately I'd like to automate something when a price is triggered ;) It's "commercial"... but if I just play with pocket money and make 20 USD gains per month, should I pay a license of 10-20 USD and basically have 0 gains for me to cover the electricity costs used by my little test bot? I think until a small amount of gains, we can consider the usage still "non-commercial". Up to 100 USD/month net gains (after fees, taxes...) I'd say still non-commercial, it's just pocket money... after 100 USD/month commercial. Fair enough?

Also, another non-commercial use would be to create a bot for educational purposes, open source, to show my coding skills. In fact I'd like to find a python job, I'm tired to receive job offers in C#, so I want to show that I'm active in python development ;) I think many students could relate to this (they can create projects based on this and put them in their CV or use them in their bachelor/master thesis etc.), they can create a project based on CCXT pro, but they should be charged 0 for this (I insist on this, 0, not even the 9 USD to pay github - we don't need this), the only condition would be to open source their project. Think about this: every open source project based on CCXT is publicity for CCXT. Same for CCXT Pro. You can make a list of notable projects using CCXT and CCX Pro, new coders or enterprises can have a look at those projects and even use that code for their own implementations.

These my 2 cents concerns and ideas. Else... I'm quite excited about this. Will try to resuscitate my little project(s) based on CCXT and open source them some day in the future ;)

Any timeline for when the CCXT Pro will be available to use/download?

[frosty00]

To me, charging more than 9 USD/month to occasional users is a no-go. Especially if you think that you have to waste 9 USD/month per sit, to pay the github license. I mean come on (this github team license is more expensive than the "evil" Apple Developer license!). I understand that it's more convenient to stay on github, but as mentioned (and you seemed to agree on that), if you rely on people honesty you can just create a new repo, leave the source code publicly accessible (just the license has to be clear), no need to pay 9 USD for any members and let everybody download it. So you still stay on github, but no need to pay 9 USD/month for people that will use this occasionally

Staking ccxt devs livelihood on people's honesty is not something we are willing to do. Your argument that some people should be exempt from paying because they use our code only occasionally makes no sense. Say I want to take the train, however I only take the train once a month, does that mean I shouldn't pay for a ticket? Like @kroitor said, the starting tier will be very affordable (probably more than $9 a month), however anybody who has written a bot before knows that trading even the smallest amount of crypto in these volatile markets and you will lose / gain 10x that amount in a matter of days.

If you are afraid that people, like this, will download the source and use it without paying for the license, then it means you don't trust the paying audience... (the premise was that you trust people's honesty).

It's not that we don't trust people, we trust very much that most people will be dishonest in a setup like this (even I would not pay tbh)

To name a single case where websockets are needed for no-commercial purposes: notifications.
As trader I like to monitor prices using my own bot and I want to be notified when a certain price is reached.

Then use regular ccxt and poll the ticker. It is not hard. And 100ms does not matter for the notifications on your phone, because it will take 100x as long for you to manually trade anything.

It's "commercial"... but if I just play with pocket money and make 20 USD gains per month, should I pay a license of 10-20 USD and basically have 0 gains for me to cover the electricity costs used by my little test bot?

Realistically this is the user's choice, the fact that you made little gains using extremely fast ws code is not our fault, it just means that you should have designed a better bot (we know it is hard...)

bot for educational purposes, open source, to show my coding skills. In fact I'd like to find a python job, I'm tired to receive job offers in C#, so I want to show that I'm active in python development ;) I think many students could relate to this

I'm a student too, and we will definitely consider giving out free educational licenses to anyone who can prove they are in education by owning an educational email address, however we don't want this system to be abused (for example i get to keep my uni email forever).

[firepol]

@frosty00 you are still wasting 9 USD/month for a github sit per person. You could give this as rebate for your users instead.

Train ticket actually is a good example: if I use the train 3 times a month I think it's fair to buy 3 tickets and not a monthly subscription ;) to note: in some countries seniors don't pay for public transports. In some cities, like Tallinn (Estonia) public transports are for free. So yes, in a greedy world where we must pay for everything, it's still possible to offer free stuff if one wants to.

For all your other points. I would not debate on it.

In the end maybe I was a bit too much dramatic, but if you can deliver a good product as you said you'll do, probably paying even more than 9 USD/month would be ok for most users.

For the few ones that won't be happy, I guess there will be free alternatives to be used, such as https://github.com/bitrich-info/xchange-stream or @lfern's open source fork.

Looking forward to see CCXT Pro.

[iuvbio]

@firepol if the purely honesty based system worked, @kroitor and devs should be making enough money to not have to take this step. The index page lists several options to sponsor, even small amounts, so anyone making any serious kind of money with CCXT should probably be donating already. I understand this step and I think it is fair and viable, as kroitor says, under one condition. In my view, this can only work if CCXT Pro stays strictly an add-on to CCXT. By that I mean that for example the websocket implementation should use all the same exchange classes from "base" CCXT, so that all updates and improvements for Pro will necessarily be made in CCXT as well. As soon as this becomes two separate projects, I can hardly imagine it to work. I fully trust that the dev team has all the intentions to keep supporting CCXT even then, after all, why should they be supporting it now otherwise. However, as soon as you do have paying customers for a separate product, obviously all priorities are set with that product first. Maybe CCXT would live on as a community project, but even that is not certain. Anyway, as I understand it, the add-on feature is exactly the plan, so I think this is a good decision and will only make better CCXT as a whole.

[AadaEa]

The fact that CCXT Pro gets to interact with the exchanges directly, without intermediaries is all that matters in my view. access to the source code got me sold on this.

I wish the CCXT team would generate an higher income from the CCXT base library alone, I suspect CCXT is used by more and bigger users than we could think of, and I can only see it growing over the years with the crypto market.

It makes sense to me to charge accordingly commercial businesses expecting some level of consistency, as opposed to individual developers using the tool more modestly, taking the risk to encounter bugs, and likely to contribute, either through commits, suggestions, or bug reports.

Streaming might justify an optional paid licence; it's hard to tell how much of a difference it would make for small investors. But it has certainly been some serious work for the team, and every work deserves something in return.

Got some questions.
Will we have access to the CCXT Pro manual before purchasing the product, to have an understanding of all the subtleties of CCXT Pro? or maybe it will be included in the base manual?

Also which technology does CCXT chose for the php streaming library? there's many solutions, reactphp, amphp, ratchet ... can we have some insights on which solution was/is preferred for use with CCXT streams? were you able to run some benchmarks to determine which solution would fit the most to this kind of tool? and how about server side, would we need to build a particular process/daemon in another language in order to handle the stream as they're getting pushed to the WS server?

[kroitor]

@AadaEa

Will we have access to the CCXT Pro manual before purchasing the product, to have an understanding of all the subtleties of CCXT Pro?

Yes. We are making them seamlessly compatible as much as we can, so, it may turn out that most of the Manual is already there ))

or maybe it will be included in the base manual?

That is also possible, yes.

Also which technology does CCXT chose for the php streaming library? there's many solutions, reactphp, amphp, ratchet ...

For PHP we're using some components of ratchet and some parts of reactphp.

can we have some insights on which solution was/is preferred for use with CCXT streams?

Ideally, you would want a link to the CCXT Pro Manual instead of this comment – that would answer all your questions, of course ) However, I can't tell a lot right now, we will post more updates along the way (in timeframes mentioned above).

were you able to run some benchmarks to determine which solution would fit the most to this kind of tool?

Yep, we did our research) If some part of it becomes a bottleneck, we will optimize it.

and how about server side, would we need to build a particular process/daemon in another language in order to handle the stream as they're getting pushed to the WS server?

Both CCXT and CCXT Pro are client-side tools, meaning that the client uses CCXT and/or CCXT Pro for connecting to the exchanges' REST APIs or to their WS servers. CCXT can be launched from a browser, from command line, from node, python or php, it can be scheduled as a script, can be run inside a web server, and your server can be a client to exchnages as well. So, it depends on how you set up your things and how they relate to each other. CCXT Pro will not have a WS server inside it.

If you want to receive a stream of data from an exchange to your server and you want to forward it to your clients, that's basically a multiplexing proxy. For it you would use CCXT on your server to get the data, and you would then stream it to your clients with userland code depending on the needs of your application, project or system. Serving websockets is outside of our scope for now. CCXT Pro is designed to be an efficient client (ccxt-connects-to-exchanges), to receive data from the exchanges and to send data to the exchanges over streaming channels, but it is not designed to work as a standalone streaming server to serve a multitude of other clients (no clients-connecting-to-ccxt). Your clients may connect to your server, but handling them is beyond the purposes of CCXT Pro. So, if you mean that specific usage, then, yes you may have to add some functionality with the existing npm modules or with your own code, to be a WS-multiplexing-intermediary-proxy-server for your clients.

If, however, you mean launching general code that uses CCXT Pro on whatever server or computer it is launched on – then, yes, it is designed to work from anywhere, just as CCXT. According to our fundamental portability principles, you won't have to code in other languages to use CCXT Pro streaming in PHP on your server-side. You will just use PHP. Or Node. Or Python. It's better for you if you know more than one programming language, however you're not required to know more than one language to use CCXT Pro to its full extent.

[AadaEa]

We had a different definition or I wasn't clear, what I call "server side" was in fact an hosted server running a copy of the CCXT library through php which gather data and runs a set of instructions with cron jobs, which can also be called a client from a different point of view :P Not seeing the use of a multiplexing proxy.

What I had trouble picturing was how CCXT would have a permanently open script, which allow php to interpret data in real time and run instructions based on the input and that doesn't generate leaks or crashes; maybe through the use of Supervisor. Which would achieve the right behavior through ratchet. Handling this kind of real time process efficiently can quickly get quite complex and require OS level knowledge.

Can't wait to learn more about this!

[eburnette]

I'm a developer, and I run a one person startup that does commercial trading bots, which of course use CCXT (in NodeJS). Up to now I've had to use other libraries for streaming, and there's a different one for each exchange. I'd like to be able to use CCXT streaming but it depends on the price. I have workarounds (the other libraries or writing it myself), but they have a cost also. I'm not opposed to paying for support but I'm not sure the Pro add-on model will work for my very small but growing business. It will be important for it to be cost effective for my business, to get all the source for debugging, and to have freedom to run it on any number of machines without having the usage be metered or otherwise limited. Please feel free to contact me to discuss early plans and I'd be happy to provide feedback and even beta testing.

[synchronizing]

Hey @kroitor, myself and my colleague @Alescontrela are willing to help out on the development side of things. We are currently running a small framework with the intention of solving the exact problem CCXT Pro is being made for; CryptoBook (universal API for crypto exchanges, as well as WebSocket support). We can diverge some of our efforts to CCXT Pro as devs and testers, as we think it might be more worthwhile to combine efforts than work on separate projects. 👍🏼

[npomfret]

Would it be possible to get gate.io ws api v3 (https://gateio.io/docs/websocket/index.html#depth-subscription) implemented?

[kroitor]

@npomfret hopefully, yes, we will look into adding it.

[MohammadrezaNargesi]

Hello kroitor,
Thanks for all your efforts.
Can you please let me know when Websocket API, Real time Orderbookfetcher, Real time buy and sell details will be added ?
I need websocket version , is it implemented ?
Regards,

[npomfret]

Hi. The kraken ws feed is blowing up with Cannot read property 'data' of undefined for some markets. Try XTZ/BTC for example.

The error lies in:

_contextGetSymbolData(conxid, event, symbol) {
    return this.websocketContexts[conxid]['events'][event][symbol]['data'];
}

The ob event is expected to look something like:

{ 'XTZ/BTC':
   { subscribed: true,
     subscribing: false,
     data: { depth: '50', limit: 200, ob: [Object], rawData: [Object] },
     params: {} } }

But, after a few seconds (or sometimes immediately) it receives something like this:

{ 'XTZ/BTC':
   { subscribed: false,
     subscribing: true,
     data: { 'sub-nonces': [Object], limit: undefined, depth: 1000 } } }

I can't print out the exact JSON because there's a circular structure somewhere in here.

My code works find with other markets (XTZ/ETH for example). I'm not subscribing twice, or resubscribing, subscribing to multiple markets or anything like that.

[xCuri0]

@npomfret are you using ifern's fork ? it has alot of bugs which probably arent going to be fixed because another websocket fork is being developed

[npomfret]

are you using ifern's fork

I am yes.

another websocket fork is being developed

Ok! Thanks

[xCuri0]

what is the status on ccxt pro ? when will release be expected ?

[kroitor]

@xCuri0 it's under very active development and we are going to post a standalone announcement within a week or two. The release is planned for Q3, this September.

[xCuri0]

@kroitor what exchanges will be supported at release ? will we be able to contribute and add more exchanges ?

[kroitor]

@xCuri0

@kroitor what exchanges will be supported at release ?

We will post more details in the upcoming announcements. Many of them will be available, we will prioritize topmost exchanges by volumes and reputation.

will we be able to contribute and add more exchanges ?

Yes, see this comment: https://github.com/ccxt/ccxt/issues/56#issuecomment-508280756

[teneon]

I am also waiting for the announcement about "pro" (paid) version. 15 days have passed, give us some more info please ;-D If the price is right i will buy the pro version as soon as it is released.

[xCuri0]

@kroitor update on status of ccxt pro ?

[kroitor]

@teneon

I am also waiting for the announcement about "pro" (paid) version. 15 days have passed, give us some more info please ;-D If the price is right i will buy the pro version as soon as it is released.

I cannot reveal the full pricing grid right now. However, as was mentioned above, the simplest plan will be comparable to a monthly cable tv subscription, around $20-25. To early birds we will also offer a discount of approx a quarter off of the price, shrinking it down to $15-20 per month. Top contributors will get a free CCXT bRO license.

What you get:

• streaming
• access to the source code
• fixes, updates and support
• new exchanges (quicker integrations)
• new functionality

@xCuri0

@kroitor update on status of ccxt pro ?

At this point we have finished the work on the licensing subsystem that operates on top of GitHub repositories. That component was mission-critical, since the source code of CCXT Pro will be available to the licensees, and we seek protection of the intellectual rights to a certain extent. The plan was to release CCXT Pro this month. We are still busy on some parts of the core code, but most of the hard work is done. You might have noticed that the frequency of updates of this MIT CCXT repo has decreased recently, that's directly related, as we now spend most of our time working to get CCXT Pro released, it's our top priority. We hope to announce the release in 4 weeks or so, and I will post more details on our progress the closer we get to the goal.

Thank you!

[maayank]

@kroitor , will the new WS framework provide 'trade' events?

[frosty00]

@maayank, there will be no events, just resolved promises. So in js and python you will call a binance.fetchWsTicker and it will return a promise / future, that completes once the message has arrived. So from the users perspective it will work seamlessly with the rest of ccxt. simply await binance.fetchWsTicker(symbol, etc). In php we will use callbacks but the derived exchange code remains transpilable thanks to some clever coding by the ccxt dev team ;)

[kroitor]

@maayank in short, yes.

[qwasko1212]

@kroitor What about pricing for students? From above msg you told that you will provide free access to your library. How do you validate students?

[xCuri0]

@frosty00 what if you want to use events ? i think ccxt pro should add support for them

[kroitor]

@xCuri0 it is done on principles similar to generators and, put simply, that functionally is a superset that covers the event model, in other words, it's flexible enough in its core. Thus, if you want events – you can easily make an event-like wrapper and we will probably include some examples of using it as a callback-style interface.

[amitkukadia]

Will the CCXT pro version have realtime pricing data? or just orderbooks and balances. Need it for an app I am working on, and would prefer websockets over polling.
Something like ccxws, just more exchanges supported.

[kroitor]

@amitkukadia

Will the CCXT pro version have realtime pricing data? or just orderbooks and balances.

When you say _"pricing data"_, can you list out which particular pieces of information you mean?
In general, CCXT Pro is going to support as many types of streams as possible, including:

• tickers
• orderbooks
• public trades (fills)
• other public streams, if any
• account updates
• balance updates
• private trades
• order updates
• other private streams, if any
• ...

[amitkukadia]

OHLCV and trade fills. Seems it will be covered. Thanks @kroitor .

[teneon]

Hi @kroitor ,
are there any news about CCXT pro? September is coming to an end. It would be okay even if it is beta, we would like to see the API and some docs, so we can start experimenting and implementing. It doesn't need to be perfect from the start? At least update us with info every few days, it is much better than silence.

kind regards!

[amitkukadia]

there will be no events, just resolved promises. So in js and python you will call a binance.fetchWsTicker and it will return a promise / future, that completes once the message has arrived.

@frosty00 Will it be possible to subscribe to a ticker or trade for an asset pair or all pairs?
something like

binance.on('trade', data =>{
    console.log(data)      
})

[xCuri0]

Any updates on CCXT Pro its October ? @kroitor

[lfern]

websocket experiment in chrome browser:
https://github.com/lfern/ccxt/blob/feature/websockets-multiple/examples/html/websocket-example.html

[lfern]

@amitkukadia , this websocket example is not part of ccxt pro future implementation. But it would be nice that ccxt pro would implement this feature.

[frosty00]

@lfern we will definitely implement this feature and will host a demo of multiple exchange orderbooks on the ccxt pro website. For some reason websocket bypasses the same-origin policy in most browsers that is supposed to stop xss attacks, so much for browser security...

@xCuri0 it is coming soon; we are working on it. I would estimate that in the next month it could be released, however this depends on many factors so we can't give an exact release date.

@amitkukadia yes it will be possible to subscribe to tickers and trades (that is one of the main selling points)

[KlausGlueckert]

@frosty00 That sounds great!
Do you already have a dedicated website for CCXT pro?

I have used https://developers.shrimpy.io, their WS was free for 16 exchanges, but now they are charging 350+/mo for it, if you subscribe to 100 coins (6 per exchange), this is way too much IMHO.

With shrimpy I had to track the book updates and cancel and re-subscribe if I missed a sequence, this was kind of inconvenient. Will you handle it similarly or automate it, so that the software generates/requests a book snapshot like every minute?

[frosty00]

Will you handle it similarly or automate it, so that the software generates/requests a book snapshot like every minute?

The entire process will be automated, so you will not have to handle orderbook deltas or updates, instead you will only have to handle the latest entire orderbook object. We will also ensure that no sequences or nonces are skipped when dealing with incremental data.

[npomfret]

Does the lfern fork support binance orderbook parameters (https://www.binance.com/en/support/articles/360032916632) allowing depth and frequency to specified?

[lfern]

@npomfret I have just added it. It seems it works, but it would need some test.
This example to test partial orderbook;

node examples/js/websocket-playground.js binance partob ETH/BTC obdepth:20 obinterval:100ms

And this one for 100ms orderboo interval

node examples/js/websocket-playground.js binance ob ETH/BTC obinterval:100ms limit:5

Does the lfern fork support binance orderbook parameters (https://www.binance.com/en/support/articles/360032916632) allowing depth and frequency to specified?

[frodoe7]

We expect that some people will object to making CCXT Pro a commercial project, and we are interested in your opinions

I'm from these people 😄
I'll post my opinion as long as you're interested to listen
I pay if I generate profit from integrating the library - what if my project still just idea or beta
I'll not generate money now - so why to pay ?
what about giving a free trial with long time , 6 months or year
to ensure the person who purchase the code will take enough time to develop and publish his project before generating money and pay.

[WoK-E]

what about giving a free trial with long time , 6 months or year
to ensure the person who purchase the code will take enough time to develop and publish his project before generating money and pay.

You are missing a "somewhat important" point. There are people that work hard to develop and share this great library, and they should be paid for their work. CCXT is a great library, it is live, it is maintained, and it is free for all of us to use. I guess you can view CCXT Pro as the next step, and there will probably be a user manual that you can access for free in order to develop your project. If you use, trust, and like CCXT, you know what to expect from CCXT Pro. The fee/cost you will be asked to pay is a different issue, and it is up to you to gauge if you can have a ROI and profits.

I pay if I generate profit from integrating the library

An even better approach would be to just ask for the profits you would like to have...

[frodoe7]

An even better approach would be to just ask for the profits you would like to have...

I understand you and I may not afford to pay before generating money ,
however , no problem - I found another library to use with CCXT.

I think , it will be a good mix

I really appreciate and would like to support these open source projects whether with financial contribution or with another shape of contribution.

Simply , my project won't be live and usable without these open source projects.
I may just have lack with budget :blush:

[KlausGlueckert]

@frodoe7 Which library do you use? shrimpy.com is offering 19/month plans now for 17 exchanges websocket streaming all channels. So I guess the ccxt pro might be priceded similarily. Shrimpy is just not so conventient to integrate, you have to track missed orderbook updates, and also there is a subscription limit per client, so you have to write extensive logic to handle that and spread subscriptions over different IPs.

[frodoe7]

@KlausGlueckert , Not Shrimpy
this one : https://www.npmjs.com/package/ccxws
it supports less exchanges , but that's initially okay with me

[WoK-E]

I understand you and I may not afford to pay before generating money ,
however , no problem - I found another library to use with CCXT.

@frodoe7 I have ho idea what your project is, and what I write below may well be a waste of your time to read, or even off topic, but I do write it with all goodness in heart, and it is what I've learnt over a number of years.

Each exchange is a different 'animal', with its own fee schedule, different precision for each currency and pair, different minimum and maximum amounts, as well as different traders, with their own (sometimes collective) mentality, different daily volumes for each pair, etc. etc. And even if you have a grasp on how an exchange "works", and you can actually make money on it, expect all this to change. I've seen it many times. Having an up-to-date orderbook all the time, is not really important, as it does not guarantee that your orders will go through when you expect them to. We are not the only ones "consuming" the right orders... Also, having a sound application logic does not guarantee that it is the right logic for the exchange that you're working on. Trading is trading; it is not 1+1=2. It is a race against other traders, who have more money, better code, faster machines and sit on faster networks. And then you have pumps and dumps, you have prices going up, up, up, and then down, down, and down, and everything else you cannot imagine will actually happen. Unless of course you have the perfect logic for all conditions and situations, as well as super fast machines sitting in each exchange's data center, next to their api servers. If so, I envy you :)

If you are not a seasoned trader and seasoned developer for trading, you should expect to lose money before you can make money. And even if you start making money with your first exchange right away, chances are you will lose with the second one. If the fee we'll be required to pay for the Pro version is similar to what has been mentioned in previous posts, it is just a fraction of the initial losses. Trust me.

With all the above been said, I suggest that you start using CCXT as is (without websocket support), and when the Pro version is released, you'll just have to replace a few calls. If you find that you're reaching the limits of certain exchanges, then it is not websockets you need.

Just my 2c

[frodoe7]

With all the above been said, I suggest that you start using CCXT as is (without websocket support), and when the Pro version is released, you'll just have to replace a few calls. If you find that you're reaching the limits of certain exchanges, then it is not websockets you need.

there's already issue in my project , and it's mandatory to use websocket right now , I even can't wait for CCXT Pro to be released

[npomfret]

@lfern thanks for the Binance improvements, trying them out now - looks great.

[npomfret]

@lfern have you considered yet using an RBTree instead of an array for storing the data? I'm trying it out on my project and it's gone well. It's pretty simple to implement and I _assume_ more efficient that using an array (but haven't tested the theory). Happy to share my code in a public gist if you're interested.

[lfern]

@npomfret while waiting for official websocket implementation, if it is really easy, of course I will try to change it in websocket-multiple branch.

[npomfret]

@lfern take a look at this: https://gist.github.com/npomfret/a52fab1144090204425f984ea132e45e

Can't promise the RBTree is faster, but I suspect it is.

[auterium]

@npomfret that looks pretty interesting. We use TreeMap for each side of the book with a custom sorter. I'm not that familiar with RBTree, but I'm curious why would you use it over TreeMap? I can tell for experience that using the TreeMap was a much better option than arrays for a live orderbook during heavy price action

[npomfret]

Ah sorry, I thought you were still using arrays, haven't read the code in a while. I don't know how efficient TreeMap is I'm afraid. A red/black tree has a well understood performance profile (O log(n) for search and deletes), perhaps TreeMap is the same.

An RBTree is very simple, I've used a NPM class but you could write it yourself in 10 minutes I'm sure.

[edit] ha, it is the same - the performance is mentioned right at the bottom of the docs. So no point in switching.

[frosty00]

@lfern have you considered yet using an RBTree instead of an array for storing the data?

Why would you use this over using a regular hashmap indexed by prices which is O(1)? Storing and fetching data from a binary tree is at least log(N) asymptotically so it doesn't really make a lot of sense - https://docs.google.com/presentation/d/1FVENq6nVfWEHohE8j3oQC6uutxWOghacBfJixFV3KQU/edit#slide=id.g5240c8ba42_0_1460

[auterium]

@frosty00 I'm not expert in data structures (I lack formal education on it and must ashamely acknowledge I've never researched properly on it), so please correct me if I'm wrong.

For what I've seen online, hashmaps are typically better for high single-element insert/update/delete, whereas self-balancing trees have better performance for iterating over ranges. Based on this, I would say that opting for one or another would depend on your read use case (in ours we always fetch the top X after any insert/update/delete, therefore a RBTree makes more sense).

On the other hand, hashmaps would normally work based on bucketing by the hash, but this is prone to speed loss if there's high concentrations in a single bucket while other buckets have too few elements.

[firepol]

@lfern (or @frosty00 ?) long time no update from my side, I tried to transpile the actual state of lfern's websockets-multiple branch. I get many errors in python when I run ob_updater.py in my https://github.com/firepol/ccxt-websockets-db-updater project... I tried to fix some of them manually (see last commits in the dev branch of my other repo, where I usually commit the transpiled files: https://github.com/firepol/ccxt-websockets/tree/dev), to see if I get my program working, but in the end there is always one more syntax error that pops up. Here the syntax errors I got and how I fixed them:

File "\venv\lib\site-packagesccxt\kucoin.py", line 1623
lastSeqId++
^
SyntaxError: invalid syntax

Solution: search all "++" and replace with " += 1"

File "..,\venv\lib\site-packagesccxt\poloniex.py", line 1301
else:
^
IndentationError: expected an indented block

Solution: a comment is not enough for python, in python I add e.g. "pass", ideally don't use an empty elif or else block, but put a comment before or after... if you really need an empty block, declare a dummy variable such as foo = 'foo' or so...

File "...\venv\lib\site-packagesccxtasync_support\baseexchange.py", line 1059
response = await getattr(this_param, method)(*params)
^
SyntaxError: 'await' outside async function

Solution: no idea... you have to find a fix for this...

thank you

[teneon]

Guys please set a release date for CCXT pro and then release it at a specified date whatever you have, there will always be some bugs for sure from the beginning, but let's take care of this later on, we need to start implementing and experimenting, even if it is not perfect from the start! In case release of "ccxt pro" will move into 2020, please say so! We should then perhaps implement "ccxws" package in the meanwhile into our project. But if "ccxt pro" is around the corner i don't feel like implementing "ccxws" just for a week or two and then reimplement "ccxt pro". So please, it is important for us and probably everyone else, so we can plan ahead.

[kroitor]

Guys please set a release date for CCXT pro and then release it at a specified date whatever you have, there will always be some bugs for sure from the beginning, but let's take care of this later on, we need to start implementing and experimenting, even if it is not perfect from the start!

We totally agree with you and it's been our topmost priority since we announced it.

In case release of "ccxt pro" will move into 2020, please say so!

We are determined to not let it move into 2020 and now we are pretty sure that it won't. From here it may take 2 to 4 weeks to roll out the initial version if nothing else pops up. But, anyways, it will be released before the end of this year.

But if "ccxt pro" is around the corner i don't feel like implementing "ccxws" just for a week or two and then reimplement "ccxt pro".

It is around the corner and we are working very hard to deliver it as soon as we can, seriously. We can't tell the exact date, since we may be able to deliver sooner than that. However, it will be released immediately when ready. If you still want the exact date, assume "before the end of 2019".

[lfern]

@firepol I think I have fixed all python errors you reported in websockets-multiple branch., Please check it out

[WoK-E]

@kroitor Do you have any info on licensing and fee schedule that you can share before Pro is released?

[kroitor]

@WoK-E we have posted on this matter here:

• https://github.com/ccxt/ccxt/issues/56#issuecomment-508280756
• https://github.com/ccxt/ccxt/issues/56#issuecomment-527194069

Will update you on the pricing grid before the release as soon as we have it (our goal is to roll out a good and flexible tech design for the users, first and foremost).

[eckozen84]

@kroitor will CCXT pro have proper handling of diff orderbook messages that arrive out of sequence?

EDIT: Ignore this question. I just saw your post above. Thank you.

[npomfret]

@frodoe7 you can always implement web sockets yourself for one or two exchanges while you wait to see if paying for CCXT pro is worth paying for. I've been doing just that while I wait for it to be released and it's really not so difficult. I've got public (orderbooks), and private (my trades and orders) ws-feeds working for 6 exchanges and I can't say that any were particularly difficult to do.

[n01ukn0w]

Any word on when CCXT Pro will be released? Jan 1, 2020 is approaching quickly!

[kroitor]

@n01ukn0w we are now closer than ever to getting it finally released, and we will announce beta-testing shortly.

[lookis]

@n01ukn0w we are now closer than ever to getting it finally released, and we will announce beta-testing shortly.

So exciting, I am pay much attention on this issue for month, looking forward to try the Pro at the very first time.

[radiant-alex]

Would be nice to have snippets of usages for various JS frameworks, specially for Angular/Typescript.
If it can handle things like Nativescript, would be awesome! Though from my recent tests ws npm library does not play well with NS. Wondering what is used for streaming under the hood?

[KlausGlueckert]

@kroitor : so excited as well and thanks a lot for all the work!!! a few questions up front:

• will there be a Python wrapper from the begining?
• will ccxt pro handle order book sequences automatically ? (no manual connect, reconnect logic etc)
• will ccxt pro automatically handle current orderbooks in memory? some exchanges only provide REST API orderbook snapshots so you have to build your own logic to take the REST snapshots and update them with the websocket orderbook updates :(
• if you need any input or feedback, do let me know, i have quite recently integrated shrimpy.io, and it just doesnt solve all the problems a developer has, ccxt pro could be so much better than everything available atm

[kroitor]

@KlausGlueckert

will there be a Python wrapper from the begining?

Yep.

will ccxt pro handle order book sequences automatically ? (no manual connect, reconnect logic etc)

Yes, and the other types of data feeds as well.

will ccxt pro automatically handle current orderbooks in memory? some exchanges only provide REST API orderbook snapshots so you have to build your own logic to take the REST snapshots and update them with the websocket orderbook updates :(

Yes, that is also handled.

if you need any input or feedback, do let me know, i have quite recently integrated shrimpy.io, and it just doesnt solve all the problems a developer has, ccxt pro could be so much better than everything available atm

Thx a lot for your support, will let you know if anything!

[juso]

In case release of "ccxt pro" will move into 2020, please say so!

We are determined to not let it move into 2020 and now we are pretty sure that it won't. From here it may take 2 to 4 weeks to roll out the initial version if nothing else pops up. But, anyways, it will be released before the end of this year.

But if "ccxt pro" is around the corner i don't feel like implementing "ccxws" just for a week or two and then reimplement "ccxt pro".

It is around the corner and we are working very hard to deliver it as soon as we can, seriously. We can't tell the exact date, since we may be able to deliver sooner than that. However, it will be released immediately when ready. If you still want the exact date, assume "before the end of 2019".

@kroitor - how does the release look like now? I assume it not gonna happen before 2020, so what is the next target? Better keep ppl up-to-date even with moving target than having no updates at all - in this case one feels nothing gonna happen at all since your priorities might have changed to something else

[kroitor]

@juso the work on it is very actively moving to the release (it's a matter of days now). We already have a working transpileable repo, and we are going to invite beta-testers in the beginning of January. The lack of updates does not mean that our priorities have changed, it's the opposite – it is still our topmost priority and we're so occupied with it that we can't post updates too frequently, other than "almost there". Another aspect to this is that we want to release a stable working product. Our current progress is 85-90%. We will post more news and will disclose the pricing grid in the coming days.

[WoK-E]

@kroitor where/how will you be posting any news on CCXT Pro, and/or invitations to beta testers? Will it be in this thread?

[kroitor]

@WoK-E we will open a standalone announcement thread, and will post links here as well to notify everybody in this issue.

[kroitor]

Happy New Year to everyone! Just published a small update/intermediary roadmap for Q1 here: https://github.com/ccxt/ccxt/issues/6332. We are doing everything we can to roll it out this month.

[HoDaDor]

We have unified the balance API, as written here, in issue #36 . Now your feature request can be implemented easily ) We will add a dirty balance method that would save the balance on a market instance, first. And when we are ready to release the Websocket implementations, we will make it auto-updating, of course. Keep in touch! )

Was scrolling through after seeing the update about ccxt pro plans and couldn't help but notice this/your reply to someone asking about polling the api in intervals? And I'l just like to point out that Binance has a websocket endpoint for trade and account data. Just need to subscribe to the "userData" stream: https://github.com/binance-exchange/binance-official-api-docs/blob/master/user-data-stream.md

Also, in case anyone is interested, I've been using this for my binance websocket needs, hope you guys find it as helpful as I have: https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/

[kroitor]

@HoDaDor this is exactly what CCXT Pro is all about, using the WebSockets for streaming: https://github.com/ccxt/ccxt/issues/6332#issuecomment-586626802 (today)

[kroitor]

Hello, dear CCXT users!

Today we bring you the good news.

We have released CCXT Pro with support for WebSockets!
Please, see the announcement here: #6543

Thank you to all who have been waiting so patiently for it.
As always, your feedback is appreciated a lot!

Finally, we are closing this issue )

Yours, CCXT Dev Team