Ccxt: For-loops are translated incorrectly for Python

[orkenstein]

When I run npm run build, the piece of JS, like:

        for (var i = 0; i < rawOrders.length; i++) {
          let order = rawOrders[i];
          result.push (this.parseOrder (order, market));
        }

translated into Python code:

        for(var i = 0 i < len(rawOrders) i++) {
          order = rawOrders[i]
          result.append(self.parse_order(order, market))
        }

which is wrong and annoying 👎

[kroitor]

So, your code does not get transpiled, because you violated the rules from here: https://github.com/ccxt/ccxt/blob/master/CONTRIBUTING.md#derived-exchange-classes

Look how other for-loops are formatted and copy that, if you want to make them transpileable.

Example:
https://github.com/ccxt/ccxt/blob/0df8e1be879e2bb78d2b250d45b085f9730532d5/js/kraken.js#L490

[orkenstein]

@kroitor thanks for quick reply.
Don't quite understand, what's wrong?
My for-loop looks the same to the example you shared. Please, clarify.

[kroitor]

My for-loop looks the same to the example you shared.

Yep it does look the same, but there's still a few differences, in your example the padding is 2 spaces, whereas we use 4, you also used the var keyword instead of let and the context of that forloop can also make a difference.

I added an example to clarify on this, in a separate branch:

https://github.com/ccxt/ccxt/blob/c6e6139918cfe73d337fc0ffbef2bf7348beef82/js/kraken.js#L494

That method gets transpiled successfully: https://travis-ci.org/ccxt/ccxt/builds/308369044#L900

So if your method does not get transpiled, break it down into pieces to see which of them is incorrect. Or follow the style literally (this may not be very intuitive, but still, current transpiler is our best option for now, we don't have a better tool for that job yet, however, we are working on it).

[orkenstein]

@kroitor yep, thanks, found and issue right before you posted a comment. Cheers.
P.S: working on some Livecoin updates.

[kroitor]

@orkenstein we welcome all PRs and contributions! Feel free to ask questions, if you have more. Thx!

[orkenstein]

@kroitor short question:
What is the best portable approach for string.includes(substring)?

[kroitor]

@orkenstein currenty, it's the long version of it:

if (string.indexOf (substring) >= 0) {
}

This is because includes can be used on arrays in JS in an interchangeable fashion, whereas in other languages the methods for lists and strings are different.

I know it can be frustrating sometimes, but until we have a full-featured AST any ←→ any converter, we have to be syntactically precise. Feel free to ask if you have any difficulty with it. Thx!

[orkenstein]

@kroitor
How to deal with timeout exceptions?

Traceback (most recent call last):
  File "/Users/milo/Development/ccxt/python/ccxt/base/exchange.py", line 309, in fetch
    response = opener.open(request, timeout=int(self.timeout / 1000))
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1321, in do_open
    r = h.getresponse()
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "Bot.py", line 104, in <module>
    print(api.fetch_orders())
  File "/Users/milo/Development/ccxt/python/ccxt/livecoin.py", line 260, in fetch_orders
    response = self.privateGetExchangeClientOrders(self.extend(request, params))
  File "/Users/milo/Development/ccxt/python/ccxt/livecoin.py", line 321, in request
    response = self.fetch2(path, api, method, params, headers, body)
  File "/Users/milo/Development/ccxt/python/ccxt/base/exchange.py", line 268, in fetch2
    return self.fetch(request['url'], request['method'], request['headers'], request['body'])
  File "/Users/milo/Development/ccxt/python/ccxt/base/exchange.py", line 315, in fetch
    raise RequestTimeout(' '.join([self.id, method, url, 'request timeout']))
ccxt.base.errors.RequestTimeout: livecoin GET https://api.livecoin.net/exchange/client_orders request timeout

[kroitor]

@orkenstein well, in general, catch & retry is the only way we can handle those... A proxy might also help determine if it's a local issue or a global one.

[kroitor]

... but we should not do any handling of those inside the library, of course, we just throw them out and let the user decide if he wants to retry the attempt or not.

[orkenstein]

@kroitor that's because the request time might be quite long for account with long history. The ruby version of the same request works fine. What about changing time-out delay?

[kroitor]

@orkenstein oh, then you can raise the timeout (defaults to 10000 ms), just do exchange.timeout = 20000 or pass 'timeout': 20000 along with other constructor params upon instantiation.

[orkenstein]

@kroitor how Date and Time should be handled?

[kroitor]

@orkenstein those should be handled like milliseconds timestamps in UTC. There's a set of functions:

this.parse8601 (string)  // parse ISO8601 string to milliseconds-timestamp
this.iso8601 (timestamp) // convert a milliseconds-timestamp to ISO8601 string
this.microseconds ()     // get current timestamps in microseconds
this.milliseconds ()     // ...
this.seconds ()          // ...

[orkenstein]

@kroitor what's the crossplatform type conversion method. Say float -> int.
Sorry, for asking too much. Why don't add this stuff to contribution guide?

[kroitor]

@orkenstein

what's the crossplatform type conversion method. Say float -> int.

parseInt (floatNumberOrString)
parseFloat (StringOrInt)
...

Sorry, for asking too much.

No worries at all, feel free to ask questions, even if you think they might not be worth it...

Why don't add this stuff to contribution guide?

Would love to, but didn't have enough time for it just yet, that is why it's a little outdated... I guess an answer to that sort of questions would always be "we never said we don't want to do that, we just didn't have time for that yet". In other words, if you see something missing, it's most likely in progress. Your help on this would be appreciated a lot! I mean, the contribution guide is editable, if you think we should add something there, feel free to submit the edits, I'll merge them all asap.

[kroitor]

@orkenstein parseInt and parseFloat are built into JS and they get transpiled to int(a) and float(b) in Python.

[orkenstein]

@kroitor, I see maker and taker fields here and there. Any special place for fees info? Like in market structure?

[kroitor]

@orkenstein those fields end up being on the market structure, exactly.

[orkenstein]

@kroitor, what am I missing? https://github.com/ccxt/ccxt/wiki/Manual#market-structure

[orkenstein]

@kroitor, aha, the info field. It's bit confusing here:

'info':      { ... }, // the original unparsed market info from the exchange

and:

• info. An associative array of non-common market properties, including fees, rates, limits and other general market information. The internal info array is different for each particular market, its contents depend on the exchange.

I think it's better to keep info the same original JSON for all the entities, and add special fee fields to market structure.

[kroitor]

@orkenstein info indeed contains the unparsed untouched fields "as is", including the fees and other data that comes for each particular market directly from the exchange. We don't touch it there. Most of the time you don't need to use market['info'] or look into it at all (for fees or whatever), because it's a raw unparsed structure.

what am I missing?

You're missing the fact that the parsed taker and maker fields were added a few days ago and have not been documented yet (don't ask why ;)). But they are there on each market. Don't confuse the market structure itself with the info inside it. Most of the time you need market['taker'] and market['maker'] if you want to get the trading fees for a pair, because those are uniformly parsed floats.

I think it's better to keep 'infothe same original JSON for all the entities, and add specialfee` fields to market structure.

This is exactly what we do. taker and maker should be there on each market (mostly).

[orkenstein]

@kroitor, so maker is a fee to sell, and taker to buy? Both are percents, e.g. 0.18% == 0.0018?

[kroitor]

@orkenstein

so maker is a fee to sell, and taker to buy?

Oh, no. The maker-taker is a different logic, does not relate to buy or sell.

The maker fee is payed when your order adds liquidity to the market. So if you place an order and it does not match another existing order in the orderbook, your order will be placed there and will contribute with its volume "adding to/making overall market liquidity". When someone else places an order that matches your existing maker, a fill-trade is generated and you pay the maker fee for the traded amount.

On the other hand, if you place an order that does get matched to an existing order in the orderbook, the fill-trade is generated immediately. And you are kinda "taking from overall market liquidity", by removing someone's maker offered order volume from the orderbook with your taker order.

So it does not matter whether you buy or sell, but it matters whether you make the market, or take from it. Market-makers usually pay much less in fees, some exchanges, like HitBTC, even pay rebates to market makers (they earn a part of the taker-side fee).

[orkenstein]

@kroitor, wow, never seen it before :)
What exchange work like this? On Livecoin it's dead simple: you just pay percent for every order.

[kroitor]

@orkenstein , well, most of them... Kraken, GDAX, HitBTC, Binance, Liqui, Poloniex, BitMEX, Bitfinex, OK*... I'd say 80% of exchanges implement maker/taker logic, and it's a very common thing. Half of the actually trading bots are maker bots (due to fees). So, in fact, this is a very important concept that can make a difference between a win or lose.

Some exchanges have maker fee = taker fee, like Livecoin, but in most cases, maker fee is much less than taker or it may even be a negative value (meaning that if you're a maker, you don't pay the fee, but earn the fee).

[kroitor]

@orkenstein https://www.investopedia.com/articles/active-trading/042414/what-makertaker-fees-mean-you.asp

[kroitor]

[kroitor]

Here's an example of a negative maker fee (a rebate/reward) from HitBTC (note that takers still pay 0.1% of fees, while makers earn 0.01% of fees):

[kroitor]

Also, explained in more detail here: https://github.com/ccxt/ccxt/issues/336#issuecomment-338380765

[beevabeeva]

Also here:
https://www.youtube.com/watch?v=u9RIA1uUbAs&index=2&list=PLj-hivYUpPZsHeLWy7EK7hOHtLQBKE05F
and here:
https://www.youtube.com/watch?v=7kkohJKFSLk&index=1&list=PLj-hivYUpPZsHeLWy7EK7hOHtLQBKE05F

On Thu, Nov 30, 2017 at 1:32 PM, Igor Kroitor [e-mail]
wrote:

Also, explained in somewhat more detail here: #336 (comment)
https://github.com/ccxt/ccxt/issues/336#issuecomment-338380765

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ccxt/ccxt/issues/694#issuecomment-348162733, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ASb-CYGTBhmv1XXUbEhDVmsvgUP6qHYCks5s7pJlgaJpZM4Qsa6S
.

[orkenstein]

@kroitor, soooo, there's even a reward for making the market? Wow...
Thanks for the clarification @kroitor, @beevabeeva.

[kroitor]

@orkenstein a difference of maker and taker can really make up to 50% of your profit or even more in a serious trading operation. Say, if your trading profit is $10k and the fee difference adds up to $5k, I guess, you will be interested in learning everything about it.

[orkenstein]

@kroitor, fees are in range of 0.01% -0.1% usually, not sure about such a dramatic difference... 🤔

[kroitor]

@orkenstein if your profit is 0.5% and the fee is 0.1%, that is a loss of 20% of your profit, don't know if it doesn't make a difference to you, but it usually does for most of the traders ;) Many exchanges have a fee of up to 0.25-0.8%, btw.

[orkenstein]

@kroitor don't trade that much yet :)

[kroitor]

@orkenstein the point is: you will never trade that much if you don't account for the above. So, if you wish to trade that much some day, you have to consider this very carefully.

[orkenstein]

@kroitor, got a question about market limits:

• limits. The minimums and maximums for prices, amounts (volumes) and costs (where cost = price * amount).

What is exactly, amount, price and cost?

[kroitor]

• amount is what you submit as amount argument to createOrder, so, it's the ordered amount (the total volume of an order in base currency)
• price is the price of a limit order in question, or the market price for the symbol in question
• cost is the product of the above (the total volume of the order in quote currency)

[orkenstein]

@kroitor, so we have createOrder(). How it should work for successful/failed orders?
Throw an exception in case of fail?

[kroitor]

@orkenstein yep, it should throw an exception. We also have handleErrors() to catch the errors before the standard default HTTP error handler, you might want to put all checks into handleErrors(), see other existing implementations for examples. Most of the time it does throw an exception of a specific type (if recognized, like InsufficientFunds or InvalidOrder), or a generic exception (like DDoSProtection, NetworkError or ExchangeError).

More on error handling here: https://github.com/ccxt/ccxt/wiki/Manual#error-handling

[orkenstein]

@kroitor, what should be included into PR?
Not sure about ccxt.browser.js

[kroitor]

@orkenstein most of the files are generated from JS, so we only edit base classes in JS/Python/PHP + derived exchange classes in JS. All derived exchanges are transpiled from JS to other languages, including the browser version. But base classes are not transpiled.

In other words:

• Base classes are language-specific
• Derived classes are language-agnostic

You don't need to include ccxt.browser.js, you only need to include:

• js/*
• python/base/*
• python/async/base/*
• php/base/*

[orkenstein]

@kroitor Made a PR https://github.com/ccxt/ccxt/pull/775

[kroitor]

@orkenstein thanks! Reviewing it now.

[orkenstein]

@kroitor, tried to add better exception handling: https://github.com/ccxt/ccxt/pull/837

[orkenstein]

@kroitor, returning to maker/taker fees again: I can see fetch_fees and calculate_fee for different exchanges. So, who's the winner?