Zipline: Order 150 securities at same time. Get OverflowError Python int too large to convert to C long"

Created on 29 Jun 2018 · 11Comments · Source: quantopian/zipline

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

Operating System: (Linux 4.4.0-36-generic Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux)
Python Version: python --version
Python Bitness: 64
How did you install Zipline: pip
Python packages: absl-py==0.2.2
alembic==0.9.9
astor==0.6.2
attrs==15.2.0
Babel==1.3
backports.shutil-get-terminal-size==1.0.0
backports.weakref==1.0.post1
bcolz==0.12.1
bleach==1.5.0
Bottleneck==1.2.1
casadi==3.2.0
chardet==2.3.0
Cheetah==2.4.4
click==6.7
configobj==5.0.6
configparser==3.5.0
contextlib2==0.5.5
cryptography==1.2.3
cycler==0.10.0
cyordereddict==1.0.0
Cython==0.28.3
decorator==4.0.6
docutils==0.12
empyrical==0.4.3
entrypoints==0.2.2
enum34==1.1.2
functools32==3.2.3.post2
futures==3.2.0
gast==0.2.0
Genshi==0.7
gyp==0.1
html5lib==0.9999999
idna==2.0
intervaltree==2.1.0
ipaddress==1.0.16
ipykernel==4.6.1
ipython==5.4.1
ipython-genutils==0.2.0
ipywidgets==6.0.0
Jinja2==2.8
jsonpatch==1.10
jsonpointer==1.9
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.0.1
jupyter-console==5.1.0
jupyter-core==4.3.0
Landscape-Client==16.3+bzr834
Logbook==1.4.0
lru-dict==1.1.6
madopt==0.5
Mako==1.0.7
Markdown==2.6.11
MarkupSafe==0.23
matplotlib==2.0.2
mistune==0.7.4
multipledispatch==0.5.0
nbconvert==5.2.1
nbformat==4.3.0
ndg-httpsclient==0.4.0
networkx==1.11
notebook==5.0.0
numexpr==2.6.5
numpy==1.14.5
oauth==1.0.1
PAM==0.4.2
pandas==0.18.1
pandas-datareader==0.5.0
pandocfilters==1.4.1
pathlib2==2.2.1
patsy==0.5.0
pexpect==4.0.1
pickleshare==0.7.4
Pillow==3.1.2
ply==3.7
prettytable==0.7.2
prompt-toolkit==1.0.14
protobuf==3.6.0
ptyprocess==0.5
pyasn1==0.1.9
pyasn1-modules==0.0.7
pycrypto==2.6.1
pycurl==7.43.0
Pygments==2.1
PyMySQL==0.7.2
pyOpenSSL==0.15.1
pyparsing==2.2.0
pyserial==3.0.1
python-apt==1.1.0b1+ubuntu0.16.4.1
python-dateutil==2.6.0
python-debian==0.1.27
python-editor==1.0.3
pytz==2018.4
pyxdg==0.25
PyYAML==3.11
pyzmq==15.2.0
qtconsole==4.3.0
requests==2.9.1
requests-file==1.4.3
requests-ftp==0.3.1
roman==2.0.0
scandir==1.5
scipy==1.1.0
scour==0.32
service-identity==16.0.0
simplegeneric==0.8.1
six==1.10.0
sortedcontainers==2.0.4
SQLAlchemy==1.2.8
statsmodels==0.9.0
style==1.1.0
subprocess32==3.2.7
tables==3.4.3
tensorboard==1.8.0
termcolor==1.1.0
terminado==0.6
testpath==0.3.1
toolz==0.9.0
tornado==4.2.1
Trac==1.0.9
traitlets==4.3.2
Twisted==16.0.0
update==0.0.1
urllib3==1.13.1
virtualenv==15.0.1
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.14.1
widgetsnbextension==2.0.0
zipline==1.2.0
zope.interface==4.1.3

Now that you know a little about me, let me tell you about the issue I am
having:

Description of Issue

In a monthly schedule function, I order 150 securities using order_target_value the target I set is just 100$. Then I got an OverflowError message.

OverflowError: Python int too large to convert to C long", call = py_call_impl(callable, dots$args, dots$keywords)
`

| stop(structure(list(message = "OverflowError: Python int too large to convert to C long", call = py_call_impl(callable, dots$args, dots$keywords), cppstack = structure(list(file = "", line = -1L, stack = c("/home/mz3/R/x86_64-pc-linux-gnu-library/3.4/reticulate/libs/reticulate.so(Rcpp::exception::exception(char const*, bool)+0x82) [0x7fcea59d7802]", "/home/mz3/R/x86_64-pc-linux-gnu-library/3.4/reticulate/libs/reticulate.so(Rcpp::stop(std::__cxx11::basic_string, std::allocator > const&)+0x27) [0x7fcea59d7937]", ...
-- | --

Here is how you can reproduce this issue on your machine:

Reproduction Steps

Schedule a function monthly based,
Every month call order 150 different securities using order_target_value
Run algo set windows 3 years daily based.
...

I am looking into the source code. but does anyone know what is the limitations of the zipline
...

Sincerely,
Mo

Question

Source

zhaimac

All 11 comments

This looks like you are calling into some R library to do some calculation. Do you get the same error if remove all logic except for a simple order?

Are you using a custom slippage or commission model?

llllllllll on 29 Jun 2018

👍1

Thanks! I just tried. After I removed slippage and commission I still got this issue.

I do call some R library. but it is just for feeding data, after I removed these R code it sill give me same errors.

zhaimac on 29 Jun 2018

Can you post the full traceback? The one given shows R and C++ code which zipline does not contain.

llllllllll on 29 Jun 2018

Sure. Currently, We are using the R calling zipline. We do re-balance every month. with ranked scores.
This code works well for now. But if I updat the n_top to 200. I got the issue above.

rebalance <-function(context, data){

  tryCatch({
    message(paste0("rebalance at:", context$i ))

    n_top <- 100
    target_value <- 10000L

    tops<- universe[day==Days[context$i] & !is.na(COMPOSITE) & !startsWith(as.character(security),'@'), security][1:n_top]

    if(context$lastrb>0){
      last_tops<- universe[day==Days[context$lastrb] & !is.na(COMPOSITE) & !startsWith(as.character(security),'@'), security][1:n_top]

      no_change <- intersect(tops, last_tops)
      tops <- setdiff(tops, no_change)
      sell_list <- setdiff(last_tops, no_change)

      # sell not on the list 
      message('sell_list> ', sell_list)
      for(sell in sell_list){
        fly_order_target_value(asset = fly_symbol(sell), target = 0L)
      }
    }

    context$lastrb = context$i

    # buy on the list
    message('buy_list> ', tops)
    for(buy in tops){
      if(data$can_trade(fly_symbol(buy)) ){
        fly_order_target_value(fly_symbol(buy), target = target_value)
      }
    }
  }, error = function(e){
    message("error :", e)
  })

  message('context$lastrb=' , context$lastrb )
}

zhaimac on 29 Jun 2018

We don't support any R interface. The R interface you are using is a third party package, you may want to open an issue on that repo. What is likely happening is that some integer is growing larger than the size of a long on your system[1]. In Python, this wouldn't be an issue, but the code used to marshal Python data back to R is failing to account for arbitrarily sized Python ints. The code is helpful, but I would also need the full traceback of the error, though no guarantees that I can help.

gcc -x c <(printf '#include<stdio.h>\nint main(){printf("%%d\\n", sizeof(long));return 0;}') -o /tmp/sizeoflong && /tmp/sizeoflong && rm -f /tmp/sizeoflong

If that prints 4, this error seems basically unavoidable.

llllllllll on 29 Jun 2018

👍1

@llllllllll Hi Thank you very much for this info. I just tried on gcc code you sent to me. it give me back

8.
Here is the full traceback of the error, btw.

Error in py_call_impl(callable, dots$args, dots$keywords) : OverflowError: Python int too large to convert to C long
15.
stop(structure(list(message = "OverflowError: Python int too large to convert to C long", call = py_call_impl(callable, dots$args, dots$keywords), cppstack = structure(list(file = "", line = -1L, stack = c("/home/mz3/R/x86_64-pc-linux-gnu-library/3.4/reticulate/libs/reticulate.so(Rcpp::exception::exception(char const*, bool)+0x82) [0x7fcea59d7802]", "/home/mz3/R/x86_64-pc-linux-gnu-library/3.4/reticulate/libs/reticulate.so(Rcpp::stop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x27) [0x7fcea59d7937]", ...
14.
zipline at _protocol.pxd#14
13.
__setattr__ at position.py#68
12.
handle_split at position.py#120
11.
handle_splits at ledger.py#136
10.
process_splits at ledger.py#533
9.
handle_splits at tracker.py#180
8.
once_a_day at tradesimulation.py#174
7.
transform at tradesimulation.py#209
6.
run at algorithm.py#730
5.
_run at run_algo.py#208
4.
run_algorithm at run_algo.py#398
3.
py_run at <string>#50
2.
main$py_run(fly_initialize = initialize, fly_handle_data = handle_data, start = start, end = end, capital_base = capital_base, before_trading_start = before_trading_start, analyze = analyze, data_frequency = data_frequency, data = data, bundle = bundle, bundle_timestamp = bundle_timestamp, trading_calendar = trading_calendar, ...
1.
fly_run_algorithm(initialize = fly_initialize, before_trading_start = fly_before_trading_start, handle_data = fly_handle_data, analyze = fly_analyze, bundle = "mdo_csv4", capital_base = 1e+06, trading_calendar = calendar, data_frequency = "daily", start = start_date, end = end_date)

I will try to migrate our code to pure python later and check if it still an issue. Let me work on it .

zhaimac on 29 Jun 2018

just a small formatting comment, when you want to monospace format multiple lines, like the traceback you posted, you need to use three backtick characters, not just one. I edited your comment to do that.

llllllllll on 29 Jun 2018

Given this traceback it looks like the overflow is actually in Zipline's python->cython boundary. This appears to be in update the position size after a split. What is likely happening is that you are ordering a lot of shares of one of the assets and then there is a massive split causing the share count to grow very rapidly and exceed 2 ** 63 - 1 shares. Some things to check: