Zipline: Ingesting quantopian-quandl bundle populates assets-5.sqlite but not assets-3.sqlite

Created on 29 Aug 2016  路  8Comments  路  Source: quantopian/zipline

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

  • Operating System: Darwin 15.6.0 Darwin Kernel Version 15.6.0: Thu Jun 23 18:25:34 PDT 2016; root:xnu-3248.60.10~1/RELEASE_X86_64 x86_64
  • Python Version: Python 3.4.5 :: Anaconda custom (x86_64)
  • Python Bitness: 64
  • How did you install Zipline: (conda also tried pip)
  • Python packages: ```
    # packages in environment at /Users/yixin/anaconda3/envs/py34:
    #
    _nb_ext_conf 0.2.0 py34_0
    alabaster 0.7.8 py34_0
    anaconda-client 1.4.0 py34_0
    anaconda custom py34_0
    anaconda-navigator 1.2.1 py34_0
    appnope 0.1.0 py34_0
    appscript 1.0.1 py34_0
    argcomplete 1.0.0 py34_1
    astropy 1.0.4 np19py34_0
    babel 2.3.3 py34_0
    backports 1.0 py34_0
    backports_abc 0.4 py34_0
    beautifulsoup4 4.4.1 py34_0
    bitarray 0.8.1 py34_0
    blaze 0.10.1 py34_0
    bokeh 0.12.0 py34_0
    boto 2.40.0 py34_0
    bottleneck 1.0.0 np19py34_0
    cffi 1.6.0 py34_0
    chest 0.2.3 py34_0
    click 6.6 py34_0
    cloudpickle 0.2.1 py34_0
    clyent 1.2.2 py34_0
    colorama 0.3.7 py34_0
    configobj 5.0.6 py34_0
    contextlib2 0.5.3 py34_0
    cryptography 1.4 py34_0
    curl 7.49.0 0
    cycler 0.10.0 py34_0
    cython 0.24 py34_0
    cytoolz 0.8.0 py34_0
    dask 0.10.0 py34_0
    datashape 0.5.2 py34_0
    decorator 4.0.10 py34_0
    dill 0.2.5 py34_0
    docutils 0.12 py34_2
    entrypoints 0.2.2 py34_0
    et_xmlfile 1.0.1 py34_0
    fastcache 1.0.2 py34_1
    flask 0.11.1 py34_0
    flask-cors 2.1.2 py34_0
    freetype 2.5.5 1
    get_terminal_size 1.0.0 py34_0
    gevent 1.1.1 py34_0
    greenlet 0.4.10 py34_0
    h5py 2.5.0 np19py34_3
    hdf5 1.8.15.1 2
    heapdict 1.0.0 py34_1
    idna 2.1 py34_0
    imagesize 0.7.1 py34_0
    ipykernel 4.3.1 py34_0
    ipython 4.2.0 py34_1
    ipython_genutils 0.1.0 py34_0
    ipywidgets 4.1.1 py34_0
    itsdangerous 0.24 py34_0
    jbig 2.1 0
    jdcal 1.2 py34_1
    jedi 0.9.0 py34_1
    jinja2 2.8 py34_1
    jpeg 8d 1
    jsonschema 2.5.1 py34_0
    jupyter 1.0.0 py34_3
    jupyter_client 4.3.0 py34_0
    jupyter_console 4.1.1 py34_0
    jupyter_core 4.1.0 py34_0
    libpng 1.6.17 0
    libtiff 4.0.6 2
    libxml2 2.9.2 0
    libxslt 1.1.28 2
    llvmlite 0.11.0 py34_0
    locket 0.2.0 py34_1
    lxml 3.6.0 py34_0
    markupsafe 0.23 py34_2
    matplotlib 1.4.3 np19py34_3
    mistune 0.7.2 py34_1
    mkl 11.3.3 0
    mkl-service 1.1.2 py34_2
    mpmath 0.19 py34_1
    multipledispatch 0.4.8 py34_0
    nb_anacondacloud 1.1.0 py34_0
    nb_conda 1.1.0 py34_0
    nb_conda_kernels 1.0.3 py34_0
    nbconvert 4.2.0 py34_0
    nbformat 4.0.1 py34_0
    nbpresent 3.0.2 py34_0
    networkx 1.11 py34_0
    nltk 3.2.1 py34_0
    nose 1.3.7 py34_1
    notebook 4.2.1 py34_0
    numba 0.26.0 np19py34_0
    numpy 1.9.3 py34_3
    odo 0.5.0 py34_1
    openpyxl 2.3.2 py34_0
    openssl 1.0.2h 1
    pandas 0.16.2 np19py34_0
    pandas-datareader 0.2.1 py34_0
    partd 0.3.4 py34_0
    path.py 8.2.1 py34_0
    pathlib2 2.1.0 py34_0
    patsy 0.4.1 py34_0
    pep8 1.7.0 py34_0
    pexpect 4.0.1 py34_0
    pickleshare 0.7.2 py34_0
    pillow 3.2.0 py34_1
    pip 8.1.2 py34_0
    ply 3.8 py34_0
    psutil 4.3.0 py34_0
    ptyprocess 0.5.1 py34_0
    py 1.4.31 py34_0
    pyasn1 0.1.9 py34_0
    pycosat 0.6.1 py34_1
    pycparser 2.14 py34_1
    pycrypto 2.6.1 py34_4
    pycurl 7.43.0 py34_0
    pyflakes 1.2.3 py34_0
    pygments 2.1.3 py34_0
    pyopenssl 0.16.0 py34_0
    pyparsing 2.0.3 py34_0
    pyqt 4.11.4 py34_3
    pytables 3.2.2 np19py34_0
    pytest 2.9.2 py34_0
    python 3.4.5 0
    python-dateutil 2.5.3 py34_0
    python.app 1.2 py34_4
    pytz 2016.4 py34_0
    pyyaml 3.11 py34_4
    pyzmq 15.2.0 py34_1
    qt 4.8.7 3
    qtconsole 4.2.1 py34_0
    qtpy 1.0.2 py34_0
    bcolz 0.12.1 np19py34_0 Quantopian
    cachetools 1.1.6 py34_0 Quantopian
    cyordereddict 0.2.2 py34_0 Quantopian
    intervaltree 2.1.0 py34_0 Quantopian
    logbook 0.12.5 py34_0 Quantopian
    numexpr 2.4.6 np19py34_0 Quantopian
    sortedcontainers 1.4.4 py34_0 Quantopian
    zipline 1.0.1 np19py34_3_g8f0b3f4 Quantopian
    readline 6.2 2
    redis 3.2.0 0
    redis-py 2.10.5 py34_0
    requests 2.10.0 py34_0
    requests-file 1.4 py34_0
    rope 0.9.4 py34_1
    ruamel_yaml 0.11.7 py34_0
    scikit-image 0.11.3 np19py34_0
    scikit-learn 0.16.1 np19py34_0
    scipy 0.17.1 np19py34_1
    setuptools 23.0.0 py34_0
    simplegeneric 0.8.1 py34_1
    singledispatch 3.4.0.3 py34_0
    sip 4.16.9 py34_0
    six 1.10.0 py34_0
    snowballstemmer 1.2.1 py34_0
    sockjs-tornado 1.0.3 py34_0
    sphinx 1.4.1 py34_0
    sphinx_rtd_theme 0.1.9 py34_0
    spyder 2.3.9 py34_0
    sqlalchemy 1.0.13 py34_0
    sqlite 3.13.0 0
    statsmodels 0.6.1 np19py34_0
    sympy 1.0 py34_0
    terminado 0.6 py34_0
    tk 8.5.18 0
    toolz 0.8.0 py34_0
    tornado 4.3 py34_1
    traitlets 4.2.1 py34_0
    unicodecsv 0.14.1 py34_0
    werkzeug 0.11.10 py34_0
    wheel 0.29.0 py34_0
    xlrd 1.0.0 py34_0
    xlsxwriter 0.9.2 py34_0
    xlwings 0.7.2 py34_0
    xlwt 1.1.2 py34_0
    xz 5.2.2 0
    yaml 0.1.6 0
    zlib 1.2.8 3 

Now that you know a little about me, let me tell you about the issue I am
having:

# Description of Issue

* Tried to ingest quantopian-quandl dataset, and run the starting out buy_appl.py algo but it failed...
* `$ zipline ingest -b quantopian-quandl` results in `assets-5.sqlite` being populated in `~/.zipline/data/quantopian-quandl/2016-08-29T15;57;26.244897` but not `assets-3.sqlite`
* Result of running `$ zipline run -b quantopian-quandl --algofile ./buyapple.py --start 2010-1-1 --end 2014-1-1 -o buyapple_out.pickle` - `sqlalchemy.exc.InvalidRequestError: Could not reflect: requested table(s) not available in sqlite:////Users/yixin/.zipline/data/quantopian-quandl/2016-08-29T15;57;26.244897/assets-3.sqlite: (version_info, equities, futures_root_symbols, futures_contracts, futures_exchanges, asset_router)`
* Replacing `assets-3.sqlite` with content from `assets-5.sqlite` does not work as it spits out version difference error...
* To note - I ingested yahoo and quandl data, it works fine. This only applies to quantopian-quandl

Here is how you can reproduce this issue on your machine:

## Reproduction Steps

1. `$ zipline ingest quantopian-quandl`
2. `$ zipline run -b quantopian-quandl --algofile ./buyapple.py --start 2010-1-1 --end 2014-1-1 -o buyapple_out.pickle`
3. `sqlalchemy.exc.InvalidRequestError: Could not reflect: requested table(s) not available in sqlite:////Users/yixin/.zipline/data/quantopian-quandl/2016-08-29T15;57;26.244897/assets-3.sqlite: (version_info, equities, futures_root_symbols, futures_contracts, futures_exchanges, asset_router)`
...

## What steps have you taken to resolve this already?

Instead of using quantopian-quandl I used yahoo and quandl data, works fine. But I still want to resolve this issue because it blocks lot of people who want to peek into zipline but can't due to outdated documentation...

...

# Anything else?

No, but thank you guys for looking into it. Great work!

...

Sincerely,
Yixin Xia
Bug Data Bundle
Source
picture wongasta

Most helpful comment

Thanks @wongasta . We unintentionally started pushing a more recent version. Fixing it now...

picture richafrank on 29 Aug 2016
👍3

All 8 comments

Thanks @wongasta . We unintentionally started pushing a more recent version. Fixing it now...

richafrank picture richafrank on 29 Aug 2016
👍3

Thanks man! Appreciate all the great work!!!

wongasta picture wongasta on 30 Aug 2016

Should be fixed now. Does a new ingest work for you?

richafrank picture richafrank on 30 Aug 2016

zipline 1.0.2 should fix this now. Definitely re-open if that's not the case.

richafrank picture richafrank on 9 Sep 2016

@richafrank I'm running into a similar issue, although with zipline 1.0.2. I just installed it (conda) and ran zipline ingest.

sqlalchemy.exc.InvalidRequestError: Could not reflect: requested table(s) not available in sqlite:////Users/paul/.zipline/data/quantopian-quandl/2016-12-18T15;39;05.317714/assets-5.sqlite: (asset_router, futures_root_symbols, version_info, equity_symbol_mappings, equities, futures_contracts, futures_exchanges)

Here's the content of my ~/.zipline/data/quantopian-quandl/2016-12-18T15\;47\;58.666481/ data dir:

adjustments.sqlite
assets-3.sqlite
assets-6.sqlite
daily_equities.bcolz
minute_equities.bcolz

It looks like assets-5.sqlite wasn't ingested or ASSET_DB_VERSION isn't set correctly (?).

Thanks!

pgeez picture pgeez on 18 Dec 2016

Ah thanks - I'll bump the published bundle to include assets-5...

richafrank picture richafrank on 19 Dec 2016

FWIW We want to remove these manual steps but haven't gotten to it.

richafrank picture richafrank on 19 Dec 2016

any update on fixing sqlalchemy.exc.InvalidRequestError?

sqlalchemy.exc.InvalidRequestError: Could not reflect: requested table(s) not available in Engine(sqlite:////home/wenchen/.zipline/data/quantopian-quandl/2017-02-28T19;35;45.731723/assets-6.sqlite): (equity_supplementary_mappings, futures_root_symbols, version_info, equity_symbol_mappings, equities, futures_contracts, futures_exchanges, asset_router)

I'm running the buyapple example and after the suggestion from https://github.com/quantopian/zipline/issues/1316, I still got the same error on ubuntu 16.04.

zipline was install initially with pip but later from sudo -H pip install -e git+https://github.com/quantopian/zipline@master#egg=zipline_master

do removed ~/.zipline but still the same issue.

Please help,
Thanks

WenchenLi picture WenchenLi on 28 Feb 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

michaeljohnbennett picture michaeljohnbennett  路  4Comments
RJUNS picture RJUNS  路  8Comments
ishandutta2007 picture ishandutta2007  路  9Comments
richdevboston picture richdevboston  路  4Comments
suitablyquantified picture suitablyquantified  路  6Comments