Zipline: Type Error on ingesting quandl default bundle

[sekuri]

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: (Windows Version or $ uname --all)

  • Darwin MY.home 16.4.0 Darwin Kernel Version 16.4.0: Thu Dec 22 22:53:21 PST 2016; root:xnu-3789.41.3~3/RELEASE_X86_64 x86_64 i386 MacBookAir4,1 Darwin

• Python Version: $ python --version

  • Python 3.4.6

• Python Bitness: $ python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))'

  • 64

• How did you install Zipline: (pip, conda, or other (please explain))

  • pip

• Python packages: $ pip freeze or $ conda list

  • alembic==0.9.2
  • bcolz==0.12.1
  • Bottleneck==1.2.1
  • certifi==2017.4.17
  • chardet==3.0.4
  • click==6.7
  • contextlib2==0.5.5
  • cyordereddict==1.0.0
  • Cython==0.25.2
  • decorator==4.0.11
  • empyrical==0.2.2
  • idna==2.5
  • intervaltree==2.1.0
  • Logbook==1.0.0
  • lru-dict==1.1.6
  • Mako==1.0.6
  • MarkupSafe==1.0
  • multipledispatch==0.4.9
  • networkx==1.11
  • numexpr==2.6.2
  • numpy==1.13.0
  • pandas==0.18.1
  • pandas-datareader==0.4.0
  • patsy==0.4.1
  • python-dateutil==2.6.0
  • python-editor==1.0.3
  • pytz==2017.2
  • requests==2.18.1
  • requests-file==1.4.2
  • requests-ftp==0.3.1
  • scipy==0.19.0
  • six==1.10.0
  • sortedcontainers==1.5.7
  • SQLAlchemy==1.1.10
  • statsmodels==0.8.0
  • tables==3.4.2
  • toolz==0.8.2
  • urllib3==1.21.1
  • zipline==1.1.0

Now that you know a little about me, let me tell you about the issue I am
having:

Description of Issue

Type error on trying to ingest quandl bundle for first time on clean virtual env install

• What did you expect to happen?
  
  
  
  • Expected the ingestion to work seamless exactly like it did for the quantopian-quandl bundle.
  
  
• What happened instead?
  
  
  
  • Got the following error:
    
    `$ QUANDL_API_KEY="" zipline ingest -b quandl
  
  

Downloading WIKI metadata: [####################################]

Traceback (most recent call last):

File "/Users/my/Documents/Python/Virtualenvs/venv-py34-zipline-1.1.0/bin/zipline", line 11, in load_entry_point('zipline==1.1.0', 'console_scripts', 'zipline')()

File "/Users/my/Documents/Python/Virtualenvs/venv-py34-zipline-1.1.0/lib/python3.4/site-packages/click/core.py", line 722, in __call__ return self.main(args, *kwargs)

File "/Users/my/Documents/Python/Virtualenvs/venv-py34-zipline-1.1.0/lib/python3.4/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)

File "/Users/my/Documents/Python/Virtualenvs/venv-py34-zipline-1.1.0/lib/python3.4/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))

File "/Users/my/Documents/Python/Virtualenvs/venv-py34-zipline-1.1.0/lib/python3.4/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)

File "/Users/my/Documents/Python/Virtualenvs/venv-py34-zipline-1.1.0/lib/python3.4/site-packages/click/core.py", line 535, in invoke
return callback(args, *kwargs)

File "/Users/my/Documents/Python/Virtualenvs/venv-py34-zipline-1.1.0/lib/python3.4/site-packages/zipline/__main__.py", line 312, in ingest
show_progress,

File "/Users/my/Documents/Python/Virtualenvs/venv-py34-zipline-1.1.0/lib/python3.4/site-packages/zipline/data/bundles/core.py", line 451, in ingest
pth.data_path([name, timestr], environ=environ),

File "/Users/my/Documents/Python/Virtualenvs/venv-py34-zipline-1.1.0/lib/python3.4/site-packages/zipline/data/bundles/quandl.py", line 295, in quandl_bundle
show_progress=show_progress,

File "/Users/my/Documents/Python/Virtualenvs/venv-py34-zipline-1.1.0/lib/python3.4/site-packages/zipline/data/bundles/quandl.py", line 129, in fetch_symbol_metadata_frame
data['auto_close_date'] = data['end_date'] + pd.Timedelta(days=1)

File "/Users/my/Documents/Python/Virtualenvs/venv-py34-zipline-1.1.0/lib/python3.4/site-packages/pandas/core/ops.py", line 609, in wrapper
na_op)

File "/Users/my/Documents/Python/Virtualenvs/venv-py34-zipline-1.1.0/lib/python3.4/site-packages/pandas/core/ops.py", line 569, in maybe_convert_for_time_op
return cls(left, right, name, na_op)

File "/Users/my/Documents/Python/Virtualenvs/venv-py34-zipline-1.1.0/lib/python3.4/site-packages/pandas/core/ops.py", line 282, in __init__
lvalues = self._convert_to_array(left, name=name)

File "/Users/my/Documents/Python/Virtualenvs/venv-py34-zipline-1.1.0/lib/python3.4/site-packages/pandas/core/ops.py", line 398, in _convert_to_array
if (inferred_type in ('datetime64', 'datetime', 'date', 'time') or

TypeError: data type "datetime" not understood
`

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1. Fresh zipline install in virtual Python envirtonment (actually doesn't matter if it's a fresh install or not. This error is still thrown)
2. Run in terminal: $ QUANDL_API_KEY="<MY_QUANDL_API_KEY>" zipline ingest -b quandl

What steps have you taken to resolve this already?

Used the alternative default bundle quantopian-quandl

Sincerely,
Sekuri

[freddiev4]

x-ref https://github.com/quantopian/zipline/issues/1295

Hey @sekuri looks like this has been an issue. Ideally, the next steps are to make zipline/data/bundles/quandl.py use the latest Quandl API so that data is able to be requested properly.

[ebarcikowski]

I added this to the bottom of fetch_symbol_metadata_frame in quandl.py. Seems to work.

index = pd.DatetimeIndex(data['end_date'])
data['auto_close_date'] = index + pd.tslib.Timedelta(days=1)

Basically, you can add time offsets to a DatetimeIndex, but columns don't support that. At least, with pandas 0.18. I think it's supported in later versions though.

Elliott

[freddiev4]

We're using the latest Quandl API in our most recent release (1.2.0). Going to close this but feel free to re-open if you're still having issues or open a new issue 🙂