Zipline: backtest fails trying to download SPY data for benchmark with quandl

Created on 27 Jun 2019 · 13Comments · Source: quantopian/zipline

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

Operating System: MacOS Sierra 10.12.6
Python Version: 3.5.4
How did you install Zipline: pip
Python packages:
alembic==1.0.11
appnope==0.1.0
attrs==19.1.0
backcall==0.1.0
bcolz==0.12.1
bleach==3.1.0
Bottleneck==1.2.1
certifi==2019.6.16
chardet==3.0.4
Click==7.0
contextlib2==0.5.5
cyordereddict==1.0.0
Cython==0.29.10
decorator==4.4.0
defusedxml==0.6.0
empyrical==0.5.0
entrypoints==0.3
idna==2.8
intervaltree==3.0.2
ipykernel==5.1.1
ipython==7.5.0
ipython-genutils==0.2.0
ipywidgets==7.4.2
jedi==0.14.0
Jinja2==2.10.1
jsonschema==3.0.1
jupyter==1.0.0
jupyter-client==5.2.4
jupyter-console==6.0.0
jupyter-core==4.5.0
Logbook==1.4.3
lru-dict==1.1.6
lxml==4.3.4
Mako==1.0.12
MarkupSafe==1.1.1
mistune==0.8.4
mock==3.0.5
multipledispatch==0.6.0
nbconvert==5.5.0
nbformat==4.4.0
networkx==1.11
notebook==5.7.8
numexpr==2.6.9
numpy==1.16.4
pandas==0.22.0
pandas-datareader==0.7.0
pandocfilters==1.4.2
parso==0.5.0
patsy==0.5.1
pexpect==4.7.0
pickleshare==0.7.5
prometheus-client==0.7.1
prompt-toolkit==2.0.9
ptyprocess==0.6.0
Pygments==2.4.2
pyrsistent==0.15.2
python-dateutil==2.8.0
python-editor==1.0.4
pytz==2019.1
pyzmq==18.0.1
qtconsole==4.5.1
requests==2.22.0
requests-file==1.4.3
scipy==1.3.0
Send2Trash==1.5.0
six==1.12.0
sortedcontainers==2.1.0
SQLAlchemy==1.3.5
statsmodels==0.10.0
tables==3.5.2
terminado==0.8.2
testpath==0.4.2
toolz==0.9.0
tornado==6.0.3
trading-calendars==1.8.0
traitlets==4.3.2
urllib3==1.25.3
wcwidth==0.1.7
webencodings==0.5.1
widgetsnbextension==3.4.2
wrapt==1.11.2
zipline==1.3.0

Now that you know a little about me, let me tell you about the issue I am
having:

Description of Issue

I signed up and got an api key for quandl
I ran: QUANDL_API_KEY= zipline ingest -b quandl
I created a file called buyapple.py with the following contents:

from zipline.api import symbol, order, record


def initialize(context):
        pass


def handle_data(context, data):
    order(symbol('AAPL'), 10)
    record(AAPL=data.current(symbol('AAPL'), 'price'))

I ran the command zipline run --bundle quandl -f buyapple.py --start 2018-1-1 --end 2018-1-1 --output outapple.pickle

What did you expect to happen?
I expected the backtest to run
What happened instead?
I got the following error:

$ zipline run --bundle quandl -f buyapple.py --start 2018-01-01 --end 2019-01-01 --output output_apple.pickle
[2019-06-27 01:53:19.343551] INFO: Loader: Cache at /Users/richardwuebker/.zipline/data/SPY_benchmark.csv does not have data from 2018-01-02 00:00:00+00:00 to 2018-12-31 00:00:00+00:00.

[2019-06-27 01:53:19.344034] INFO: Loader: Downloading benchmark data for 'SPY' from 2017-12-29 00:00:00+00:00 to 2018-12-31 00:00:00+00:00
Traceback (most recent call last):
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/bin/zipline", line 11, in <module>
    load_entry_point('zipline==1.3.0', 'console_scripts', 'zipline')()
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/site-packages/zipline/__main__.py", line 107, in _
    return f(*args, **kwargs)
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/site-packages/zipline/__main__.py", line 276, in run
    blotter=blotter,
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/site-packages/zipline/utils/run_algo.py", line 159, in _run
    trading_days=trading_calendar.schedule[start:end].index,
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/site-packages/zipline/finance/trading.py", line 103, in __init__
    self.bm_symbol,
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/site-packages/zipline/data/loader.py", line 149, in load_market_data
    environ,
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/site-packages/zipline/data/loader.py", line 216, in ensure_benchmark_data
    data = get_benchmark_returns(symbol)
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/site-packages/zipline/data/benchmarks.py", line 35, in get_benchmark_returns
    data = r.json()
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/site-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/richardwuebker/.pyenv/versions/3.5.4/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Here is how you can reproduce this issue on your machine:

Reproduction Steps

create a new virtual environment with python 3.5.4 with pyenv
pip install zipline
create the buyapple.py file above
get a quandl api key, and ingest the data
Run the command above
...

What steps have you taken to resolve this already?

I've upgraded my operating system, removed my python, brew installed python, brew installed pyenv, created a new envrionment in a directory with 3.5.4, replaced my quandl api key, ingested the quandl data again and retried. I also tried with a jupyter notebook with no luck
...

Anything else?

...

Sincerely,
Rick

Benchmark

Source

rwuebker

Most helpful comment

So here is what I did, go to [(https://iexcloud.io/)], create a free account, get an API token, and modify the following file:
~/anaconda3/envs/env_zipline/lib/python3.5/site-packages/zipline/data/benchmarks.py

do:
import os
on top of the file
IEX_TOKEN=os.environ["IEX_PUBLIC_TOKEN"] r = requests.get( #'https://api.iextrading.com/1.0/stock/{}/chart/5y'.format(symbol), 'https://cloud.iexapis.com/stable/stock/{}/chart/5y?token={}'.format(symbol, IEX_TOKEN) )
Also define IEX_PUBLIC_TOKEN as and environment variable in your bashrc and copy your token key there.
This can get you going for the time being.

kouroshHakha on 12 Aug 2019

👍2

All 13 comments

It is because IEX deprecated the old API for downloading benchmark data.
You can try the several workarounds here #2480

MarikoKujo on 28 Jun 2019

👍1

I am having the same issue. @rwuebker have you found a working solution? Thanks

arslanakhtar61 on 29 Jun 2019

@arslanakhtar61 no I haven't. Please let me know if you think of something. Thanks!

rwuebker on 15 Jul 2019

kouroshHakha on 12 Aug 2019

👍2

getting same error too.

goswamig on 27 Oct 2019

also meet it , have any answers ?

BubleH on 11 Nov 2019

This stackoverflow answer works for me.

tanwaarpornthip on 24 Nov 2019

$ zipline run -f buyapple.py –-start 2000-1-1 –-end 2014-1-1 --output buyapple_out.pickle

The error was due to the follows related to Forbidden while running. It could be limited to access

[2019-11-28 01:01:25.502826] INFO: Loader: Downloading benchmark data for 'SPY' from 1999-12-31 00:00:00+00:00 to 2013-12-31 00:00:00+00:00
s:  <class 'str'> Forbidden

Added print('s: ', type(s), s) to trace it.

$ cat /PATH-TO/lib/python3.6/json/decoder.py

    def decode(self, s, _w=WHITESPACE.match):
        """Return the Python representation of ``s`` (a ``str`` instance
        containing a JSON document).

        """
        print('s: ', type(s), s)
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
        end = _w(s, end).end()
        if end != len(s):
            raise JSONDecodeError("Extra data", s, end)
        return obj

caot on 28 Nov 2019

https://github.com/quantopian/zipline/issues/2480 is related.

caot on 28 Nov 2019

I'm having this problem as well. @jasonsears solution worked for me: https://github.com/jasonsears/zipline/commit/88ebb3426e547b80a40954068dcbbf1afd80cd24

daraul on 4 Apr 2020

Get same error anyone has a working solution?

ZhengxunWu on 26 May 2020

On my repo (get benchmark returns) I have a method to do it manually, I also. provide the data from 1993 to 2020 of SPY daily data. It works, but its a bit limited in the sense that you cant use the api.

cemal95 on 8 Jun 2020

Hi all, I'm going to close this in favor of the consolidated issue for the benchmark downloading problems: https://github.com/quantopian/zipline/issues/2627.

https://github.com/quantopian/zipline/pull/2642 should help now, if you're using master, but it isn't released on PyPI yet.

Feel free to ping there or to reopen this if you think there's information in this issue that's not captured in the other one.