Zipline: Quandl ingest does not produce expected .sqlite file

[jraviotta]

Environment

quantopian/zipline docker container on MacOS host

• Operating System:
  Linux 657988c09d2a 4.9.87-linuxkit-aufs #1 SMP Wed Mar 14 15:12:16 UTC 2018 x86_64 GNU/Linux

• Python Version:
  Python 3.5.5

• Python Bitness:
  64

• How did you install Zipline:
  docker run -v notebooks:/projects -v ~/.docker_containers/.zipline:/root/.zipline -p 8888:8888/tcp --name zipline -it quantopian/zipline
  edited Dockerfile ln 65 to && pip install jupyterlab

• Python packages:

alembic==0.9.9
backcall==0.1.0
bcolz==0.12.1
bleach==2.1.3
Bottleneck==1.2.1
certifi==2018.4.16
chardet==3.0.4
click==6.7
contextlib2==0.5.5
cycler==0.10.0
cyordereddict==1.0.0
Cython==0.28.3
decorator==4.3.0
empyrical==0.5.0
entrypoints==0.2.3
html5lib==1.0.1
idna==2.7
intervaltree==2.1.0
ipykernel==4.8.2
ipython==6.4.0
ipython-genutils==0.2.0
jedi==0.12.0
Jinja2==2.10
jsonschema==2.6.0
jupyter-client==5.2.3
jupyter-core==4.4.0
jupyterlab==0.32.1
jupyterlab-launcher==0.10.5
kiwisolver==1.0.1
Logbook==1.4.0
lru-dict==1.1.6
Mako==1.0.7
MarkupSafe==1.0
matplotlib==2.2.2
mistune==0.8.3
multipledispatch==0.5.0
nbconvert==5.3.1
nbformat==4.4.0
networkx==1.11
notebook==5.5.0
numexpr==2.6.5
numpy==1.14.5
pandas==0.18.1
pandas-datareader==0.5.0
pandocfilters==1.4.2
parso==0.2.1
patsy==0.5.0
pexpect==4.6.0
pickleshare==0.7.4
prompt-toolkit==1.0.15
ptyprocess==0.6.0
Pygments==2.2.0
pyparsing==2.2.0
python-dateutil==2.7.3
python-editor==1.0.3
pytz==2018.4
pyzmq==17.0.0
requests==2.19.1
requests-file==1.4.3
requests-ftp==0.3.1
scipy==0.19.1
Send2Trash==1.5.0
simplegeneric==0.8.1
six==1.11.0
sortedcontainers==2.0.4
SQLAlchemy==1.2.8
statsmodels==0.9.0
TA-Lib==0.4.17
tables==3.4.4
terminado==0.8.1
testpath==0.3.1
toolz==0.9.0
tornado==5.0.2
traitlets==4.3.2
urllib3==1.23
wcwidth==0.1.7
webencodings==0.5.1

Description of Issue

After building the container,

• navigate to localhost:8888
• login to jupyterlab
• open terminal in jupyterlab
• run /bin/bash to open bash shell
• run export QUANDL_API_KEY=<key>
• run zipline ingest -b quantopian-quandl

Observe:

Downloading Bundle: quantopian-quandl  [####################################]  100%
[2018-06-25 22:20:40.846461] INFO: zipline.data.bundles.quandl: Writing data to /root/.zipline/data/quantopian-quandl/2018-06-25T22;19;39.980603.

ls /root/.zipline/data/quantopian-quandl/
2018-06-25T22;19;39.980603

Prove zipline works:

zipline run --help
Usage: zipline run [OPTIONS]

  Run a backtest for the given algorithm.

Options:
  -f, --algofile FILENAME         The file that co...[truncated output]

Example fails:

root@657988c09d2a:/projects# pwd
/projects
root@657988c09d2a:/projects# zipline run -f ../../zipline/examples/buyapple.py --start 2016-1-1 --end 2018-1-1 -o buyapple_out.pickle
Usage: zipline run [OPTIONS]

Error: Invalid value for "-f" / "--algofile": Could not open file: ../../zipline/examples/buyapple.py: No such file or directory
root@657988c09d2a:/projects#

Try jupyter notebook:

IN %reload_ext zipline
OUT /zipline/zipline/__init__.py:40: UserWarning: Found TradingCalendar instances after zipline import.
Zipline startup will be much slower until this is fixed!
  "Found TradingCalendar instances after zipline import.\n"

IN %env QUANDL_API_KEY=<key with no quotes>
OUT env: QUANDL_API_KEY='<key redacted>'



md5-848c78f4960a38bf24ee8b755241be3c



IN %zipline ingest -b quantopian-quandl
OUT ---------------------------------------------------------------------------
UsageError                                Traceback (most recent call last)
<ipython-input-20-7c0c9aebe8e1> in <module>()
----> 1 get_ipython().run_line_magic('zipline', 'ingest -b quantopian-quandl')

/usr/local/lib/python3.5/site-packages/IPython/core/interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
   2129                 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
   2130             with self.builtin_trap:
-> 2131                 result = fn(*args,**kwargs)
   2132             return result
   2133 

/zipline/zipline/__main__.py in zipline_magic(line, cell)
    292             '%s%%zipline' % ((cell or '') and '%'),
    293             # don't use system exit and propogate errors to the caller
--> 294             standalone_mode=False,
    295         )
    296     except SystemExit as e:

/usr/local/lib/python3.5/site-packages/click/core.py in main(self, args, prog_name, complete_var, standalone_mode, **extra)
    694         try:
    695             try:
--> 696                 with self.make_context(prog_name, args, **extra) as ctx:
    697                     rv = self.invoke(ctx)
    698                     if not standalone_mode:

/usr/local/lib/python3.5/site-packages/click/core.py in make_context(self, info_name, args, parent, **extra)
    619         ctx = Context(self, info_name=info_name, parent=parent, **extra)
    620         with ctx.scope(cleanup=False):
--> 621             self.parse_args(ctx, args)
    622         return ctx
    623 

/usr/local/lib/python3.5/site-packages/click/core.py in parse_args(self, ctx, args)
    883             ctx.fail('Got unexpected extra argument%s (%s)'
    884                      % (len(args) != 1 and 's' or '',
--> 885                         ' '.join(map(make_str, args))))
    886 
    887         ctx.args = args

/usr/local/lib/python3.5/site-packages/click/core.py in fail(self, message)
    474         :param message: the error message to fail with.
    475         """
--> 476         raise UsageError(message, self)
    477 
    478     def abort(self):

UsageError: Got unexpected extra argument (ingest)



md5-848c78f4960a38bf24ee8b755241be3c



IN %%zipline --start 2016-1-1 --end 2018-1-1
from zipline.api import symbol, order, record

def initialize(context):
    pass

def handle_data(context, data):
    order(symbol('AAPL'), 10)
    record(AAPL=data[symbol('AAPL')].price)

OUT ---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-21-522b5c60fc93> in <module>()
----> 1 get_ipython().run_cell_magic('zipline', '--start 2016-1-1 --end 2018-1-1', "from zipline.api import symbol, order, record\n\ndef initialize(context):\n    pass\n\ndef handle_data(context, data):\n    order(symbol('AAPL'), 10)\n    record(AAPL=data[symbol('AAPL')].price)")

/usr/local/lib/python3.5/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2165             magic_arg_s = self.var_expand(line, stack_depth)
   2166             with self.builtin_trap:
-> 2167                 result = fn(magic_arg_s, cell)
   2168             return result
   2169 

/zipline/zipline/__main__.py in zipline_magic(line, cell)
    292             '%s%%zipline' % ((cell or '') and '%'),
    293             # don't use system exit and propogate errors to the caller
--> 294             standalone_mode=False,
    295         )
    296     except SystemExit as e:

/usr/local/lib/python3.5/site-packages/click/core.py in main(self, args, prog_name, complete_var, standalone_mode, **extra)
    695             try:
    696                 with self.make_context(prog_name, args, **extra) as ctx:
--> 697                     rv = self.invoke(ctx)
    698                     if not standalone_mode:
    699                         return rv

/usr/local/lib/python3.5/site-packages/click/core.py in invoke(self, ctx)
    893         """
    894         if self.callback is not None:
--> 895             return ctx.invoke(self.callback, **ctx.params)
    896 
    897 

/usr/local/lib/python3.5/site-packages/click/core.py in invoke(*args, **kwargs)
    533         with augment_usage_errors(self):
    534             with ctx:
--> 535                 return callback(*args, **kwargs)
    536 
    537     def forward(*args, **kwargs):

/usr/local/lib/python3.5/site-packages/click/decorators.py in new_func(*args, **kwargs)
     15     """
     16     def new_func(*args, **kwargs):
---> 17         return f(get_current_context(), *args, **kwargs)
     18     return update_wrapper(new_func, f)
     19 

/zipline/zipline/__main__.py in run(ctx, algofile, algotext, define, data_frequency, capital_base, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace)
    257         metrics_set=metrics_set,
    258         local_namespace=local_namespace,
--> 259         environ=os.environ,
    260     )
    261 

/zipline/zipline/utils/run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ)
    136             bundle,
    137             environ,
--> 138             bundle_timestamp,
    139         )
    140 

/zipline/zipline/data/bundles/core.py in load(name, environ, timestamp)
    522         return BundleData(
    523             asset_finder=AssetFinder(
--> 524                 asset_db_path(name, timestr, environ=environ),
    525             ),
    526             equity_minute_bar_reader=BcolzMinuteBarReader(

/zipline/zipline/assets/assets.py in __init__(self, engine, future_chain_predicates)
    265     @preprocess(engine=coerce_string_to_eng(require_exists=True))
    266     def __init__(self, engine, future_chain_predicates=CHAIN_PREDICATES):
--> 267         self.engine = engine
    268         metadata = sa.MetaData(bind=engine)
    269         metadata.reflect(only=asset_db_table_names)

/zipline/zipline/utils/input_validation.py in preprocessor(func, argname, arg)
    790     def preprocessor(func, argname, arg):
    791         if isinstance(arg, from_):
--> 792             return to(arg, **to_kwargs)
    793         return arg
    794     return preprocessor

/zipline/zipline/utils/sqlite_utils.py in check_and_create_engine(path, require_exists)
     44 def check_and_create_engine(path, require_exists):
     45     if require_exists:
---> 46         verify_sqlite_path_exists(path)
     47     return sa.create_engine('sqlite:///' + path)
     48 

/zipline/zipline/utils/sqlite_utils.py in verify_sqlite_path_exists(path)
     33 def verify_sqlite_path_exists(path):
     34     if path != ':memory:' and not os.path.exists(path):
---> 35         raise ValueError("SQLite file {!r} doesn't exist.".format(path))
     36 
     37 

ValueError: SQLite file '/root/.zipline/data/quandl/2018-06-25T21;53;44.880174/assets-6.sqlite' doesn't exist.



md5-848c78f4960a38bf24ee8b755241be3c



IN import os
os.listdir('/root/.zipline/data/quandl/')
OUT ['2018-06-25T21;53;44.880174', '2018-06-25T21;44;20.642592', '.cache']



md5-848c78f4960a38bf24ee8b755241be3c



IN import os
os.listdir('/root/.zipline/data/quandl/2018-06-25T21;53;44.880174/')
OUT []



md5-848c78f4960a38bf24ee8b755241be3c



IN import os
os.listdir('/root/.zipline/data/quandl/2018-06-25T21;44;20.642592')
OUT []

...

What steps have you taken to resolve this already?

destroyed container, re-built, same results

...

Thanks in advance for your suggestions.

[jraviotta]

You can close this. I resolved the issue by allocating 4 GB of ram to Docker on my Mac. with more memory the ingest process completed and I can run algorithms.

[ssanderson]

@jraviotta awesome! Thanks for closing the loop (and sorry for the slow replies here).