Zipline: Running Pipeline Outside Quantopian

[bradyh20]

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: Windows 10
• Python Version: python 3.6.3
• Python Bitness: 64
• How did you install Zipline: pip
• Python packages:
  alembic==0.9.8
  alphalens==0.2.1
  astroid==1.5.3
  bcolz==0.12.1
  beautifulsoup4==4.6.0
  bleach==1.5.0
  Bottleneck==1.2.1
  certifi==2017.7.27.1
  chardet==3.0.4
  click==6.7
  colorama==0.3.9
  contextlib2==0.5.5
  cycler==0.10.0
  cyordereddict==1.0.0
  Cython==0.27.3
  decorator==4.1.2
  empyrical==0.3.4
  enum34==1.1.6
  et-xmlfile==1.0.1
  ez-setup==0.9
  futures==3.1.1
  h5py==2.7.1
  html5lib==0.9999999
  idna==2.5
  inflection==0.3.1
  int-date==0.1.8
  intervaltree==2.1.0
  ipython==6.2.1
  ipython-genutils==0.2.0
  isort==4.2.15
  jdcal==1.3
  jedi==0.11.0
  keras==2.0.8
  lazy-object-proxy==1.3.1
  Logbook==1.2.1
  lru-dict==1.1.6
  lxml==4.1.0
  Mako==1.0.7
  Markdown==2.6.9
  MarkupSafe==1.0
  matplotlib==2.1.0
  mccabe==0.6.1
  more-itertools==3.2.0
  multipledispatch==0.4.9
  networkx==2.1
  numexpr==2.6.4
  numpy==1.13.3+mkl
  openpyxl==2.4.9
  pandas==0.18.1
  pandas-datareader==0.6.0
  parso==0.1.0
  patsy==0.4.1
  pickleshare==0.7.4
  prompt-toolkit==1.0.15
  protobuf==3.4.0
  Pygments==2.2.0
  pylint==1.7.4
  pyparsing==2.2.0
  python-dateutil==2.6.1
  python-editor==1.0.3
  pytrends==4.3.0
  pytz==2017.2
  pyyaml==3.12
  Quandl==3.2.0
  requests==2.17.3
  requests-file==1.4.3
  requests-ftp==0.3.1
  scikit-learn==0.19.0
  scipy==1.0.0rc2
  seaborn==0.8.1
  simplegeneric==0.8.1
  simplejson==3.13.2
  six==1.11.0
  sklearn==0.0
  sortedcontainers==1.5.9
  SQLAlchemy==1.2.3
  statsmodels==0.8.0
  stockstats==0.2.0
  tables==3.4.2
  tensorflow-gpu==1.4.0rc1
  tensorflow-tensorboard==0.4.0rc1
  toolz==0.9.0
  traitlets==4.3.2
  urllib3==1.21.1
  wcwidth==0.1.7
  Werkzeug==0.12.2
  wrapt==1.10.11
  xlrd==1.1.0
  xlwt==1.3.0
  zipline==1.1.1

Now that you know a little about me, let me tell you about the issue I am
having:

Description of Issue

• I'm trying to get an algorithm that uses pipeline to run outside of quantopian. I based my algorithm on the momentum pipeline example file. Unfortunately, I keep getting a TypeError from this line:

def before_trading_start(context, data): context.pipeline_data = pipeline_output('my_pipeline')

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1. run the attached sma_diff_long_short.py file from your terminal/console
   sma_diff_long_short.txt

What steps have you taken to resolve this already?

I searched this site, examined the help docs, and stackoverflow, but couldn't figure out the issue

Anything else?

I'm not the best coder, so I apologize if this is a simple issue and a waste of your time.

Sincerely,
Robert

[llllllllll]

Hey, it looks like the issue is that you are mixing up the pipeline expression
objects and the data they represent. In the Pipeline API, computation happens in
two steps:

1. Define the entire computation to run.
2. Execute the computation.

These two steps are completely isolated and act on different data types.

Defining the computation

The first step, defining the computation, acts on Pipeline objects like
Filter, Factor, and Classifier. These objects represent a step in the
computation you want to perform. The general name for objects in this space is a
Term, which encompasses all of the different types of computations which can
be run. There are two high level types of terms:

1. loadable terms
2. computable terms

A loadable term is a term which represents loading data into memory. For
example, USEquityPricing.close represents the work needed to fetch the close
prices into memory. These terms can only serve as inputs to computable terms.

A computable term represents some computation on the in-memory data which are
either loaded from loadable terms, or are the results of executing other
computable terms. For example, USequityPricing.close.latest represents the
computation needed to slice off just the most recent day of close prices.

Pipeline defines a rich set of operations which act on these symbolic
objects. These objects may be used in arithmetic expressions to define new
computable terms. They also have a lot of methods to express common
computations, for example: .abs().

Execute the computation

The second step, execute the computation, acts on 2d numpy arrays of shape
(dates, assets). This is where we translate the symbolic computation defined in
the first step into actual data and perform the given operations.

Because we have defined the computation up front, we can perform optimizations
like common subexpression elimination and temporary elimination. Overall, this
makes the computation faster than a naive numpy solution. By controlling the
execution environment, we can also ensure that adjustments and amendments are
being applied correctly in a point-in-time way.

CustomFactor

CustomFactor is a little weird because it straddles the two steps. A
CustomFactor is a symbolic representation of a computation _paired_ with the
concrete implementation of the computation. The instance itself is a Term, and
the compute() method is what gets called in the execute step, meaning it acts
on numpy arrays.

CustomFactor exists for the cases where you cannot express a computation
naturally with the built-in methods or as an arithmetic expression. In general,
using Pipeline terms in expressions is both more readable and runs faster, but
CustomFactor is like the escape hatch to add new novel operations.

The issue you are having is that you are trying to use the symbolic pipeline
objects in the compute() method of your custom factor.

def compute(self, today, assets, out, values):
    slow_sma = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=kwargs['slow_window_length'])
    fast_sma = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=kwargs['fast_window_length'])
    out[:] = (fast_sma - slow_sma) / USEquityPricing.close

Because we are in compute(), we have already moved on to step two, execute;
however, you are trying to define new symbolic terms: slow_sma and
fast_sma. Your pipeline can actually be defined without a CustomFactor at
all:

def make_pipeline():
    slow_sma = SimpleMovingAverage(
        inputs=[USEquityPricing.close],
        window_length=200,
    )
    fast_sma = SimpleMovingAverage(
        inputs=[USEquityPricing.close],
        window_length=30,
    )

    difference = (fast_sma - slow_sma) / USEquityPricing.close.latest
    universe = difference.notnull()
    return Pipeline(
        {
            'testing_factor': difference,
            'longs': difference.top(10),
            'shorts': difference.bottom(10)
        },
        screen=universe,
    )

This defines slow_sma and fast_sma as symbolic objects which represent the
work needed to compute their values. We can then use them in the same arithmetic
expression that you were attempting in the compute(); however, this time it is
still in step one, define the computation, so there are no issues. Some small
notes: you cannot divide through by USEquityPricing.close, you need to add the
.latest because that is what represents the most recent day's close. Also,
for float data type terms like difference, I believe .notnull() and
.notnan() are the same so you only need one of them.

Using this definition, I was able to run your algorithm for the given time
window without any other errors.

Zipline CLI

One last thing, which you may already know, Zipline has a CLI which can be used
to run algorithms. Instead of putting the call to run_algorithm at the bottom
of your algorithm, you can invoke Zipline on your file like:

$ zipline run -f algo.py -s 2015 -e 2017 --capital-base 10000

I hope this helps explain your issue, and helps you understand a little more
about pipeline in general!

[bradyh20]

You're awesome! Thank you sooo much for taking the time to explain this to me. i_owe_ya +=1