Zipline: Methods of accessing Data in zipline (For Beginners)

[RJUNS]

Dear Quantopian / Zipline Maintainers,

There's good documentation on Orders, Portfolio optimization, Slippage/Commission models. But, am surprised to know that their hardly any guide on accessing data in zipline other than data.history() function.
Maybe it has some easiest way to do it Or it might be so hard to explain that no one wants to talk about.
I had a hard time finding the solution.
The problem is, when we want to perform operations like finding the Max value from a dataframe we usually do:

import pandas_datareader.data as web
import datetime
start = dt.datetime(2017, 5, 1)
end = dt.datetime(2017, 7, 30)
df = web.DataReader('MSFT', 'google', start, end)
highs = df['High'].max()

But when we want to do the same in zipline, I start figuring out whether this code shall be put into the def initialize(context) or it shall be written outside of both initialize() and handle data().

And how to use that data which i received through google finance so that it is compatible with zipline.
I have no idea, how the zipline uses the data and unfortunately their has been no documentation on it as per my understanding. the only thing i know is how data.history() works, which has limited applications. But am more interested in how to access and use data from the code like the above in zipline.
Will i get any help?

[freddiev4]

Hi @RJUNS thanks for posting this issue. So it looks like there are a few questions/comments here that I'll address.

Accessing Data
Zipline uses Data Bundles, which are collections of pricing data, adjustment data, and an asset database. The default data bundle is the quantopian-quandl bundle, which uses the Quandl WIKI dataset.

When you run a zipline backtest, zipline gets its data from the default bundle or the bundle that you specify (e.g. if you make your own bundle). The way to download that data to begin with is by running zipline ingest -b <bundle-name>. If you want to use your own data, you'd have to create a custom bundle (the "how to" on that is in the docs I linked above). There's also an open PR that lets you ingest data from csv files which I'd like to get around to reviewing at some point.

data.history()
Sometimes you might want to compare the most recent bar data to previous bars, which is what data.history() is for. I usually refer to the Quantopian Docs for better understanding usages of data.history()

Where to put code?
There are descriptions of where you should do certain computations in various functions like handle_data or initialize in both the Beginner Tutorial and also the Quantopian docs.

Before the start of the algorithm, zipline calls the initialize() function and passes in a context 
variable. context is a persistent namespace for you to store variables you need to access from one
algorithm iteration to the next.

After the algorithm has been initialized, zipline calls the handle_data() function once for each event.
At every call, it passes the same context variable and an event-frame called data containing the
current trading bar with open, high, low, and close (OHLC) prices as well as volume for each stock
in your universe.

If there's any other questions you have, feel free to post them 🙂

[RJUNS]

@FreddieV4 Thank you for explaining it. Now i understood how zipline works. Actually at the time i was posting this issue, i thought we are bound to write our code only in initialize() and handle_Data() which was getting difficult to accept. and the Quantopian Docs. was somewhat hidden from me, i was just referring to zipline docs and Quantopian's getting started Tuts. But i was glad by looking at the Quantopian 's rich Documentation.
Thanks again freddie for offering to ask more questions, if you don't mind, i will keep this thread open as someone like me has any questions, they may post here. :)

[RJUNS]

%%zipline --start 2017-5-1 --end 2017-6-26

from zipline.api import order, record, symbol

def initialize(context):
    context.assets = symbol('TSLA')

def before_trading_start(context, data):
    # Create a window
    window_1 = data.history(context.assets,'high',39, '1d')
    print(window_1)

When i run the above code, the data.history() returns window starting from 2017-03-07 instead of 2017-5-1

2017-04-24 00:00:00+00:00    310.55
2017-04-25 00:00:00+00:00    313.98
2017-04-26 00:00:00+00:00    314.50
2017-04-27 00:00:00+00:00    313.09
2017-04-28 00:00:00+00:00    314.80
2017-05-01 00:00:00+00:00    327.25
Freq: C, Name: Equity(2574 [TSLA]), dtype: float64
2017-04-25 00:00:00+00:00    313.980
2017-04-26 00:00:00+00:00    314.500
2017-04-27 00:00:00+00:00    313.090
2017-04-28 00:00:00+00:00    314.800
2017-05-01 00:00:00+00:00    327.250
2017-05-02 00:00:00+00:00    327.659

Freq: C, Name: Equity(2574 [TSLA]), dtype: float64
...
...

What could be the issue? @FreddieV4 @llllllllll

[freddiev4]

Hi @RJUNS. data.history() is a convenience function that keeps a rolling window of data for you. In your third argument of your data.history() call, you're passing the number 39, meaning you're using a 39 day window before the start date of your algorithm to use for various analyses.

We can test this & see it using zipline's calendar_utils:

In [1]: from zipline.utils.calendars.calendar_utils import get_calendar

In [2]: cal = get_calendar('NYSE')

In [3]: cal
Out[3]: <zipline.utils.calendars.exchange_calendar_nyse.NYSEExchangeCalendar at 0x1136997b8>

In [4]: cal.sessions_window(pd.Timestamp('2017-5-1'), -39)
Out[4]: 
DatetimeIndex(['2017-03-06', '2017-03-07', '2017-03-08', '2017-03-09',
               '2017-03-10', '2017-03-13', '2017-03-14', '2017-03-15',
               '2017-03-16', '2017-03-17', '2017-03-20', '2017-03-21',
               '2017-03-22', '2017-03-23', '2017-03-24', '2017-03-27',
               '2017-03-28', '2017-03-29', '2017-03-30', '2017-03-31',
               '2017-04-03', '2017-04-04', '2017-04-05', '2017-04-06',
               '2017-04-07', '2017-04-10', '2017-04-11', '2017-04-12',
               '2017-04-13', '2017-04-17', '2017-04-18', '2017-04-19',
               '2017-04-20', '2017-04-21', '2017-04-24', '2017-04-25',
               '2017-04-26', '2017-04-27', '2017-04-28', '2017-05-01'],
              dtype='datetime64[ns, UTC]', freq='C')

There are mentions of this in the Zipline Beginner Tutorial.

[jefferythewind]

I know this is closed but its the only relevant thing I see on the web now. I am trying zipline out. I am completely familiar with the purpose of the algorithms and how they are set up. I have used both Quantopian and Quant Connect.

But I am now just trying to do some very general exploratory data analysis. I just want to load some data from the default Quandl bundle ( which I have already ingested ) in a python script so I can look at it and play around with it. I haven't found anything in the API docs or elsewhere that says you can do that since the load_bars_from_yahoo was removed.

Let's say I had a list of tickers, and I just wanted to get all the data from the past 500 days, for example, loaded into a data frame or multiple data frames if need be. Is this possible?

[MoffyJiang]

I know this is closed but its the only relevant thing I see on the web now. I am trying zipline out. I am completely familiar with the purpose of the algorithms and how they are set up. I have used both Quantopian and Quant Connect.

But I am now just trying to do some very general exploratory data analysis. I just want to load some data from the default Quandl bundle ( which I have already ingested ) in a python script so I can look at it and play around with it. I haven't found anything in the API docs or elsewhere that says you can do that since the load_bars_from_yahoo was removed.

Let's say I had a list of tickers, and I just wanted to get all the data from the past 500 days, for example, loaded into a data frame or multiple data frames if need be. Is this possible?

It can be done by using DataPortal.

from zipline.data.data_portal import DataPortal
from zipline.data import bundles
import pandas as pd

bundle_name = "a bundle name"
ticker_name = "StockA"
end_date = pd.Timestamp('2018-08-01', tz='utc')
calendar_name = "the calendar name"
window=100 # how many days you want to look back

bundle_data = bundles.load(bundle_name)
data_por = DataPortal(bundle_data.asset_finder, 
                      get_calendar(calendar_name),
                      bundle_data.equity_daily_bar_reader.first_trading_day,
                      equity_minute_reader=bundle_data.equity_minute_bar_reader,
                      equity_daily_reader=bundle_data.equity_daily_bar_reader,
                      adjustment_reader=bundle_data.adjustment_reader)

sym = data_por.asset_finder.lookup_symbol(ticker_name, end_date)
data = data_por.get_history_window(assets=[sym],
                                   end_dt=end_date,
                                   bar_count=window,
                                   frequency='1d',
                                   data_frequency='daily',
                                   field='close')

I don't know whether this is what you expected, but here in data you get a dataframe containing all the close price for StockA for the past 100 days until 2018-08-01. @jefferythewind

[rameshwarsingh11]

Hi Experts,

I was wondering if there is any way to connect Zipline API directly to AWS S3 data lakes & AWS data pipelines, get real data from data lake tables for ingestion and then apply backtesting logic on it.

[caot]

The following worked:

from zipline.data.data_portal import DataPortal
from zipline.data import bundles
# from zipline.utils.calendars.calendar_utils import get_calendar
from trading_calendars import register_calendar, get_calendar

cal = get_calendar('NYSE')

import pandas as pd

bundle_name = 'quandl' # "a bundle name"
ticker_name = "IBM"
end_date = pd.Timestamp('2019-11-27', tz='utc')
calendar_name = 'NYSE' # "the calendar name"
window=100 # how many days you want to look back

bundle_data = bundles.load(bundle_name)
data_por = DataPortal(bundle_data.asset_finder, 
                      get_calendar(calendar_name),
                      bundle_data.equity_daily_bar_reader.first_trading_day,
                      equity_minute_reader=bundle_data.equity_minute_bar_reader,
                      equity_daily_reader=bundle_data.equity_daily_bar_reader,
                      adjustment_reader=bundle_data.adjustment_reader)

sym = data_por.asset_finder.lookup_symbol(ticker_name, end_date)
data = data_por.get_history_window(assets=[sym],
                                   end_dt=end_date,
                                   bar_count=window,
                                   frequency='1d',
                                   data_frequency='daily',
                                   field='close')