Zipline: is there a way to view the data in a data bundle?
I have ingest the data to my customized bundle. Now i want to view the data to insure if it is right. for example, i want to plot the adjusted close price. what should I do？
weiguang-zz
All 12 comments
In zipline.data.bundles.core, there is a namedtuple object called BundleData; we can use the asset_finder, adjustments etc to create some way to view the data more easily (maybe in a DataFrame?)
freddiev4
on 22 May 2017
I was able to do it like this. My custom bundle name is ETFs and I had already ingested it. I have no idea if this is the 'correct' way, but it worked for me.
```
from zipline.data.bundles import load, register
from zipline.data.data_portal import DataPortal
from zipline.utils.calendars import get_calendar
from zipline.data.bundles.ETFs import ETFs
import pandas as pd
symbols = ['VOO','IEF']
register('ETFs', ETFs(symbols))
etfs_bundle = load('ETFs')
start_dt = pd.Timestamp('2014-02-03', tz = 'utc')
end_dt = pd.Timestamp('2017-05-31', tz = 'utc')
etfs_data = DataPortal(etfs_bundle.asset_finder, get_calendar('NYSE'),
etfs_bundle.equity_daily_bar_reader.first_trading_day,
equity_minute_reader=etfs_bundle.equity_minute_bar_reader,
equity_daily_reader=etfs_bundle.equity_daily_bar_reader,
adjustment_reader=etfs_bundle.adjustment_reader)
etfs_symbols = []
for ticker in symbols:
etfs_symbols.append(etfs_data.asset_finder.lookup_symbol(ticker,end_dt))
etfs_pricing = etfs_data.get_history_window(etfs_symbols,end_dt,1000,'1d','close')
etfs_pricing.asfreq('D').dropna().plot()
walterkissling
on 21 Jun 2017
I ingested data from quandl, and I need to check what has been loaded i.e. start and end date as well as the list of symbols.
the above code shared by @walterkissling looks promising. I tried to adapt it to quandl bundle but failed.
any idea how I could resolve it?
from zipline.data.bundles import load, register
from zipline.data.data_portal import DataPortal
from zipline.utils.calendars import get_calendar
from zipline.data.bundles.quandl import quandl_bundle
import pandas as pd
symbols = ['AAPL']
register('quandl_bundle', quandl_bundle(symbols))
qd_bundle = load('quandl_bundle')
start_dt = pd.Timestamp('2014-02-03', tz = 'utc')
end_dt = pd.Timestamp('2017-05-31', tz = 'utc')
quandl_data = DataPortal(quandl_bundle.asset_finder, get_calendar('NYSE'),
quandl_bundle.equity_daily_bar_reader.first_trading_day,
equity_minute_reader=quandl_bundle.equity_minute_bar_reader,
equity_daily_reader=quandl_bundle.equity_daily_bar_reader,
adjustment_reader=quandl_bundle.adjustment_reader)
quandl_symbols = []
for ticker in symbols:
quandl_symbols.append(quandl_data.asset_finder.lookup_symbol(ticker,end_dt))
quandl_pricing = quandl_data.get_history_window(quandl_symbols,end_dt,1000,'1d','close')
quandl_pricing.asfreq('D').dropna().plot()
point049
on 16 Feb 2018
Did you ever get this to work?
pluviosilla
on 28 Mar 2018
What if you wanted to view a list of equities present in a bundle? The DataPortal forces you to specify the equities you want to view ahead of time, when you instantiate the DataPortal object, so by definition, it can't provide a list of what the bundle contains.
pluviosilla
on 29 Mar 2018
You can use the asset finder in the bundle to retrieve a list of asset objects:
In [1]: from zipline.data import bundles
In [2]: bundle = bundles.load('csvdir')
In [3]: bundle.asset_finder.retrieve_all(bundle.asset_finder.sids)
Out[3]: [Equity(0 [AAPL]), Equity(1 [IBM]), Equity(2 [KO]), Equity(3 [MSFT])]
'csvdir' is the name of the bundle I want to look up.
llllllllll
on 29 Mar 2018
The data portal is not really meant to be a public interface, using it correctly is tricky. Our standard tool for querying data for all assets is through the pipeline API. Here is some sample code that allows you to quickly run a pipeline against an arbitrary bundle:
import pandas as pd
import toolz
from zipline.data import bundles
from zipline.pipeline.data import USEquityPricing
from zipline.pipeline.loaders import USEquityPricingLoader
from zipline.pipeline.engine import SimplePipelineEngine
@toolz.memoize
def _pipeline_engine_and_calendar_for_bundle(bundle):
"""Create a pipeline engine for the given bundle.
Parameters
----------
bundle : str
The name of the bundle to create a pipeline engine for.
Returns
-------
engine : zipline.pipleine.engine.SimplePipelineEngine
The pipeline engine which can run pipelines against the bundle.
calendar : zipline.utils.calendars.TradingCalendar
The trading calendar for the bundle.
"""
bundle_data = bundles.load(bundle)
pipeline_loader = USEquityPricingLoader(
bundle_data.equity_daily_bar_reader,
bundle_data.adjustment_reader,
)
def choose_loader(column):
if column in USEquityPricing.columns:
return pipeline_loader
raise ValueError(
'No PipelineLoader registered for column %s.' % column
)
calendar = bundle_data.equity_daily_bar_reader.trading_calendar
return (
SimplePipelineEngine(
choose_loader,
calendar.all_sessions,
bundle_data.asset_finder,
),
calendar,
)
def run_pipeline_against_bundle(pipeline, start_date, end_date, bundle):
"""Run a pipeline against the data in a bundle.
Parameters
----------
pipeline : zipline.pipeline.Pipeline
The pipeline to run.
start_date : pd.Timestamp
The start date of the pipeline.
end_date : pd.Timestamp
The end date of the pipeline.
bundle : str
The name of the bundle to run the pipeline against.
Returns
-------
result : pd.DataFrame
The result of the pipeline.
"""
engine, calendar = _pipeline_engine_and_calendar_for_bundle(bundle)
start_date = pd.Timestamp(start_date, tz='utc')
if not calendar.is_session(start_date):
# this is not a trading session, advance to the next session
start_date = calendar.minute_to_session_label(
start_date,
direction='next',
)
end_date = pd.Timestamp(end_date, tz='utc')
if not calendar.is_session(end_date):
# this is not a trading session, advance to the previous session
end_date = calendar.minute_to_session_label(
end_date,
direction='previous',
)
return engine.run_pipeline(pipeline, start_date, end_date)
The usage looks like:
In [1]: from zipline.pipeline import Pipeline
In [2]: from zipline.pipeline.data import USEquityPricing
In [3]: from run_pipeline_against_bundle import run_pipeline_against_bundle
In [4]: run_pipeline_against_bundle(
...: Pipeline({'close': USEquityPricing.close.latest}),
...: '2012',
...: '2013',
...: bundle='csvdir'
...: )
Out[4]:
close
2012-01-04 00:00:00+00:00 Equity(0 [AAPL]) 58.747
Equity(1 [IBM]) 186.300
Equity(2 [KO]) 35.070
Equity(3 [MSFT]) 26.770
2012-01-05 00:00:00+00:00 Equity(0 [AAPL]) 59.062
Equity(1 [IBM]) 185.539
Equity(2 [KO]) 34.849
Equity(3 [MSFT]) 27.400
2012-01-06 00:00:00+00:00 Equity(0 [AAPL]) 59.718
Equity(1 [IBM]) 184.660
Equity(2 [KO]) 34.685
Equity(3 [MSFT]) 27.680
2012-01-09 00:00:00+00:00 Equity(0 [AAPL]) 60.342
Equity(1 [IBM]) 182.539
Equity(2 [KO]) 34.465
Equity(3 [MSFT]) 28.110
2012-01-10 00:00:00+00:00 Equity(0 [AAPL]) 60.247
Equity(1 [IBM]) 181.589
Equity(2 [KO]) 34.465
Equity(3 [MSFT]) 27.740
2012-01-11 00:00:00+00:00 Equity(0 [AAPL]) 60.462
Equity(1 [IBM]) 181.309
Equity(2 [KO]) 34.669
Equity(3 [MSFT]) 27.840
2012-01-12 00:00:00+00:00 Equity(0 [AAPL]) 60.364
Equity(1 [IBM]) 182.320
Equity(2 [KO]) 34.029
Equity(3 [MSFT]) 27.719
2012-01-13 00:00:00+00:00 Equity(0 [AAPL]) 60.198
Equity(1 [IBM]) 180.550
... ...
2012-12-19 00:00:00+00:00 Equity(2 [KO]) 37.279
Equity(3 [MSFT]) 27.559
2012-12-20 00:00:00+00:00 Equity(0 [AAPL]) 75.187
Equity(1 [IBM]) 195.080
Equity(2 [KO]) 36.779
Equity(3 [MSFT]) 27.309
2012-12-21 00:00:00+00:00 Equity(0 [AAPL]) 74.532
Equity(1 [IBM]) 194.770
Equity(2 [KO]) 37.049
Equity(3 [MSFT]) 27.680
2012-12-24 00:00:00+00:00 Equity(0 [AAPL]) 74.190
Equity(1 [IBM]) 193.419
Equity(2 [KO]) 36.889
Equity(3 [MSFT]) 27.450
2012-12-26 00:00:00+00:00 Equity(0 [AAPL]) 74.309
Equity(1 [IBM]) 192.399
Equity(2 [KO]) 36.730
Equity(3 [MSFT]) 27.059
2012-12-27 00:00:00+00:00 Equity(0 [AAPL]) 73.285
Equity(1 [IBM]) 191.949
Equity(2 [KO]) 36.419
Equity(3 [MSFT]) 26.860
2012-12-28 00:00:00+00:00 Equity(0 [AAPL]) 73.580
Equity(1 [IBM]) 192.710
Equity(2 [KO]) 36.419
Equity(3 [MSFT]) 26.959
2012-12-31 00:00:00+00:00 Equity(0 [AAPL]) 72.798
Equity(1 [IBM]) 189.830
Equity(2 [KO]) 35.970
Equity(3 [MSFT]) 26.549
[996 rows x 1 columns]
NOTE: The dates in the output pipline are the day we would show you the data in the simulator, so they are shifted forward by one trading day. For example, the rows for 2012-01-05 show you the close price from the 2012-01-04 trading session.
llllllllll
on 29 Mar 2018
Thanks very much for taking time, Joe. Just minutes ago I found the same answer buried in a thread on the Zipline Google group.
import os
from pandas import Timestamp
from zipline.data.bundles import load
now = Timestamp.utcnow()
bundle = load('quantopian-quandl', os.environ, now)
symbols = set(str(asset.symbol)
for asset in bundle.asset_finder.retrieve_all(
bundle.asset_finder.equities_sids))
I tried this code on python 3.5 with quandl data bundle. it works. Thanks!
`
import os
import pandas as pd
from pandas import Timestamp
from zipline.data.bundles import load
from zipline.data.bundles.quandl import quandl_bundle
from zipline.data.data_portal import DataPortal
from zipline.utils.calendars import get_calendar
now = Timestamp.utcnow()
bundle = load('quandl', os.environ, now)
all_assets = bundle.asset_finder.retrieve_all(bundle.asset_finder.sids)
symbols = set(
str(asset.symbol) for asset in bundle.asset_finder.retrieve_all(bundle.asset_finder.equities_sids)
)
print(symbols)
quandl_data = DataPortal(asset_finder= bundle.asset_finder,
trading_calendar = get_calendar('NYSE'),
first_trading_day = bundle.equity_daily_bar_reader.first_trading_day,
equity_minute_reader=bundle.equity_minute_bar_reader,
equity_daily_reader=bundle.equity_daily_bar_reader,
adjustment_reader=bundle.adjustment_reader)
print(quandl_data)
`
zhaimac
on 28 Jun 2018
I was able to do it like this. My custom bundle name is ETFs and I had already ingested it. I have no idea if this is the 'correct' way, but it worked for me.
from zipline.data.bundles import load, register from zipline.data.data_portal import DataPortal from zipline.utils.calendars import get_calendar from zipline.data.bundles.ETFs import ETFs import pandas as pd symbols = ['VOO','IEF'] register('ETFs', ETFs(symbols)) etfs_bundle = load('ETFs') start_dt = pd.Timestamp('2014-02-03', tz = 'utc') end_dt = pd.Timestamp('2017-05-31', tz = 'utc') etfs_data = DataPortal(etfs_bundle.asset_finder, get_calendar('NYSE'), etfs_bundle.equity_daily_bar_reader.first_trading_day, equity_minute_reader=etfs_bundle.equity_minute_bar_reader, equity_daily_reader=etfs_bundle.equity_daily_bar_reader, adjustment_reader=etfs_bundle.adjustment_reader) etfs_symbols = [] for ticker in symbols: etfs_symbols.append(etfs_data.asset_finder.lookup_symbol(ticker,end_dt)) etfs_pricing = etfs_data.get_history_window(etfs_symbols,end_dt,1000,'1d','close') etfs_pricing.asfreq('D').dropna().plot()
How did you ingest the ETFs from Quandl? I have access to the paid version and I was wondering how to ingest the ETFs into zipline.
life-elixir
on 7 Apr 2020
I don't use Quandl data. I've been using Norgate and they have a zipline package which handles all the ingestion for their data. The link is this: https://pypi.org/project/zipline-norgatedata/
If you read over the source code you can modify the _pricing_iter_equities() function to read in from your own data instaed of their software and use that. They also have ingestion for futures.
walterkissling
on 16 Apr 2020
It's down towards the bottom of the page in the section "Books/publications that use Zipline, adapted for Norgate Data use"
RichardDale
on 10 Jun 2020
Related issues
mosfiqur-rahman
·
4Comments
wongasta
·
8Comments
ikfdarba
·
4Comments
tonyyuandao
·
3Comments
Monsieurvishal
·
5Comments
Most helpful comment
I was able to do it like this. My custom bundle name is ETFs and I had already ingested it. I have no idea if this is the 'correct' way, but it worked for me.
```
from zipline.data.bundles import load, register
from zipline.data.data_portal import DataPortal
from zipline.utils.calendars import get_calendar
from zipline.data.bundles.ETFs import ETFs
import pandas as pd
symbols = ['VOO','IEF']
register('ETFs', ETFs(symbols))
etfs_bundle = load('ETFs')
start_dt = pd.Timestamp('2014-02-03', tz = 'utc')
end_dt = pd.Timestamp('2017-05-31', tz = 'utc')
etfs_data = DataPortal(etfs_bundle.asset_finder, get_calendar('NYSE'),
etfs_bundle.equity_daily_bar_reader.first_trading_day,
equity_minute_reader=etfs_bundle.equity_minute_bar_reader,
equity_daily_reader=etfs_bundle.equity_daily_bar_reader,
adjustment_reader=etfs_bundle.adjustment_reader)
etfs_symbols = []
for ticker in symbols:
etfs_symbols.append(etfs_data.asset_finder.lookup_symbol(ticker,end_dt))
etfs_pricing = etfs_data.get_history_window(etfs_symbols,end_dt,1000,'1d','close')
etfs_pricing.asfreq('D').dropna().plot()