Zipline: Write example script that shows use of custom data.

[ehebert]

There have been a few requests for a guide to using the CUSTOM_DATA type in Zipline, ala the fetch_csv functionality on quantopian.com.

Thinking we should have a script in /examples that uses a custom csv source that has some signal type data.

[thiagof]

Nothing yet?

[llllllllll]

We now support fetch_csv in zipline so I don't think this is needed.

edit: Actually, we should probably have an example algo that uses fetch_csv.

[freddiev4]

x-ref https://github.com/quantopian/zipline/pull/1860

[bartosh]

I can help with this as I'm working on PR #1860. I think it shouldn't be a script, but rather piece of documentation.

When PR #1860 is merged ingesting data from the set of CVS files would be as simple as putting files into the directory with the expected structure and running

CSVDIR=<directory> zipline ingest -b csvdir

Example of thee directory structure for ingesting daily data:

$ find ./csvdir.test
./csvdir.test/
./csvdir.test/daily
./csvdir.test/daily/AA.csv
./csvdir.test/daily/AA_P.csv
./csvdir.test/daily/AA_P_B.csv
./csvdir.test/daily/AAC.csv
./csvdir.test/daily/AADR.csv
./csvdir.test//daily/AAL.csv
./csvdir.test/daily/AAMC.csv
./csvdir.test/daily/AAN.csv
./csvdir.test/daily/AAOI.csv
./csvdir.test/daily/AAON.csv
./csvdir.test/daily/AAP.csv
./csvdir.test/daily/AAPL.csv
./csvdir.test/daily/AAT.csv
./csvdir.test/daily/AAU.csv
./csvdir.test/daily/AAV.csv
./csvdir.test/daily/AAVL.csv
./csvdir.test/daily/AAWW.csv
./csvdir.test/daily/AAXJ.csv

and the ingesting command:

$ CSVDIR=./csvdir.test/ zipline ingest -b csvdir
Loading custom pricing data:   [##----------------------------------]    5%[2017-09-25 10:01:04.736329] DEBUG: zipline.data.bundles.csvdir: AA: sid 0
Loading custom pricing data:   [####--------------------------------]   11%[2017-09-25 10:01:04.827743] DEBUG: zipline.data.bundles.csvdir: AAC: sid 1
Loading custom pricing data:   [######------------------------------]   16%[2017-09-25 10:01:04.849958] DEBUG: zipline.data.bundles.csvdir: AADR: sid 2
Loading custom pricing data:   [########----------------------------]   22%[2017-09-25 10:01:04.880284] DEBUG: zipline.data.bundles.csvdir: AAL: sid 3
Loading custom pricing data:   [##########--------------------------]   27%[2017-09-25 10:01:04.917352] DEBUG: zipline.data.bundles.csvdir: AAMC: sid 4
Loading custom pricing data:   [############------------------------]   33%[2017-09-25 10:01:04.943192] DEBUG: zipline.data.bundles.csvdir: AAN: sid 5
Loading custom pricing data:   [##############----------------------]   38%[2017-09-25 10:01:04.998001] DEBUG: zipline.data.bundles.csvdir: AAOI: sid 6
Loading custom pricing data:   [################--------------------]   44%[2017-09-25 10:01:05.032909] DEBUG: zipline.data.bundles.csvdir: AAON: sid 7
Loading custom pricing data:   [##################------------------]   50%[2017-09-25 10:01:05.089297] DEBUG: zipline.data.bundles.csvdir: AAP: sid 8
Loading custom pricing data:   [####################----------------]   55%[2017-09-25 10:01:05.130081] DEBUG: zipline.data.bundles.csvdir: AAPL: sid 9
Loading custom pricing data:   [######################--------------]   61%[2017-09-25 10:01:05.181852] DEBUG: zipline.data.bundles.csvdir: AAT: sid 10
Loading custom pricing data:   [########################------------]   66%[2017-09-25 10:01:05.217686] DEBUG: zipline.data.bundles.csvdir: AAU: sid 11
Loading custom pricing data:   [##########################----------]   72%[2017-09-25 10:01:05.253095] DEBUG: zipline.data.bundles.csvdir: AAV: sid 12
Loading custom pricing data:   [############################--------]   77%[2017-09-25 10:01:05.311245] DEBUG: zipline.data.bundles.csvdir: AAVL: sid 13
Loading custom pricing data:   [##############################------]   83%[2017-09-25 10:01:05.333659] DEBUG: zipline.data.bundles.csvdir: AAWW: sid 14
Loading custom pricing data:   [################################----]   88%[2017-09-25 10:01:05.377255] DEBUG: zipline.data.bundles.csvdir: AAXJ: sid 15
Loading custom pricing data:   [##################################--]   94%[2017-09-25 10:01:05.402305] DEBUG: zipline.data.bundles.csvdir: AA_P: sid 16
Loading custom pricing data:   [####################################]  100%[2017-09-25 10:01:05.430881] DEBUG: zipline.data.bundles.csvdir: AA_P_B: sid 17
Loading custom pricing data:   [####################################]  100%
Merging daily equity files:  [####################################] 

Note, that csv files should have this header: date,open,high,low,close,volume,dividend,split

$ head csvdir.test/daily/AAPL.csv 
date,open,high,low,close,volume,dividend,split
1996-01-02,32.25,32.25,31.75,32.13,1243700.0,0.0,1.0
1996-01-03,32.0,32.88,31.87,32.13,3837800.0,0.0,1.0
1996-01-04,32.38,32.38,31.37,31.56,2680200.0,0.0,1.0
1996-01-05,31.62,34.25,31.37,34.25,3981500.0,0.0,1.0
1996-01-08,34.5,35.5,34.0,34.63,1083400.0,0.0,1.0
1996-01-09,34.63,34.63,32.75,32.75,2243000.0,0.0,1.0
1996-01-10,32.5,34.75,32.25,34.25,3262800.0,0.0,1.0
1996-01-11,32.63,35.0,32.38,35.0,6756600.0,0.0,1.0
1996-01-12,34.75,34.75,33.25,33.88,3588000.0,0.0,1.0

Regards,
Ed

[freddiev4]

Closing in favor of #2060