Pandas-datareader: Google finance - Failed to read symbol, replacing with NaN.

[ranaroussi]

Since Yahoo is dead for now, I'm switching my scripts to use get_data_google() instead of get_data_yahoo().

However, I'm unable to download market indices data from Google and I'm getting:

SymbolWarning: Failed to read symbol: 'INDEXSP:.INX', replacing with NaN.

Was anyone able to download index data from Google?

Here are some of the Yahoo/Google tickers I've tried:

Index                     Google Ticker            Yahoo Ticker
-----------------------------------------------------------------------
S&P 500                   INDEXSP:.INX / .INX      ^GSPC
Dow Jones Industrial      INDEXDJX:.DJI / .DJI     ^DJI
VIX                       INDEXCBOE:VIX            ^VIX
13wk Treasury Bills       INDEXCBOE:IRX            ^IRX

Thanks!

[arose13]

I'm getting the same thing too. I think it's from making too many requests. After I get 3-5 symbols I lose the ability to get any. (The world is ending 😭 )

[dcgithubaccount]

Another problem with google Finance is its coverage. I am able to access only US and UK stocks and not the wider European or Japanese stocks.

Can yahoo be fixed with crumps in pandas datareader ?

[bashtage]

@dcgithubaccount Works for me:

pdr.get_data_google('LON:VOD')

[bashtage]

Oops, I saw the US part and missed the UK part.

[bashtage]

Actually,

pdr.get_data_google('ETR:BAS')

also works.

[dcgithubaccount]

@bashtage. Strange only germany is working. I tried few and below ones didn't work.
web.get_data_google("CPH:NOVO-B") --> Copenhagen
web.get_data_google("SGX:S53") --> Singapore
web.get_data_google("AMS:AGN") --> Amsterdam.
web.get_data_google("EPA:FP") --> Paris

All of them spit out download error it seems only these 3 countries tickers are allowed to download that is my guess.
pandas_datareader._utils.RemoteDataError: Unable to read URL: http://www.google.com/finance/historical

[krinkere]

Yep... same here and I am in exactly the same boat as you are... I was happily using yahoo until they shut down the service without any warnings... LAME

and when I switched to google and tried to pull nasdaq information, it threw an error... sucks

[arose13]

Yeah for me yahoo died first then google died 2 days later. I've switched to quandl data

[dcgithubaccount]

Not sure if this is going to help but i looked at the code of @ranaroussi and others and changed my yahoo access layer. This code works for all sites except HK and Japan. I still couldn't figure why it is not working for Japan and HK. It is not 100% reliable code but accurancy can be improved if sleep is induced in the code.

def cookieAndcrumb(tic):
    url = "https://uk.finance.yahoo.com/quote/{0}/history".format(tic)
    r = requests.get(url)
    txt = r.text
    cookie = r.cookies['B']
    pattern = re.compile('.*"CrumbStore":\{"crumb":"(?P<crumb>[^"]+)"\}')
    for line in txt.splitlines():
        m = pattern.match(line)
        if m is not None:
            crumb = m.groupdict()['crumb']

    return(cookie, crumb)


def getYahooData(tic, start, end):
    start = start.strftime('%s')
    end = end.strftime('%s')
    cc = cookieAndcrumb(tic)
    dataDir = os.path.expanduser('~') + '/twpData'
    if not os.path.exists(dataDir):
        os.mkdir(dataDir)
    data = {'cookie': cc[0], 'crumb': cc[1]}
    dataFile = os.path.join(dataDir, 'yahoo_cookie.yml')

    with open(dataFile, 'w') as fid:
        yaml.dump(data, fid)

    inputData = (tic, int(start), int(end), cc[1])
    url = "https://query1.finance.yahoo.com/v7/finance/download/{0}?period1={1}&period2={2}&interval=1d&events=history&crumb={3}".format(
        *inputData)
    finalData = requests.get(url, cookies={'B': cc[0]})

    buf = io.StringIO(finalData.text)  # create a buffer
    df = pd.read_csv(buf, index_col=0)
    return df

[dcgithubaccount]

@arose13 Is quandl free. Last time i looked there i couldn't find anything cheap ?

[krinkere]

@dcgithubaccount - "Free to Use
The Quandl API is free to use and grants access to all free databases. Users only pay to access Quandl’s premium databases."
from https://www.quandl.com/tools/api

[krinkere]

@dcgithubaccount - Thanks for sharing your code as a work around... at this point, I moved like others suggested to Quandl since they have what I need and I rather not to hack with getting data from Yahoo that seems to be on their way down.

[gliptak]

Looking at the historical pages

https://www.google.com/finance/historical?q=NYSE%3AGS&ei=GB5cWcjcA4iTecCgvbgG has download link (right side middle)

but

https://www.google.com/finance/historical?q=CPH%3ANOVO-B&ei=8RxcWbCTHoe-e5zpjPAE has no download link

[bashtage]

As of v0.7.0 Google finance and Morningstar have been immediately deprecated due to large changes in their API and no stable replacement.