Pandas-datareader: data_source='yahoo': reading data fails since July 01
Since 2021-07-01, reading data from Yahoo fails with a nondescript error message.
Calling the URL (see RemoteDataError below) directly from a browser succeeds. wget returns error 404: Not Found.
Interesting is the name of the background image in the HTML message: "s.yimg.com/nn/img/sad-panda-201402200631.png"
_Pandas datareader version 0.9.0_
Code snippet:
import pandas as pd
import pandas_datareader as pdr
# start date is arbitrary
pdr.DataReader('GOOGL', data_source='yahoo', start='2021-07-01')
Error message:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/iago/.local/lib/python3.9/site-packages/pandas/util/_decorators.py", line 199, in wrapper
return func(*args, **kwargs)
File "/usr/lib/python3.9/site-packages/pandas_datareader/data.py", line 376, in DataReader
return YahooDailyReader(
File "/usr/lib/python3.9/site-packages/pandas_datareader/base.py", line 253, in read
df = self._read_one_data(self.url, params=self._get_params(self.symbols))
File "/usr/lib/python3.9/site-packages/pandas_datareader/yahoo/daily.py", line 153, in _read_one_data
resp = self._get_response(url, params=params)
File "/usr/lib/python3.9/site-packages/pandas_datareader/base.py", line 181, in _get_response
raise RemoteDataError(msg)
pandas_datareader._utils.RemoteDataError: Unable to read URL: https://finance.yahoo.com/quote/GOOGL/history?period1=1625104800&period2=1625363999&interval=1d&frequency=1d&filter=history
Response Text:
b'<!DOCTYPE html>\n <html lang="en-us"><head>\n <meta http-equiv="content-type" content="text/html; charset=UTF-8">\n <meta charset="utf-8">\n <title>Yahoo</title>\n <meta name="viewport" content="width=device-width,initial-scale=1,minimal-ui">\n <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">\n <style>\n html {\n height: 100%;\n }\n body {\n background: #fafafc url(https://s.yimg.com/nn/img/sad-panda-201402200631.png) 50% 50%;\n background-size: cover;\n height: 100%;\n text-align: center;\n font: 300 18px "helvetica neue", helvetica, verdana, tahoma, arial, sans-serif;\n }\n table {\n height: 100%;\n width: 100%;\n table-layout: fixed;\n border-collapse: collapse;\n border-spacing: 0;\n border: none;\n }\n h1 {\n font-size: 42px;\n font-weight: 400;\n color: #400090;\n }\n p {\n color: #1A1A1A;\n }\n #message-1 {\n font-weight: bold;\n margin: 0;\n }\n #message-2 {\n display: inline-block;\n *display: inline;\n zoom: 1;\n max-width: 17em;\n _width: 17em;\n }\n </style>\n <script>\n document.write(\'<img src="//geo.yahoo.com/b?s=1197757129&t=\'+new Date().getTime()+\'&src=aws&err_url=\'+encodeURIComponent(document.URL)+\'&err=%<pssc>&test=\'+encodeURIComponent(\'%<{Bucket}cqh[:200]>\')+\'" width="0px" height="0px"/>\');var beacon = new Image();beacon.src="//bcn.fp.yahoo.com/p?s=1197757129&t="+new Date().getTime()+"&src=aws&err_url="+encodeURIComponent(document.URL)+"&err=%<pssc>&test="+encodeURIComponent(\'%<{Bucket}cqh[:200]>\');\n </script>\n </head>\n <body>\n <!-- status code : 404 -->\n <!-- Not Found on Server -->\n <table>\n <tbody><tr>\n <td>\n <img src="https://s.yimg.com/rz/p/yahoo_frontpage_en-US_s_f_p_205x58_frontpage.png" alt="Yahoo Logo">\n <h1 style="margin-top:20px;">Will be right back...</h1>\n <p id="message-1">Thank you for your patience.</p>\n <p id="message-2">Our engineers are working quickly to resolve the issue.</p>\n </td>\n </tr>\n </tbody></table>\n </body></html>'
aeolio
All 39 comments
I am seeing this same issue.
nlauchande
on 3 Jul 2021
yes i have the same problem too. Many thanks to those who will make the adjustment
beginner000001
on 3 Jul 2021
Hello, all,
Here I share a few tricks to solve this problem:
import pandas
from pandas_datareader import data as pdr
import yfinance as yfin
yfin.pdr_override()
then
df = pdr.get_data_yahoo("TSLA", start="yyyy-mm-dd", end="yyyy-mm-dd")
print(df)
Hopefully it can be useful, thanks :)
@aeolio @nlauchande @beginner000001
ericsnth
on 3 Jul 2021
Hello, all,
Here I share a few tricks to solve this problem:
import pandas
from pandas_datareader import data as pdr
import yfinance as yfin
yfin.pdr_override()then
df = pdr.get_data_yahoo("TSLA", start="yyyy-mm-dd", end="yyyy-mm-dd")
print(df)Hopefully it can be useful, thanks :)
@aeolio @nlauchande @beginner000001
thank you very much.
I could not wait and made a not so nice workoround, but it is working and I can fix it when yhoo changes the syntax:
import numpy as np
import pandas as pd
import requests
from datetime import datetime, timedelta, date
result = requests.get('https://query1.finance.yahoo.com/v7/finance/download/BABA?period1=000000001&period2=9999999999&interval=1d&events=history&includeAdjustedClose=true')
f = open('file.csv', "w")
f.write(result.text)
f.close()
df_daten = pd.read_csv('file.csv')
df_daten["Date"]= pd.to_datetime(df_daten["Date"])
df_daten.set_index('Date', inplace=True)
start_remove = pd.to_datetime('2001-1-1')
end_remove = pd.to_datetime('2021-7-1')
df_neu = df_daten.query('Date < @start_remove or Date > @end_remove')
Hello, all,
Here I share a few tricks to solve this problem:
import pandas
from pandas_datareader import data as pdr
import yfinance as yfin
yfin.pdr_override()then
df = pdr.get_data_yahoo("TSLA", start="yyyy-mm-dd", end="yyyy-mm-dd")
print(df)Hopefully it can be useful, thanks :)
@aeolio @nlauchande @beginner000001
Thanks for the alternative proposal. Although it requires installing an additional package, it solves the problem as a workaround. But, the original problem with pandas-datareader still stands.
hozgryldz
on 3 Jul 2021
Same issue being discussed in #867
thanks for the workaround suggestions
shortsallday
on 4 Jul 2021
Is it likely we will get a fix, or should we use the yfinance workaround?
spot92
on 5 Jul 2021
A solution is outlined in issue #867, i.e. patch base.py to create a request header if none exists.
If nobody objects, I propose to close this issue as a duplicate.
aeolio
on 6 Jul 2021
Hello, all,
Here I share a few tricks to solve this problem:
import pandas
from pandas_datareader import data as pdr
import yfinance as yfin
yfin.pdr_override()then
df = pdr.get_data_yahoo("TSLA", start="yyyy-mm-dd", end="yyyy-mm-dd")
print(df)Hopefully it can be useful, thanks :)
@aeolio @nlauchande @beginner000001
do you get real time data?, i get only historical data, not real time as before
lymadvisors
on 6 Jul 2021
Any one know how to get real-time data? I am using AlphaVantage to get now my data, but it have only historical data (upto the previous closed trading day)
Dark-Mars
on 9 Jul 2021
Hello, all,
Here I share a few tricks to solve this problem:
import pandas
from pandas_datareader import data as pdr
import yfinance as yfin
yfin.pdr_override()then
df = pdr.get_data_yahoo("TSLA", start="yyyy-mm-dd", end="yyyy-mm-dd")
print(df)Hopefully it can be useful, thanks :)
@aeolio @nlauchande @beginner000001
I was doing a course when I encountered this issue. The problem I am facing in this workaround is that if I name pandas_DataReader as wb (or anything apart from pdr as mentioned) it shows an issue of yfin.wb_override() not being an attribute of finance.
Could someone explain this to me, because as far as I understand the nickname is only for the help of the programmer and nothing else?
sakshambhutani5
on 9 Jul 2021
I think that yahoo is prohibiting to download data, at least today friday
lymadvisors
on 9 Jul 2021
Hello, all,
Here I share a few tricks to solve this problem:
import pandas
from pandas_datareader import data as pdr
import yfinance as yfin
yfin.pdr_override()
then
df = pdr.get_data_yahoo("TSLA", start="yyyy-mm-dd", end="yyyy-mm-dd")
print(df)
the above workaround does not work anymore.. I get the following error:
Warning (from warnings module):
File "C:UsersBranAppDataLocalProgramsPythonPython38libsite-packagespandas_datareadercompat__init__.py", line 7
from pandas.util.testing import assert_frame_equal
FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:UsersBranAppDataLocalProgramsPythonPython38libthreading.py", line 932, in _bootstrap_inner
self.run()
File "C:UsersBranAppDataLocalProgramsPythonPython38libthreading.py", line 870, in run
self._target(self._args, *self._kwargs)
File "C:UsersBranAppDataLocalProgramsPythonPython38libsite-packagesmultitasking__init__.py", line 102, in _run_via_pool
return callee(args, *kwargs)
File "C:UsersBranAppDataLocalProgramsPythonPython38libsite-packagesyfinancemulti.py", line 166, in _download_one_threaded
data = _download_one(ticker, start, end, auto_adjust, back_adjust,
File "C:UsersBranAppDataLocalProgramsPythonPython38libsite-packagesyfinancemulti.py", line 178, in _download_one
return Ticker(ticker).history(period=period, interval=interval,
File "C:UsersBranAppDataLocalProgramsPythonPython38libsite-packagesyfinancebase.py", line 156, in history
data = data.json()
File "C:UsersBranAppDataLocalProgramsPythonPython38libsite-packagesrequestsmodels.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "C:UsersBranAppDataLocalProgramsPythonPython38libjson__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "C:UsersBranAppDataLocalProgramsPythonPython38libjsondecoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:UsersBranAppDataLocalProgramsPythonPython38libjsondecoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
bran888
on 10 Jul 2021
Same issue here.
I think that yahoo is prohibiting to download data, at least today friday
That's not the problem.
simon-r
on 10 Jul 2021
I think that yahoo is prohibiting to download data, at least today friday
Got also "Forbidden" yesterday on friday. Had to change the way.
beginner000001
on 10 Jul 2021
Hello all,
Looks like the previous trick is no longer usable; so I made a new trick to solve the problem.
Below is a code that can be an alternative to get yahoo data created natively
import io
import pandas
from datetime import datetime
import requests
class YahooData:
def fetch(ticker, start, end):
headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9'
}
url = "https://query1.finance.yahoo.com/v7/finance/download/" + str(ticker)
x = int(datetime.strptime(start, '%Y-%m-%d').strftime("%s"))
y = int(datetime.strptime(end, '%Y-%m-%d').strftime("%s"))
url += "?period1=" + str(x) + "&period2=" + str(y) + "&interval=1d&events=history&includeAdjustedClose=true"
r = requests.get(url, headers=headers)
pd = pandas.read_csv(io.StringIO(r.text), index_col=0, parse_dates=True)
return pd
then, how to use:
df = YahooData.fetch("TSLA", start="2000-01-01", end="2021-12-31")
print(df)
Hopefully it can be useful, thanks :)
@beginner000001 @simon-r @bran888 @lymadvisors @sakshambhutani5 @Dark-Mars
ericsnth
on 10 Jul 2021
Upgrade yfinance to 0.1.62
spot92
on 10 Jul 2021
Same error with python 3.8 and 0.1.63 ... :( simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
LinuxpowerLudo
on 11 Jul 2021
yfinance made a similar fix as outlined in #867 , a pull request for pandas_datareader is 'in the process', hopefully merged 'soon'.
galashour
on 11 Jul 2021
Consolidating to #867.
bashtage
on 12 Jul 2021
Please try master to see if it is working now.
pip install git+https://github.com/pydata/pandas-datareader.git
If there are positive reports, I'll release very soon.
bashtage
on 12 Jul 2021
Please try master to see if it is working now.
pip install git+https://github.com/pydata/pandas-datareader.gitIf there are positive reports, I'll release very soon.
Just upgraded to the latest version (as well as yfinance) and it seems to work great! Thanks for the fix!
ngopal
on 13 Jul 2021
Please try master to see if it is working now.
pip install git+https://github.com/pydata/pandas-datareader.git
Works for me also
aeolio
on 13 Jul 2021
0.10.0 is out so hopefully solved for now.
bashtage
on 13 Jul 2021
0.10.0 is out so hopefully solved for now.
Still cannot use import pandas_datareader as web to download Yahoo prices since July 2, 2021. Do I have to download new version? How do I do that? Thanks for any help.
uad1098
on 13 Jul 2021
tried to install pip pandas datareader but installed 0.9.0.
Do I have to use shell python -m pip install git+https://github.com/pydata/pandas-datareader.git to install 0.10.0 since it is a development version.
uad1098
on 13 Jul 2021
@uad1098 pip install pandas-datareader --upgrade
You can check the version of your package using something like:
import pandas_datareader as wb
wb._ _ version__
(no space between the underscores; it bolded the first time)
spot92
on 13 Jul 2021
pip install pandas-datareader -U
On Tue, Jul 13, 2021, 21:03 uad1098 @.*> wrote:
0.10.0 is out so hopefully solved for now.
Still cannot use import pandas_datareader as web to download Yahoo prices
since July 2, 2021. Do I have to download new version? How do I do that?
Thanks for any help.—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/pydata/pandas-datareader/issues/868#issuecomment-879364480,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKTSRKGH3B6SMGWJEPGCPLTXSL2NANCNFSM47XZJ7BA
.
bashtage
on 13 Jul 2021
I got datareader 0.10.0 installed and works in python but cannot call python from java as follows anymore. Do I have to use import pandas_datareader as pdr instead as Web?
import sys
startDate = sys.argv[1]
endDate=sys.argv[2]
stock=sys.argv[3]
import pandas_datareader as web
iimport datetime
start=startDate#when sys.argv[1]
end=endDate#when sys.argv[2]
df = web.DataReader(stock, 'yahoo', start, end)
print(df[['High','Low','Close']])
path= 'c:/PythonPrograms/CallPythonApp/'+stock+"Update1.csv"
df.to_csv(path)
uad1098
on 14 Jul 2021
Why don't you use Python to verify that it works? Calling from Java (or anything other than CPython) is not a project goal.
bashtage
on 14 Jul 2021
It does work from python. It used to work with a call from java in v0.9.0. and passing arguments with import sys but with v0.10.0 it no longer works using import sys. I have a java program that reads csv file that python creates when I call python using import sys but now can't pass arguments to the new python version so only way it works is to manually activate python for each stock to create csv files and then use java to those read files and chart with trade recommendations Not so good. Might as well manually enter prices in csv file each day so python solution not as efficient as it was. Better if could pass arguments to python to create csv files as it did before. Just some feedback that would improve v0.10.0 working like it did before.
uad1098
on 14 Jul 2021
Thanks Kevin,
Yes I did finally install 0.10.0 and it works using python only. It also works when I pass arguments using cmd. But it no longer works using Java runtime to pass arguments but it did work nicely using datareader 0.9.0. Still trying to figure out why.
From: Kevin Sheppard
Sent: Tuesday, July 13, 2021 4:34 PM
To: pydata/pandas-datareader @.>
Cc: uad1098 *@.>; Mention @.**>
Subject: Re: [pydata/pandas-datareader] data_source='yahoo': reading data fails since July 01 (#868)
Why don't you use Python to verify that it works? Calling from Java (or anything other than CPython) is not a project goal.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/pydata/pandas-datareader/issues/868#issuecomment-879449767 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AUYT2BLDFAQUMP2RZ42WBUTTXS5MVANCNFSM47XZJ7BA . https://github.com/notifications/beacon/AUYT2BJOJ3LQ63OSUPEF2VLTXS5MVA5CNFSM47XZJ7BKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGRVVNJY.gif
uad1098
on 14 Jul 2021
Thanks Kevin,
Yes I did finally install 0.10.0 and it works using python only. It also works when I pass arguments using cmd. But it no longer works using Java runtime to pass arguments but it did work nicely using datareader 0.9.0. Still trying to figure out why.
I noticed that I can get arguments passed to python script using java runtime but no print(df[['High','Low','Close']]) so not sure
df = web.DataReader(stock, 'yahoo', start, end) is working. Also tried pdr.DataReader and still did not work.
uad1098
on 14 Jul 2021
This 80 year old dummy solved his problem. Was directing Java Runtime to the old 0.9.0 file and not the 0.10.0 even though I successfully downloaded the latest version. Sorry about that.
uad1098
on 14 Jul 2021
YahooData.fetch("TSLA", start="2000-01-01", end="2021-12-31")
I tried in python 2.7 and return 'TypeError: unbound method fetch() must be called with YahooData instance as first argument (got unicode instance instead)'
bkcollection
on 14 Jul 2021
You must use Python 3.6+
bashtage
on 14 Jul 2021
I'm experiencing the same issue with IEX and requests.
I'm on python 3.9 conda, pycharm running jupyter notebook, per @bashtage I've pip install pandas-datareader -U no effect.
stocks = pd.read_csv('sp_500_stocks.csv')
from secrets import IEX_CLOUD_API_TOKEN
symbol = 'AAPL'
api_url = f'https://sandbox.iexapis.com/stable/stock/{symbol}/stats?token={IEX_CLOUD_API_TOKEN}'
data = requests.get(api_url).json()
data
JSONDecodeError Traceback (most recent call last)
/var/folders/m3/jsfh6v792_bg33tq9dyqyv200000gn/T/ipykernel_44001/1737658719.py in <module>
1 symbol = 'AAPL'
2 api_url = f'https://sandbox.iexapis.com/stable/stock/{symbol}/stats?token={IEX_CLOUD_API_TOKEN}'
----> 3 data = requests.get(api_url).json()
4 data
5
~/.conda/envs/algorithmic-trading-python/lib/python3.9/site-packages/requests/models.py in json(self, **kwargs)
908 # used.
909 pass
--> 910 return complexjson.loads(self.text, **kwargs)
911
912 @property
~/.conda/envs/algorithmic-trading-python/lib/python3.9/site-packages/simplejson/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, use_decimal, **kw)
523 parse_constant is None and object_pairs_hook is None
524 and not use_decimal and not kw):
--> 525 return _default_decoder.decode(s)
526 if cls is None:
527 cls = JSONDecoder
~/.conda/envs/algorithmic-trading-python/lib/python3.9/site-packages/simplejson/decoder.py in decode(self, s, _w, _PY3)
368 if _PY3 and isinstance(s, bytes):
369 s = str(s, self.encoding)
--> 370 obj, end = self.raw_decode(s)
371 end = _w(s, end).end()
372 if end != len(s):
~/.conda/envs/algorithmic-trading-python/lib/python3.9/site-packages/simplejson/decoder.py in raw_decode(self, s, idx, _w, _PY3)
398 elif ord0 == 0xef and s[idx:idx + 3] == '\xef\xbb\xbf':
399 idx += 3
--> 400 return self.scan_once(s, idx=_w(s, idx).end())
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
datatalking
on 9 Aug 2021
Finally it worked by updating yfinance ( and datareader). I first tried updating datareader, didn`t work and after updating yfinance it worked.
hase139
on 21 Aug 2021
Finally it worked by updating yfinance ( and datareader). I first tried updating datareader, didn`t work and after updating yfinance it worked.
Both yfinance and datareader were exposed to the same issue, and thus both affected at the same time. If you used both libraries, you had to update both of them.
galashour
on 22 Aug 2021
Related issues
amoriello
·
5Comments
hmz123
·
6Comments
BajajAryan310
·
7Comments
paintdog
·
7Comments
arose13
·
3Comments
Most helpful comment
Hello, all,
Here I share a few tricks to solve this problem:
import pandasfrom pandas_datareader import data as pdrimport yfinance as yfinyfin.pdr_override()then
df = pdr.get_data_yahoo("TSLA", start="yyyy-mm-dd", end="yyyy-mm-dd")print(df)Hopefully it can be useful, thanks :)
@aeolio @nlauchande @beginner000001