What will we cover?
If you use get_data_yahoo from Pandas Datareader and it suddenly stopped working, then we will look at how to fix.
UPDATE: We will also cover a FIX of the current issue in 2023
Current 2023 ISSUE and FIX
If you get an error that ends with.
TypeError: string indices must be integers
You get that because there has been breaking changes at Yahoo! Finance.
An easy workaround is as follows.
Install yfinance.
Then this code will work.
import pandas_datareader.data as pdr
import yfinance as yf
yf.pdr_override()
from datetime import datetime
data = pdr.get_data_yahoo('^GSPC', datetime(1970, 1, 1))
The above code should work.
Below you have the full error trace.
TypeError Traceback (most recent call last)
/var/folders/5f/vgbh9pkd4wzf4ysrpf9wsr800000gn/T/ipykernel_82918/2899696086.py in <module>
----> 1 data = pdr.get_data_yahoo('AAPL', datetime(2022, 1, 1))
~/opt/anaconda3/lib/python3.9/site-packages/pandas_datareader/data.py in get_data_yahoo(*args, **kwargs)
78
79 def get_data_yahoo(*args, **kwargs):
---> 80 return YahooDailyReader(*args, **kwargs).read()
81
82
~/opt/anaconda3/lib/python3.9/site-packages/pandas_datareader/base.py in read(self)
251 # If a single symbol, (e.g., 'GOOG')
252 if isinstance(self.symbols, (string_types, int)):
--> 253 df = self._read_one_data(self.url, params=self._get_params(self.symbols))
254 # Or multiple symbols, (e.g., ['GOOG', 'AAPL', 'MSFT'])
255 elif isinstance(self.symbols, DataFrame):
~/opt/anaconda3/lib/python3.9/site-packages/pandas_datareader/yahoo/daily.py in _read_one_data(self, url, params)
151 try:
152 j = json.loads(re.search(ptrn, resp.text, re.DOTALL).group(1))
--> 153 data = j["context"]["dispatcher"]["stores"]["HistoricalPriceStore"]
154 except KeyError:
155 msg = "No data fetched for symbol {} using {}"
TypeError: string indices must be integers
The Error and Problem (old problem)
Consider this code.
import pandas_datareader as pdr
from datetime import datetime
data = pdr.get_data_yahoo('^GSPC', datetime(1970, 1, 1))
It has been working up until now. But suddenly it writes.
Traceback (most recent call last):
File "/Users/rune/PycharmProjects/TEST/test_yahoo.py", line 4, in <module>
data = pdr.get_data_yahoo('^GSPC', datetime(1970, 1, 1))
File "/Users/rune/PycharmProjects/TEST/venv/lib/python3.8/site-packages/pandas_datareader/data.py", line 86, in get_data_yahoo
return YahooDailyReader(*args, **kwargs).read()
File "/Users/rune/PycharmProjects/TEST/venv/lib/python3.8/site-packages/pandas_datareader/base.py", line 253, in read
df = self._read_one_data(self.url, params=self._get_params(self.symbols))
File "/Users/rune/PycharmProjects/TEST/venv/lib/python3.8/site-packages/pandas_datareader/yahoo/daily.py", line 153, in _read_one_data
resp = self._get_response(url, params=params)
File "/Users/rune/PycharmProjects/TEST/venv/lib/python3.8/site-packages/pandas_datareader/base.py", line 181, in _get_response
raise RemoteDataError(msg)
pandas_datareader._utils.RemoteDataError: Unable to read URL: https://finance.yahoo.com/quote/^GSPC/history?period1=10800&period2=1627523999&interval=1d&frequency=1d&filter=history
Response Text:
b'<!DOCTYPE html>n <html lang="en-us"><head>n <meta http-equiv="content-type" content="text/html; charset=UTF-8">n <meta charset="utf-8">n <title>Yahoo</title>n <meta name="viewport" content="width=device-width,initial-scale=1,minimal-ui">n <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">n <style>n html {n height: 100%;n }n body {n background: #fafafc url(https://s.yimg.com/nn/img/sad-panda-201402200631.png) 50% 50%;n background-size: cover;n height: 100%;n text-align: center;n font: 300 18px "helvetica neue", helvetica, verdana, tahoma, arial, sans-serif;n }n table {n height: 100%;n width: 100%;n table-layout: fixed;n border-collapse: collapse;n border-spacing: 0;n border: none;n }n h1 {n font-size: 42px;n font-weight: 400;n color: #400090;n }n p {n color: #1A1A1A;n }n #message-1 {n font-weight: bold;n margin: 0;n }n #message-2 {n display: inline-block;n *display: inline;n zoom: 1;n max-width: 17em;n _width: 17em;n }n </style>n <script>n document.write('<img src="//geo.yahoo.com/b?s=1197757129&t='+new Date().getTime()+'&src=aws&err_url='+encodeURIComponent(document.URL)+'&err=%<pssc>&test='+encodeURIComponent('%<{Bucket}cqh[:200]>')+'" width="0px" height="0px"/>');var beacon = new Image();beacon.src="//bcn.fp.yahoo.com/p?s=1197757129&t="+new Date().getTime()+"&src=aws&err_url="+encodeURIComponent(document.URL)+"&err=%<pssc>&test="+encodeURIComponent('%<{Bucket}cqh[:200]>');n </script>n </head>n <body>n <!-- status code : 404 -->n <!-- Not Found on Server -->n <table>n <tbody><tr>n <td>n <img src="https://s.yimg.com/rz/p/yahoo_frontpage_en-US_s_f_p_205x58_frontpage.png" alt="Yahoo Logo">n <h1 style="margin-top:20px;">Will be right back...</h1>n <p id="message-1">Thank you for your patience.</p>n <p id="message-2">Our engineers are working quickly to resolve the issue.</p>n </td>n </tr>n </tbody></table>n </body></html>'
What to do?
The fix (old fix)
There has been a breaking change and you need to update your Pandas Datareader.
You can upgrade to the newest version as follows.
pip install --upgrade pandas-datareader
It should update it to version 0.10.0 or later.
Then the code should work again.
Learn Python A BEGINNERS GUIDE TO PYTHON
- 70 pages to get you started on your journey to master Python.
- How to install your setup with Anaconda.
- Written description and introduction to all concepts.
- Jupyter Notebooks prepared for 17 projects.
Python 101: A CRASH COURSE
- How to get started with this 8 hours Python 101: A CRASH COURSE.
- Best practices for learning Python.
- How to download the material to follow along and create projects.
- A chapter for each lesson with a description, code snippets for easy reference, and links to a lesson video.
Expert Data Science Blueprint
Expert Data Science Blueprint
- Master the Data Science Workflow for actionable data insights.
- How to download the material to follow along and create projects.
- A chapter to each lesson with a Description, Learning Objective, and link to the lesson video.
Machine Learning
Machine Learning – The Simple Path to Mastery
- How to get started with Machine Learning.
- How to download the material to follow along and make the projects.
- One chapter for each lesson with a Description, Learning Objectives, and link to the lesson video.
Содержание
- Fix get_data_yahoo from Pandas Datareader
- What will we cover?
- Current 2023 ISSUE and FIX
- The Error and Problem (old problem)
- The fix (old fix)
- Learn Python
- Expert Data Science Blueprint
- Machine Learning
- 21 thoughts on “Fix get_data_yahoo from Pandas Datareader”
- ‘utf-8’ Encoding + ‘unable to read URL’ errors: when downloading financial data from yahoo and google #428
- Comments
- Yahoo Daily depricated? #549
- Comments
Fix get_data_yahoo from Pandas Datareader
What will we cover?
If you use get_data_yahoo from Pandas Datareader and it suddenly stopped working, then we will look at how to fix.
UPDATE: We will also cover a FIX of the current issue in 2023
Current 2023 ISSUE and FIX
If you get an error that ends with.
You get that because there has been breaking changes at Yahoo! Finance.
An easy workaround is as follows.
Then this code will work.
The above code should work.
Below you have the full error trace.
The Error and Problem (old problem)
Consider this code.
It has been working up until now. But suddenly it writes.
The fix (old fix)
There has been a breaking change and you need to update your Pandas Datareader.
You can upgrade to the newest version as follows.
It should update it to version 0.10.0 or later.
Then the code should work again.
Learn Python
Learn Python A BEGINNERS GUIDE TO PYTHON
- 70 pages to get you started on your journey to master Python.
- How to install your setup with Anaconda.
- Written description and introduction to all concepts.
- Jupyter Notebooks prepared for 17 projects .
Master Python EXPERTISE BOOTCAMP
- How to get started with this 8 hours Master Python Expertise Bootcamp.
- Best practices for learning Python.
- How to download the material to follow along and create projects.
- A chapter for each lesson with a description, codesnippets for easy reference and links to a lessonvideo.
Expert Data Science Blueprint
Expert Data Science Blueprint
- Master the Data Science Workflow for actionable data insights.
- How to download the material to follow along and create projects.
- A chapter to each lesson with Description, Learning Objective, and link to lesson video.
Machine Learning
Machine Learning – The Simple Path to Mastery
- How to get started with the Machine Learning.
- How to download the material to follow along and make the projects.
- One chapter for each lesson with Description, Learning Objectives, and link to lesson video.
21 thoughts on “Fix get_data_yahoo from Pandas Datareader”
While using my python script that uses pandas-DataReader v10 & pandas v1.3.3 get_data_yahoo(), when it first starts off it will show the last 5 days correctly because I am tailing this for the last 5 days, then after about a minute or two it changes to what you see below, I am using python 3.7.10 The date moves back 2 weeks and the volume from the last 4 days is off a lot, the current day is always correct ok
I run my script on linux using the “watch -n1
Code below
—————————————–
def get_data(tickers):
df = pdr.get_data_yahoo(tickers)
return df
for df in tickers:
df = get_data(df) # this function retrieve’s stock information from yahoo finance
df = pd.DataFrame(df.Close)
print(df.tail(5)
——————————————–
#1 — works correctly Close
Date Close
2021-08-15 11.2500
2021-09-16 9.9600
2021-09-17 11.0600
2021-09-20 11.7300
2021-09-21 13.7001
#2 — doesn’t work correctly Close
Date Close
2021-08-31 5.9600
2021-09-01 7.4000
2021-09-02 6.4900
2021-09-03 6.7000
2021-09-21 13.7001
how can anyone help me
It looks like you have a list of Tickers you iterate over. You get the data for all of the and print the last retrieved data.
The line taking Close in a DataFrame is maybe not needed?
What is the goal here? For each iteration see the last five days of the specific ticket?
If you share the list of tickers you are looking at and the full code I can take a look at it.
Please share a bit more what you want to achieve. Iterate over a list of tickers and print last five trading days?
I am getting information for one stock and that is the closing price and volume and I am using pandas-datareader get_data_yahoo() function, when I first run the stock one time it looks like the first photo, but what I am doing is using the “watch -n1 ” to get updates every 1 to 5 seconds so I get live updates, but after some time the data coming back goes back to September 09 2021 – August 31, 2021 like the second photo and the volume numbers don’t make any sense they are going into the billions, I have tried everything, different version of python3 + different versions of pip3 and pandas and pandas-datareader, the question is why does it keep going back to the last two weeks and why is my volume messed up, except for the current day., it only took 6 minutes for it to change to photo two
I think this will do what you want:
import pandas_datareader as pdr
from datetime import datetime, timedelta
while True:
df = pdr.get_data_yahoo(‘AAPL’, datetime.utcnow() — timedelta(days=7))
print(df)
Hi Bob,
Just make sure we have installed in the right environment.
Try to execute:
import pandas_datareader as pdr
print(pdr.__version__)
It should output: 0.10.0
If not, it is not installed (upgraded) in the right environment.
What environment are you using?
I am using Python 2.7 because it is what the author of an instruction video to scrape finance data from yahoo uses but I guess I’m gonna have to upgrade to Python 3 because I only have pandas 0.8.1.
Then you need to update your pandas-datareader.
Are you running the code from a command line or use a IDE like PyCharm?
I am using Spyder. I tried updating pandas but it said I already had the latest version so I guess 0.8.1 is the latest version for Python 2.7.
Ah yes – you need to update the pandas-datareader – see the command how to upgrade in the text above.
Hi, how can I check the version of pandas-datareader? I am using Anaconda distribution. After struggling for hours, I finally figured out how to install pandas-datareader (need to use conda install -c anaconda pandas-datareader in Anaconda Prompt).
However I still get the “Unable to read URL” error. I tried to check the datareader version using “print(dpr.__version__) but it says that there is no attribute ‘__version__’. Please help.
If you run this:
import pandas_datareader as pdr
pdr.__version__
In a cell and it does give you no attribute error, then you can run the following in a cell:
!pip freeze
Then you need to find the line with
It is ordered, so you should be able to locate it. (my output gives it for version 0.10.0, your’s might be different).
Hello Rune, I’m unable to get multiple tickers. It keeps giving me this error
ValueError Traceback (most recent call last)
/tmp/ipykernel_67023/2049989489.py in
2 start = dt.datetime(2022, 1, 1)
3
—-> 4 data = pdr.get_data_yahoo(tickers, start)
/usr/local/lib/python3.8/dist-packages/pandas_datareader/data.py in get_data_yahoo(*args, **kwargs)
78
79 def get_data_yahoo(*args, **kwargs):
—> 80 return YahooDailyReader(*args, **kwargs).read()
81
82
/usr/local/lib/python3.8/dist-packages/pandas_datareader/base.py in read(self)
256 df = self._dl_mult_symbols(self.symbols.index)
257 else:
–> 258 df = self._dl_mult_symbols(self.symbols)
259 return df
260
/usr/local/lib/python3.8/dist-packages/pandas_datareader/base.py in _dl_mult_symbols(self, symbols)
283 stocks[sym] = df_na
284 if PANDAS_0230:
–> 285 result = concat(stocks, sort=True).unstack(level=0)
286 else:
287 result = concat(stocks).unstack(level=0)
/usr/local/lib/python3.8/dist-packages/pandas/core/frame.py in unstack(self, level, fill_value)
8411 from pandas.core.reshape.reshape import unstack
8412
-> 8413 result = unstack(self, level, fill_value)
8414
8415 return result.__finalize__(self, method=”unstack”)
/usr/local/lib/python3.8/dist-packages/pandas/core/reshape/reshape.py in unstack(obj, level, fill_value)
476 if isinstance(obj, DataFrame):
477 if isinstance(obj.index, MultiIndex):
–> 478 return _unstack_frame(obj, level, fill_value=fill_value)
479 else:
480 return obj.T.stack(dropna=False)
/usr/local/lib/python3.8/dist-packages/pandas/core/reshape/reshape.py in _unstack_frame(obj, level, fill_value)
499 def _unstack_frame(obj, level, fill_value=None):
500 if not obj._can_fast_transpose:
–> 501 unstacker = _Unstacker(obj.index, level=level)
502 mgr = obj._mgr.unstack(unstacker, fill_value=fill_value)
503 return obj._constructor(mgr)
/usr/local/lib/python3.8/dist-packages/pandas/core/reshape/reshape.py in __init__(self, index, level, constructor)
138 )
139
–> 140 self._make_selectors()
141
142 @cache_readonly
/usr/local/lib/python3.8/dist-packages/pandas/core/reshape/reshape.py in _make_selectors(self)
190
191 if mask.sum() 192 raise ValueError(“Index contains duplicate entries, cannot reshape”)
193
194 self.group_index = comp_index
ValueError: Index contains duplicate entries, cannot reshape
Источник
‘utf-8’ Encoding + ‘unable to read URL’ errors: when downloading financial data from yahoo and google #428
Hi all, I am using Python 3 on Mac OS X. I would like to download S&P 500 data using yahoo finance (somehow it is still working but then throwing an error) and using google finance.
The code I am using
from datetime import *
import bs4 as bs
import datetime as dt
import os
import pandas as pd
import pandas_datareader.data as data
import pickle
import requests
SnP = pd.DataFrame()
start = dt.datetime(2000, 1, 1)
end = datetime.now()
for ticker in tickers:
—stock = data.DataReader(ticker, data_source, start, end)
—SnP[ticker] = stock[‘Open’]
- tickers is a list of stock tickers, i.e tickers = [‘MMM’, ‘ABT’, ‘ABBV’, ‘ACN’. ‘ZION’, ‘ZTS’]
- data_source is either ‘yahoo’ or ‘google’
Using yahoo API, i.e data_source = ‘yahoo’ the code works for the first few tickers then throws me the following error:
Traceback (most recent call last): File » «, line 2, in File «/usr/local/lib/python3.6/site-packages/pandas_datareader/data.py», line 121, in DataReader session=session).read() File «/usr/local/lib/python3.6/site-packages/pandas_datareader/yahoo/daily.py», line 115, in read df = super(YahooDailyReader, self).read() File «/usr/local/lib/python3.6/site-packages/pandas_datareader/base.py», line 181, in read params=self._get_params(self.symbols)) File «/usr/local/lib/python3.6/site-packages/pandas_datareader/base.py», line 79, in _read_one_data out = self._read_url_as_StringIO(url, params=params) File «/usr/local/lib/python3.6/site-packages/pandas_datareader/base.py», line 90, in _read_url_as_StringIO response = self._get_response(url, params=params) File «/usr/local/lib/python3.6/site-packages/pandas_datareader/base.py», line 139, in _get_response raise RemoteDataError(‘Unable to read URL: <0>‘.format(url)) pandas_datareader._utils.RemoteDataError: Unable to read URL: https://query1.finance.yahoo.com/v7/finance/download/AYI?period1=946681200&period2=1512946799&interval=1d&events=history&crumb=hnxNmU%5Cu002FEQJS
However, if only used with one ticker, the same code works perfectly fine, i.e
import pandas_datareader.data as data
start = datetime(2016, 12, 31)
end = datetime.now()
AAPL = data.DataReader(‘APPL’, ‘yahoo’, start, end)
When I use google API, i.e data_source = ‘google’ , I am getting this error:
Traceback (most recent call last): File » «, line 2, in File «/usr/local/lib/python3.6/site-packages/pandas_datareader/data.py», line 137, in DataReader session=session).read() File «/usr/local/lib/python3.6/site-packages/pandas_datareader/base.py», line 181, in read params=self._get_params(self.symbols)) File «/usr/local/lib/python3.6/site-packages/pandas_datareader/base.py», line 79, in _read_one_data out = self._read_url_as_StringIO(url, params=params) File «/usr/local/lib/python3.6/site-packages/pandas_datareader/base.py», line 98, in _read_url_as_StringIO out.write(bytes_to_str(text)) File «/usr/local/lib/python3.6/site-packages/pandas/compat/__init__.py», line 74, in bytes_to_str return b.decode(encoding or ‘utf-8’) UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xfc in position 18101: invalid start byte
Would you please help me out with error 1 and 2? I already googled for those errors, and saw related topics on github and . withtout success in making the code work.
ps: I am new to Python. Using it since a week.
The text was updated successfully, but these errors were encountered:
Источник
Yahoo Daily depricated? #549
Traceback (most recent call last):
File «C:UsersSamueDesktopPythonIDLEProjectsPractice1.py», line 13, in
df = web.DataReader (‘TSLA’ ,’yahoo’, start, end)
File «C:Program Files (x86)Python36-32libsite-packagespandas_datareaderdata.py», line 291, in DataReader
raise ImmediateDeprecationError(DEP_ERROR_MSG.format(‘Yahoo Daily’))
pandas_datareader.exceptions.ImmediateDeprecationError:
Yahoo Daily has been immediately deprecated due to large breaks in the API without the
introduction of a stable replacement. Pull Requests to re-enable these data
connectors are welcome.
The text was updated successfully, but these errors were encountered:
Hi,
I also met this problem, the following description for your information.
Yahoo! Finance has been immediately deprecated. Yahoo! substantially altered their API in late 2017 and the csv endpoint was retired.
Parts of Yahoo Finance (including the daily reader) were reinstated in #525. I would recommend downloading the latest development version, as described here.
Your request should succeed after doing so.
@addisonlynch I tried the new git version download and instal. Still, the problem persists as I can’t access the TSLA data. What should I do next?
Hey @addisonlynch the latest release which you linked still throws an immediate deprecation error when implemented. @naitikbhise What is the specific use case where the «Yahoo Daily» API is needed as opposed to other Financial/Stock Tick API’s provided like Robinhood, or Morningstar?
Источник
Hello, all,
Here I share a few tricks to solve this problem:
import pandas
from pandas_datareader import data as pdr
import yfinance as yfin
yfin.pdr_override()
then
df = pdr.get_data_yahoo(«TSLA», start=»yyyy-mm-dd», end=»yyyy-mm-dd»)
print(df)
the above workaround does not work anymore.. I get the following error:
Warning (from warnings module):
File «C:UsersBranAppDataLocalProgramsPythonPython38libsite-packagespandas_datareadercompat_init_.py», line 7
from pandas.util.testing import assert_frame_equal
FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
Exception in thread Thread-1:
Traceback (most recent call last):
File «C:UsersBranAppDataLocalProgramsPythonPython38libthreading.py», line 932, in _bootstrap_inner
self.run()
File «C:UsersBranAppDataLocalProgramsPythonPython38libthreading.py», line 870, in run
self._target(*self._args, **self.kwargs)
File «C:UsersBranAppDataLocalProgramsPythonPython38libsite-packagesmultitasking_init.py», line 102, in _run_via_pool
return callee(*args, **kwargs)
File «C:UsersBranAppDataLocalProgramsPythonPython38libsite-packagesyfinancemulti.py», line 166, in _download_one_threaded
data = _download_one(ticker, start, end, auto_adjust, back_adjust,
File «C:UsersBranAppDataLocalProgramsPythonPython38libsite-packagesyfinancemulti.py», line 178, in download_one
return Ticker(ticker).history(period=period, interval=interval,
File «C:UsersBranAppDataLocalProgramsPythonPython38libsite-packagesyfinancebase.py», line 156, in history
data = data.json()
File «C:UsersBranAppDataLocalProgramsPythonPython38libsite-packagesrequestsmodels.py», line 898, in json
return complexjson.loads(self.text, **kwargs)
File «C:UsersBranAppDataLocalProgramsPythonPython38libjson_init.py», line 357, in loads
return _default_decoder.decode(s)
File «C:UsersBranAppDataLocalProgramsPythonPython38libjsondecoder.py», line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File «C:UsersBranAppDataLocalProgramsPythonPython38libjsondecoder.py», line 355, in raw_decode
raise JSONDecodeError(«Expecting value», s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)