Error iterator should return strings not bytes did you open the file in text mode - Исправление ошибок и поиск оптимальных решений проблем

Sample.csv contains the following:

NAME    Id   No  Dept
Tom     1    12   CS
Hendry  2    35   EC
Bahamas 3    21   IT
Frank   4    61   EE

And the Python file contains the following code:

import csv
ifile  = open('sample.csv', "rb")
read = csv.reader(ifile)
for row in read :
    print (row)

When I run the above code in Python, I get the following exception:

File «csvformat.py», line 4, in
for row in read :
_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

How can I fix it?

asked Dec 15, 2011 at 4:18

You open the file in text mode.

More specifically:

ifile  = open('sample.csv', "rt", encoding=<theencodingofthefile>)

Good guesses for encoding is «ascii» and «utf8». You can also leave the encoding off, and it will use the system default encoding, which tends to be UTF8, but may be something else.

answered Dec 15, 2011 at 11:22

Lennart RegebroLennart Regebro

164k41 gold badges222 silver badges251 bronze badges

The reason it is throwing that exception is because you have the argument rb, which opens the file in binary mode. Change that to r, which will by default open the file in text mode.

Your code:

import csv
ifile  = open('sample.csv', "rb")
read = csv.reader(ifile)
for row in read :
    print (row)

New code:

import csv
ifile  = open('sample.csv', "r")
read = csv.reader(ifile)
for row in read :
    print (row)

Brad Koch

18.7k18 gold badges108 silver badges136 bronze badges

answered Mar 10, 2015 at 21:27

MMMMMM

1,1611 gold badge7 silver badges2 bronze badges

In Python3, csv.reader expects, that passed iterable returns strings, not bytes. Here is one more solution to this problem, that uses codecs module:

import csv
import codecs
ifile  = open('sample.csv', "rb")
read = csv.reader(codecs.iterdecode(ifile, 'utf-8'))
for row in read :
    print (row)

answered Aug 21, 2018 at 10:09

Your problem is you have the b in the open flag.
The flag rt (read, text) is the default, so, using the context manager, simply do this:

with open('sample.csv') as ifile:
    read = csv.reader(ifile) 
    for row in read:
        print (row)

The context manager means you don’t need generic error handling (without which you may get stuck with the file open, especially in an interpreter), because it will automatically close the file on an error, or on exiting the context.

The above is the same as:

with open('sample.csv', 'r') as ifile:
    ...

with open('sample.csv', 'rt') as ifile:
    ...

answered Feb 12, 2015 at 21:44

I had this error when running an old python script developped with Python 2.6.4

When updating to 3.6.2, I had to remove all ‘rb’ parameters from open calls in order to fix this csv reading error.

answered Jul 31, 2017 at 18:30

Michael FayadMichael Fayad

1,1741 gold badge16 silver badges37 bronze badges

This is a GOTCHA in Django. The open that comes with filefield always opens the file in byte mode as far as I can tell. You need to open with Python’s open instead.

So with avccont_datafile being an instance of a model with a field called datafile = django.db.models.FileField(), The following code….

with avccount_datafile.datafile.file.open('rt') as fp:
    self.avc_data = csv.DictReader(fp)

gives the error

_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

but this fixes it

filename = avccount_datafile.datafile.file.name
with open(filename, 'rt') as fp:
        self.avc_data = csv.DictReader(fp)

answered Feb 23, 2022 at 7:05

MagicLAMPMagicLAMP

97710 silver badges26 bronze badges

Источник

At the start of my csv program:

import csv     # imports the csv module
import sys      # imports the sys module

f = open('Address Book.csv', 'rb') # opens the csv file
try:
    reader = csv.reader(f)  # creates the reader object
    for row in reader:   # iterates the rows of the file in orders
        print (row)    # prints each row
finally:
    f.close()      # closing

And the error is:

    for row in reader:   # iterates the rows of the file in orders
_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

asked Mar 2, 2014 at 19:08

Instead of this (and the rest):

f = open('Address Book.csv', 'rb')

Do this:

with open('Address Book.csv', 'r') as f:
    reader = csv.reader(f) 
    for row in reader:
        print(row)

The context manager means you don’t need the finally: f.close(), because it will automatically close the file on an error, or on exiting the context.

answered Mar 18, 2014 at 0:14

The solution in this (duplicate?) question csv.Error: iterator should return strings, not bytes helped me:

f = open('Address Book.csv', "rt")

with open('Address Book.csv', "rt") as f:

or (using gzip)

import gzip
f = gzip.open('Address Book.csv', "rt")

import gzip
gzip.open('Address Book.csv', "rt") as f:

answered Aug 20, 2014 at 19:46

Garren SGarren S

5,3823 gold badges30 silver badges44 bronze badges

Источник

Iterator should return strings, not bytes (did you open the file in text mode?) error says that the csv.reader() function iterates through strings only, which are not provided. Thus, the issue arises due to the mode in which your file is opened.

If you want to clarify the same, read this article as it explains the problem and the remedies. By the end, you’ll know how to make your script work again.

Contents

Iterator Should Return Strings, Not Bytes: Causes Found
- – Csv.reader() Doesn’t Accept Files Opened in Binary Mode
- – Python 2.7 Doesn’t Support the open() Method’s Encoding Parameter
How To Suppress the Iterator Should Return Strings, Not Bytes Error?
- – Start Using the rt or r Mode
- – Use Codecs in Python 3
- – Leverage TextIOWrapper
- – Use the Built-in CSV Module
FAQ
- 1. How To Django Read inmemoryuploadedfile CSV?
- 2. What Is _CSV.Error: Line Contains Nul?
- 3. How Would You Differentiate Between the reader() and DictReader() Functions?
Conclusion
Reference

The iterator should return strings, not bytes error shows up when you try to read CSV from bytes Python using the csv.reader() function. You might do it mistakenly by not realizing that the file is opened in the byte mode and csv reader string is fond of strings only.

– Csv.reader() Doesn’t Accept Files Opened in Binary Mode

If you open the CSV file in the binary mode and try to read it through the csv.reader() function, you will get the iterator should return strings, not bytes error. It is because the csv.reader() function doesn’t accept bytes.

The given function accepts only the file and list objects that return strings every time their __next()__ method is called. Therefore, if the file is opened in the binary mode, it will return bytes instead of strings, which is unacceptable.

You can look at the problematic piece of code here.

import csv

new_file = open(‘my_file.csv’, “rb”)

content = csv.reader(new_file)

for row in content :

print (row)

– Python 2.7 Doesn’t Support the open() Method’s Encoding Parameter

You can’t specify the encoding type for the CSV file while opening it in Python 2.7 because the given version doesn’t support the encoding parameter of the open() method. So if the system default encoding doesn’t align with your file’s encoding type, the error might appear on your screen.

How To Suppress the Iterator Should Return Strings, Not Bytes Error?

You can suppress or resolve the iterator should return strings, not bytes error, by opening your CSV files in the r or rt mode, using codecs in Python 3, leveraging TextIOWrapper, or working with the built-in CSV module. The choice of the solution depends on your feasibility.

– Start Using the rt or r Mode

It would be best to set the mode to “rt” or “r” to urlopen text mode. Eventually, the file passed to the CSV reader Python function will return strings, not bytes. It will satisfy the demand of Python, and the mentioned error will vanish.

Note that the r mode represents the read mode, allowing you to read the files in the text mode by default. However, the rt mode has an optional “t” that indicates the text mode. Now, whichever mode you wish to use, you’ll be able to open the file and read it in text mode.

You can use the following code snippet for reference.

import csv

new_file = open(‘my_file.csv’, “rt”)

content = csv.reader(new_file)

for row in content :

print (row)

– Use Codecs in Python 3

You can call the codecs.iterdecode() function inside the csv.reader() function in Python 3 to push the error away from your system. The given function will decode your required file while allowing you to specify its encoding type. You’ll only need to pass the file object and encoding type.

Furthermore, it would be beneficial to note that the codecs.iterdecode() function uses an incremental decoder that decodes the input provided by the iterator iteratively. Also, you can only pass a binary file object to it.

Therefore, applying the current solution won’t ask you to change the file’s mode to r or rt while opening it.

Please feel free to use the given code block to apply the solution better.

import csv

import codecs

another_file = open(‘file1.csv’, “rb”)

read = csv.reader(codecs.iterdecode(another_file, ‘utf-8’))

for row in read :

print (row)

Lastly, don’t forget that the codecs.iterdecode() function isn’t a safe option for multibyte characters. Hence, there are mixed views about using it. You can expect it to provide perfect results or be prepared to receive the wrong output, such as newlines shifted to the next line.

– Leverage TextIOWrapper

Contrary to the codecs.iterdecode() function, using the io.TextIOWrapper class presents a better solution to fix the stated error. You’ll need to pass a binary file object and encoding type to it. Next, the csv.DictReader() function will help in iterating through the file and reading it.

The io.TextIOWrapper class works by wrapping up a binary file object and decoding it. It means that the csv.DictReader() function will get a decoded text file, which won’t result in the error mentioned above. In case you don’t know, the DictReader() function reads the first line of your CSV file.

Next, it uses each comma-separated value in the given line as a dictionary key. So, you can expect the columns in each subsequent row behave like dictionary values allowing you to access them through the respective field names or keys.

This solution will be quite helpful when you use a custom extraction method in babel to extract your required files. You must know that babel extracts the files in binary mode by default.

Therefore, extracting the given files in the text mode would be difficult. But if you use the io.TextIOWrapper class, you’ll find it easier to deal with the extracted binary files and resolve the same error.

You can study the code snippet shared below to understand the concept.

def extract_csv_file(fileobj, keywords, comment_tags, options):

import csv

import io

with io.TextIOWrapper(fileobj, encoding=’utf-8′) as my_file:

read = csv.DictReader(my_file, delimiter=’,’)

for row in read:

print(row)

– Use the Built-in CSV Module

If you are struggling while using the open() function in Python 2.7 and the “iterator should return strings, not bytes” error isn’t leaving your screen, then you should use the built-in CSV module. The CSV module has helped when the open() function fails without the encoding type.

You can use the following coding representation to fix the error in Python 2.7.

with open(students.csv, ‘r’) as students_file:

read = csv.reader(students_file)

for row in read:

print (row)

students_file.close()

FAQ

1. How To Django Read inmemoryuploadedfile CSV?

You can read the inmemoryuploadedfile CSV by using the BytesIO object. After you import BytesIO from io, you’ll have to request the file by typing “req_file = request.FILES” followed by “new_file = req_file[‘new_file’].” Lastly, read the file by running “nfile = BytesIO(new_file.read()).

2. What Is _CSV.Error: Line Contains Nul?

The line contains null error suggests that there might be a problem in your file relating to null bytes. So, check your file to see what comes inside it. Another way would be to call the repr() function to see the content of your file in a platform-independent way.

3. How Would You Differentiate Between the reader() and DictReader() Functions?

The main difference between the reader() and DictReader() functions is that the former works well with simple CSV files, while the latter is a perfect choice for reading large CSV files. Also, the DitcReader() function is more user-friendly than the reader() function.

Conclusion

The long error statement in the title results from the csv.reader() pushing back the binary file passed to it. The function knows how to iterate through the file or list objects returning strings, not bytes. You can revise the necessary details of this post by reading the following listicle:

The csv.reader() function will accept the files opened in r or rt mode and won’t frustrate you with the same error.
The codecs.iterdecode() passed inside the csv.reader() function can make it accept the binary file object but this approach isn’t safe for multibyte characters.
You can use the io.TextIOWrapper class to wrap your binary file and make things go in your favor.
The built-in CSV module can help you deal with the stated error and forms a Python 2.7 alternative to the open() function.

Lastly, it’s up to you if you want to change the opening mode of your file or move towards other solutions that make your binary file work.

Reference

https://stackoverflow.com/questions/8515053/csv-error-iterator-should-return-strings-not-bytes
https://stackoverflow.com/questions/51152023/how-to-use-python-csv-dictreader-with-a-binary-file-for-a-babel-custom-extract/51152810#51152810
https://github.com/jazzband/tablib/issues/423
https://forum.djangoproject.com/t/admin-csv-import-csv-error-iterator-should-return-strings-not-bytes/5114/3

Author
Recent Posts

Position Is Everything: Your Go-To Resource for Learn & Build: CSS,JavaScript,HTML,PHP,C++ and MYSQL.

Источник

Franek
Я думал, что цикл for перебирает по отдельным элементам (словам) только список или кортеж , но не строку.

И строку for перебирает тоже. Цикл for использует функцию iter() для перебора. А функция iter() вызывает метод __iter__() у объекта.

  
>>> 'abc'.__iter__
<method-wrapper '__iter__' of str object at 0xb74b16a0>
>>> 
>>> it = 'abc'.__iter__()
>>> it
<str_iterator object at 0xb74013ac>
>>> 
>>> next(it)
'a'
>>> next(it)
'b'
>>>

Franek
Насколько я понял, модуль csv возвращает строки.

Модуль ничего не возвращает, он предоставляет функции и классы для работы с форматом csv (вообще, с форматом dsv). А вот функция csv.reader() возвращает объект, у которого есть метод __iter__() . И этот метод __iter__() возвращает итератор по разделённым строкам.

  
>>> import csv
>>> 
>>> reader = csv.reader(open('/etc/passwd'), delimiter=':')
>>> reader
<_csv.reader object at 0xb740ce9c>
>>> 
>>> it = reader.__iter__()
>>> it
<_csv.reader object at 0xb740ce9c>
>>> 
>>> next(it)
['root', 'x', '0', '0', 'root', '/root', '/bin/bash']
>>> next(it)
['bin', 'x', '1', '1', 'bin', '/bin', '/sbin/nologin']
>>>

Тут видно, что reader.__iter__() возвращает сам себя, потому что сам является итератором — то есть содержит метод __next__(), который потом и вызывается на каждом шаге. (Функция next() вызывает метод __next__() у объекта.)

А вот метод __next__() уже и возвращает очередной элемент.

  
>>> it.__next__()
['daemon', 'x', '2', '2', 'daemon', '/sbin', '/sbin/nologin']
>>>

Отредактировано py.user.next (Дек. 4, 2016 15:40:17)

Источник

The error «_csv.error: iterator should return strings, not bytes(did you open the file in text mode?)» arises when the user tries to open a CSV file online with “urllib”, it will return bytes instead of text as expected, it is the cause. The easy solution to this error is to download the file on your local machine and use it directly without “urllib”

Photo by Nandhu Kumar

What does the error message mean #

The urllib.request module offers classes and functions that aid in opening URLs (primarily HTTP) in a complicated environment. These functions and classes include basic and digest authentication, redirections, cookies, and more. The HTTP requests sent by the urllib.request module utilizes HTTP/1.1 and contains the Connection:close header.

If no further data is required, data must be a specified object that specifies None. At the moment, only HTTP requests make use of data. Bytes, file-like objects, and iterables of bytes-like objects are among the supported object types. HTTPHandler will set the appropriate header fields based on the kind of data if neither the Content-Length nor the Transfer-Encoding header fields have been specified. Bytes objects will be sent using Content-Length.

On the other hand, if the user is trying to open a CSV file directly from the local machine he may face the same error if he will use the binary mode to open the file in some cases in the open module of python.

CSV file contains rows and columns when data will be returned into bytes or binary form the user will face this error.

What are the causes of this error #

To reproduce the error I will place some sample data in a CSV file and code to explain the reasons.

Data represented in CSV file in rows and columns.

import csv

ReadCSVfile  = open('CSVsample.csv', "rb")

read = csv.reader(ReadCSVfile)

for row in read :

    print (row)

In the above code, I tried to open a file name CSVsample.csv from my local machine in binary mode. When I will run this code, it will produce the error

File «ReadCSVfile.py», line 4, in for row in read : _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

It is because the iterable supplied to csv.reader is expected to return strings, not bytes.

Now I will jump to the next example of code to explain the same error with the urllib module. Try this code:

import csv

import urllib.request

url = "ftp://ftp.nadra.nlm.nih.gov.pk/pub/pmc/file_list.csv"

ftpstream = urllib.request.urlopen(url)

ReadCSVfile = csv.reader(ftpstream)

data = [row for row in ReadCSVfile]

It will produce the same error message as a result because urllib also will return bytes.

The Solution #

The solutions vary according to the scenario

Solution One #

The solution for the first example is:

The system default encoding, which is often UTF8, can also be disabled by the user, in which case it will use that instead or set the encoding as:

ReadCSVsample  = open('CSVsample.csv', "rt", encoding=)

ReadCSVsample  = open('CSVsample.csv', "rt", encoding=)

Another solution to this code is I used the mode to read the file “rb” which will read the file in binary mode and return it as a binary object. If the user will not place any mode by default it will be read as text and returned as text or just use “r” instead of “rb”

ReadCSVfile  = open('CSVsample.csv', "r")

Or add some futher code in the read line as:

read = csv.reader(codecs.iterdecode(ReadCSVfile, 'utf-8'))

Solution two #

In our second example I have used the urllib module to explain the reasons, and the solution to the urllib module when it will return byte type data we can manage it as:

import csv

import urllib.request

url = "ftp://ftp.nadra.nlm.nih.gov.pk/pub/pmc/file_list.csv"

ftpstream = urllib.request.urlopen(url)

ReadCSVfile = csv.reader(ftpstream.read().decode('utf-8'))  # using the right encoding

data = [row for row in ReadCSVfile]

It might be simpler to read the final line as.

data = list(ReadCSVfile)

Solution three #

It can be further simplified with this code

import codecs

from contextlib import closing

import csv

import requests

url = "ftp://ftp.nadra.nlm.nih.gov.pk/pub/pmc/file_list.csv"

with closing(requests.get(url, stream=True)) as r:

    reader = csv.reader(codecs.iterdecode(r.iter_lines(), 'utf-8'))

for row in reader:

    print row

To avoid needing to download the complete file from the network into memory first, streaming is used here as well, thanks to the requests package (which, if the file is huge, can take a while).

Solution four #

The response from post-request would be a byte object if the user is using the requests package and csv.

I first stored them as a string file in memory and decoded utf-8 before using the csv library.

import io

import csv

import requests

response = requests.post(url, data)

csv_bytes = response.content

ReadCSVfile = io.StringIO(csv_bytes.decode('utf-8'), newline='n')

reader = csv.reader(ReadCSVfile)

for row_list in reader:

    print(row_list)

str_file.close()

The Conclusion #

We have discussed details of the error «_csv.error: iterator should return strings, not bytes(did you open the file in text mode?)». Reasons for occurrence and the best possible solutions according to the scenarios. The user must have to choose the solution according to the scenario that he encountered.

Any questions from your side would be appreciated and addressed on priority. You are welcome to place your feedback, comments, and suggestions.

Источник

Question :

Read .csv file from URL into Python 3.x – _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

I’ve been struggling with this simple problem for too long, so I thought I’d ask for help. I am trying to read a list of journal articles from National Library of Medicine ftp site into Python 3.3.2 (on Windows 7). The journal articles are in a .csv file.

I have tried the following code:

import csv
import urllib.request

url = "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv"
ftpstream = urllib.request.urlopen(url)
csvfile = csv.reader(ftpstream)
data = [row for row in csvfile]

It results in the following error:

Traceback (most recent call last):
File "<pyshell#4>", line 1, in <module>
data = [row for row in csvfile]
File "<pyshell#4>", line 1, in <listcomp>
data = [row for row in csvfile]
_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

I presume I should be working with strings not bytes? Any help with the simple problem, and an explanation as to what is going wrong would be greatly appreciated.

Answer #1:

The problem relies on urllib returning bytes. As a proof, you can try to download the csv file with your browser and opening it as a regular file and the problem is gone.

A similar problem was addressed here.

It can be solved decoding bytes to strings with the appropriate encoding. For example:

import csv
import urllib.request

url = "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv"
ftpstream = urllib.request.urlopen(url)
csvfile = csv.reader(ftpstream.read().decode('utf-8'))  # with the appropriate encoding 
data = [row for row in csvfile]

The last line could also be: data = list(csvfile) which can be easier to read.

By the way, since the csv file is very big, it can slow and memory-consuming. Maybe it would be preferable to use a generator.

EDIT:
Using codecs as proposed by Steven Rumbalski so it’s not necessary to read the whole file to decode. Memory consumption reduced and speed increased.

import csv
import urllib.request
import codecs

url = "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv"
ftpstream = urllib.request.urlopen(url)
csvfile = csv.reader(codecs.iterdecode(ftpstream, 'utf-8'))
for line in csvfile:
    print(line)  # do something with line

Note that the list is not created either for the same reason.

Answer #2:

Even though there is already an accepted answer, I thought I’d add to the body of knowledge by showing how I achieved something similar using the requests package (which is sometimes seen as an alternative to urlib.request).

The basis of using codecs.itercode() to solve the original problem is still the same as in the accepted answer.

import codecs
from contextlib import closing
import csv
import requests

url = "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv"

with closing(requests.get(url, stream=True)) as r:
    reader = csv.reader(codecs.iterdecode(r.iter_lines(), 'utf-8'))
    for row in reader:
        print row

Here we also see the use of streaming provided through the requests package in order to avoid having to load the entire file over the network into memory first (which could take long if the file is large).

I thought it might be useful since it helped me, as I was using requests rather than urllib.request in Python 3.6.

Some of the ideas (e.g using closing()) are picked from this similar post

Answer #3:

I had a similar problem using requests package and csv.
The response from post request was type bytes.
In order to user csv library, first I a stored them as a string file in memory (in my case the size was small), decoded utf-8.

import io
import csv
import requests

response = requests.post(url, data)

# response.content is something like: 
# b'"City","Awb","Total"rn"Bucuresti","6733338850003","32.57"rn'    
csv_bytes = response.content

# write in-memory string file from bytes, decoded (utf-8)
str_file = io.StringIO(csv_bytes.decode('utf-8'), newline='n')
    
reader = csv.reader(str_file)
for row_list in reader:
    print(row_list)

# Once the file is closed,
# any operation on the file (e.g. reading or writing) will raise a ValueError
str_file.close()

Printed something like:

['City', 'Awb', 'Total']
['Bucuresti', '6733338850003', '32.57']

Answer #4:

urlopen will return a urllib.response.addinfourl instance for an ftp request.

For ftp, file, and data urls and requests explicity handled by legacy
URLopener and FancyURLopener classes, this function returns a
urllib.response.addinfourl object which can work as context manager…

>>> urllib2.urlopen(url)
<addinfourl at 48868168L whose fp = <addclosehook at 48777416L whose fp = <socket._fileobject object at 0x0000000002E52B88>>>

At this point ftpstream is a file like object, using .read() would return the contents however csv.reader requires an iterable in this case:

Defining a generator like so:

def to_lines(f):
    line = f.readline()
    while line:
        yield line
        line = f.readline()

We can create our csv reader like so:

reader = csv.reader(to_lines(ftps))

And with a url

url = "http://pic.dhe.ibm.com/infocenter/tivihelp/v41r1/topic/com.ibm.ismsaas.doc/reference/CIsImportMinimumSample.csv"

The code:

for row in reader: print row

Prints

>>> 
['simpleci']
['SCI.APPSERVER']
['SRM_SaaS_ES', 'MXCIImport', 'AddChange', 'EN']
['CI_CINUM']
['unique_identifier1']
['unique_identifier2']

Источник

– Csv.reader() Doesn’t Accept Files Opened in Binary Mode

– Python 2.7 Doesn’t Support the open() Method’s Encoding Parameter

How To Suppress the Iterator Should Return Strings, Not Bytes Error?

– Start Using the rt or r Mode

– Use Codecs in Python 3

– Leverage TextIOWrapper

– Use the Built-in CSV Module

FAQ

1. How To Django Read inmemoryuploadedfile CSV?

2. What Is _CSV.Error: Line Contains Nul?

3. How Would You Differentiate Between the reader() and DictReader() Functions?

Conclusion

Reference

Photo by Nandhu Kumar

Table of Contents:

What does the error message mean #

What are the causes of this error #

The Solution #

Solution One #

Solution two #

Solution three #

Solution four #

The Conclusion #

Question :

Answer #1:

Answer #2:

Answer #3:

Answer #4: