At age 30, Python has been considered a mature language. The programming language is hugely popular among data scientist and AI engineers thanks to its simplicity and easy syntax. Despite that, its vague errors usually makes new users pull their hair out to debug.
In this article, we will discuss about [Errno 32] Broken pipe – a popular error message you often see when interacting with the file system. By the end of the article, you will understand why it happens and how to avoid it as well as how to fix your code. You might also be interested in finding out more about other common error messages of Python, such as locale.Error: unsupported locale setting or unindent does not match any outer indentation level.
What causes «[Errno 32] Broken pipe» in Python?
«Broken pipe» is essentially an IOError error (short for input/output error), which happened at the Linux system level. It usually occurs when reading and writing files, or in other words, doing file input/output or network input/output (via sockets).
The corresponding Linux system error is EPIPE, excerpted from GNU libc error codes:
Macro: int EPIPE
“Broken pipe.” There is no process reading from the other end of a pipe. Every library function that returns this error code also generates a
SIGPIPEsignal; this signal terminates the program if not handled or blocked. Thus, your program will never actually seeEPIPEunless it has handled or blockedSIGPIPE.
From what we’ve just read, we know that [Errno 32] Broken pipe is caused by the system sending SIGPIPE signal, which is an inter-process communication mechanism of Linux.
For example, SIGINT is another signal used internally by Linux system. In Linux, Ctrl+C will send a SIGINT signal to end the process, or we can use the kill command to achieve the same effect.
Python does not ignore SIGPIPE by default. Instead, it translate the signal into an exception and raises IOError: [Errno 32] Broken pipe every time it receives a SIGPIPE.
[Errno 32] Broken pipe when pipe outputs in Linux terminal
If you encounter [Errno 32] Broken pipe when trying to pipe output of a Python script to another program such as the below example, read on.
python <filename>.py | headThis pipeline syntax will create a process that sends data upstream, and a process that reads data downstream. When the downstream does not need to read upstream data, it will send a SIGPIPE signal to the upstream process.
When downstream no longer needs to read upstream data? For example, the head command in the example only needs to read enough lines to tell the upstream that I no longer need to read it, and it will send the SIGPIPE signal to the upstream process.
When the upstream process is a Python program, an error such as IOError: [Errno 32] Broken pipe will occur.
Details about other cases : https://superuser.com/questions/554855/how-can-i-fix-a-broken-pipe-error
Avoid [Errno 32] Broken pipe by ignoring SIGPIPE
If you don’t care too much about properly catching SIGPIPE and just need to get things running quickly, add the code snippet below to the top of your Python program.
from signal import signal, SIGPIPE, SIG_DFL
#Ignore SIG_PIPE and don't throw exceptions on it... (http://docs.python.org/library/signal.html)
signal(SIGPIPE,SIG_DFL) What the code does is redirecting SIGPIPE signals to the default SIG_DFL, which the system usually ignore.
But beware, the Python manual on signal library warn against this type of handling SIGPIPE
Do not set
SIGPIPE’s disposition toSIG_DFLin order to avoidBrokenPipeError. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it.
Properly catch IOError to avoid [Errno 32] Broken pipe
Since [Errno 32] Broken pipe is actually a IOError, you can place a try/catch block to catch it like the code snippet below :
import sys, errno
try:
### IO operation ###
except IOError as e:
if e.errno == errno.EPIPE:
### Handle the error ###Possible solution for [Errno 32] Broken pipe in multi-process program.
In programs that uses worker processes to speed up processing and make use of multi-core CPUs, you can try reducing the number of the worker processes to see whether the error disappear or not.
A large number of worker processes may conflict with each other when they try to take control of system resources or the permission to write into disk.
Conclusion
Properly fixing [Errno 32] Broken pipe requires time and a close look at your code. Most of the time, you can safely ignore the problem if the Python program is relatively simple. Some other times, the solution involves in reducing worker processes in multi-process programs. But we hope that this article offer useful information and help you solve your problem.
В настоящее время Python считается зрелым языком программирования, который широко используется специалистами по обработке данных и инженерами по искусственному интеллекту (ИИ) из-за его простоты и легкочитаемого синтаксиса.
В данном руководстве мы обсудим [Errno 32] Broken pipe в Python, известное сообщение об ошибке, которое мы часто видим при взаимодействии с файловой системой. Мы разберем причину ее возникновения, а также способы ее избежать и исправить в коде.
«Сломанный канал» обычно считается ошибкой IOError (сокращение от «Ошибка ввода-вывода»), которая произошла на уровне системы Linux. Обычно она возникает при чтении и записи файлов или, другими словами, при выполнении ввода / вывода файлов или сетевого ввода / вывода (через сокеты).
Эквивалентная системная ошибка Linux – EPIPE, взятая из кодов ошибок GNU libc.
Макрос: int EPIPE
“Broken pipe.” означает, что на другом конце конвейера нет считывания процесса. Каждая функция библиотеки, вызывающая код ошибки, также выдает сигнал SIGPIPE; этот сигнал завершает программу, если не обрабатывается или не блокируется. Следовательно, программа никогда не отобразит EPIPE до тех пор, пока она не обработает или не заблокирует SIGPIPE.
Из приведенного выше утверждения мы можем сделать вывод, что система, отправляющая сигнал SIGPIPE, вызывает ошибку [Errno 32] Broken pipe в механизме межпроцессного взаимодействия Linux.
Например, система Linux внутренне использует другой сигнал, называемый SIGINT. В Linux команда Ctrl + C отправит сигнал SIGINT, чтобы завершить процесс, или мы можем использовать команду kill для достижения того же эффекта.
Python по умолчанию не игнорирует SIGPIPE. Однако он преобразует сигнал в исключение и вызывает ошибку – IOError: [Errno 32] Сломанный канал каждый раз, когда он получает SIGPIPE.
Ошибка “сломанный канал” при подключении к терминалу Linux
Всякий раз, когда мы сталкиваемся с ошибкой [Errno 32] Broken pipe при попытке передать вывод скрипта Python другой программе, например:
$ python file_name.py | head
Объяснение:
Вышеупомянутый синтаксис конвейера создаст процесс, отправляющий данные в восходящем направлении, и процесс, читающий данные в нисходящем направлении. Когда нисходящему потоку не нужно читать данные восходящего потока, он отправит сигнал SIGPIPE процессу восходящего потока.
Когда нисходящий поток не должен читать данные восходящего потока? Давайте разберемся в этом на примере. Команда head в этом примере должна прочитать достаточно строк, чтобы сообщить восходящему потоку, что нам больше не нужно его читать, и она отправит сигнал SIGPIPE процессу восходящего потока.
Всякий раз, когда восходящий процесс является программой Python, возникает ошибка типа IOError: [Errno 32] Broken pipe.
Как избежать ошибки “сломанный канал”?
Если мы не заботимся о правильном перехвате SIGPIPE и нам нужно быстро запустить процесс, вставьте следующий фрагмент кода в начало программы Python.
Синтаксис:
from signal import signal, SIGPIPE, SIG_DFL #Ignore SIG_PIPE and don't throw exceptions on it...(http://docs.python.org/library/signal.html) signal(SIGPIPE,SIG_DFL)
Объяснение:
В приведенном выше фрагменте кода мы перенаправили сигналы SIGPIPE на стандартный SIG_DFL, который система обычно игнорирует.
Однако рекомендуется остерегаться руководства Python по библиотеке сигналов, чтобы предостеречь от такой обработки SIGPIPE.
Перехват IOError во избежание ошибки Broken pipe
Поскольку ошибка Broken pipe является ошибкой IOError, мы можем разместить блок try / catch, чтобы ее перехватить, как показано в следующем фрагменте кода:
Синтаксис:
import sys, errno
try:
### IO operation ###
except IOError as e:
if e.errno == errno.EPIPE:
### Handle the error ###
Объяснение:
В приведенном выше фрагменте кода мы импортировали модуль sys и errno и разместили блок try / catch, чтобы перехватить возникшее исключение и обработать его.
Возможное решение проблемы в многопроцессорной программе
В программах, которые используют рабочие процессы для ускорения обработки и многоядерные процессоры, мы можем попытаться уменьшить количество рабочих процессов, чтобы проверить, сохраняется ли ошибка или нет.
Большое количество рабочих процессов может конфликтовать друг с другом при попытке взять под контроль ресурсы системы или разрешение на запись на диск.
Изучаю Python вместе с вами, читаю, собираю и записываю информацию опытных программистов.
In this article, we will discuss Pipe Error in python starting from how an error is occurred in python along with the type of solution needed to be followed to rectify the error in python. So, let’s go into this article to understand the concept well.
With the advancement of emerging technologies in the IT sector, the use of programming language is playing a vital role. Thus the proper language is considered for the fast executions of the functions. In such a case, Python emerges as the most important language to satisfy the needs of the current problem execution because of its simplicity and availability of various libraries. But along with the execution, the errors during the execution also comes into existence and it becomes difficult for the programmers to rectify the errors for the processing of the problem.
The Emergence of Broken Pipe Error
A broken Pipe Error is generally an Input/Output Error, which is occurred at the Linux System level. The error has occurred during the reading and writing of the files and it mainly occurs during the operations of the files. The same error that occurred in the Linux system is EPIPE, but every library function which returns its error code also generates a signal called SIGPIPE, this signal is used to terminate the program if it is not handled or blocked. Thus a program will never be able to see the EPIPE error unless it has handled or blocked SIGPIPE.
Python interpreter is not capable enough to ignore SIGPIPE by default, instead, it converts this signal into an exception and raises an error which is known as IOError(INPUT/OUTPUT error) also know as ‘Error 32’ or Broken Pipe Error.
Broken Pipe Error in Python terminal
python <filename>.py | head
This pipeline code written above will create a process that will send the data upstream and a process that reads the data downstream. But when the downstream process will not be able to read the data upstream, it will raise an exception by sending SIGPIPE signal to the upstream process. Thus upstream process in a python problem will raise an error such as IOError: Broken pipe error will occur.
Example:
Python3
for i in range(4000):
print(i)
When we run this file from unix commands:
python3 main.py | head -n3000
Procedure to avoid Broken Pipe Error
Approach 1: To avoid the error we need to make the terminal run the code efficiently without catching the SIGPIPE signal, so for these, we can add the below code at the top of the python program.
from signal import signal, SIGPIPE, SIG_DFL signal(SIGPIPE,SIG_DFL)
Python3
from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE,SIG_DFL)
for i in range(4000):
print(i)
Output:
0 1 20 1 2 3 4 5 6 7 8 9 3 4 5 6 7 8 9
Explanation:
The above code which is placed on the top of the python code is used to redirect the SIGPIPE signals to the default SIG_DFL signal, which the system generally ignores so that the rest part of the code can be executed seamlessly. But Approach 11 is not effective because in the Python manual on the signal library, which is mentioned that this type of signal handling should be avoided and should not be practiced in any part of the code. So for this reason we will go for the second approach.
Approach 2: We can handle this type of error by using the functionality of try/catch block which is already approved by the python manual and is advised to follow such procedure to handle the errors.
import sys, errno
try:
# INPUT/OUTPUT operation #
except IOError as e:
if e.errno == errno.EPIPE:
# Handling of the error
Example:
Python3
import sys
import errno
try:
for i in range(4000):
print(i)
except IOError as e:
if e.errno == errno.EPIPE:
pass
Output:
0 1 2 3 4 5 6 7 8 9
Explanation:
In the above piece of code, we have used the built-in library of python which is the Sys and Errno module, and use the try/catch block in order to catch the raised SIGPIPE exception and handle it before it stops the program execution.
Nowadays, Python is considered a mature programming language that has been used popularly by data scientists and Artificial Intelligence (AI) engineers because of its simplicity and easy-to-read syntax. Apart from this, the vague errors of the programming language generally make new programmers pull their hair out to debug.
In the following tutorial, we will discuss [Errno 32] Broken pipe, a famous error message we have often seen while interacting with the file system. We will also understand the cause of this, along with the methods to avoid it and fix it in the code.
What causes «[Errno 32] Broken pipe» in Python?
«Broken pipe» is usually considered an IOError (short for Input/Output Error) error, which occurred at the Linux system level. It is generally raised while reading and writing files, or in other terms, performing file input/output or network input/output (through sockets).
The equivalent Linux system error is EPIPE, excerpted from GNU libc error codes.
Macro: int EPIPE
«Broken pipe.» there is no process reading from the other end of a pipe. Every function of the library raising error code also produces a SIGPIPE signal; this signal terminates the program if not handled or blocked. Hence, the program will never actually display EPIPE until it has handled or blocked SIGPIPE.
From the above statement, we can conclude that the system sending SIGPIPE signal causes the [Errno 32] Broken pipe error, an inter-process communication mechanism of Linux.
For instance, the Linux system uses another signal called SIGINT internally. In Linux, the command Ctrl+C will send a SIGINT signal in order to end the process, or we can utilize the kill command in order to achieve the same effect.
Python does not disregard SIGPIPE by default. However, it translates the signal into an exception and raises an error — IOError: [Errno 32] Broken pipe every time it receives a SIGPIPE.
Broken Pipe error when piping results in Linux terminal
Whenever we encounter [Errno 32] Broken pipe error while attempting to pipe the output of a Python script to another program, such as shown in the following example:
Example:
Explanation:
The above syntax of the pipeline will create a process sending data upstream and a process reading data downstream. When the downstream does not have to read upstream data, it will send a SIGPIPE signal to the process of upstream.
When does downstream not have to read upstream data? Let us understand this with an example. The head command in the example only has to read enough lines in order to tell the upstream that we no longer have to read it, and it will send the SIGPIPE signal to the process of upstream.
Whenever the upstream process is a Python program, an error like IOError: [Errno 32] Broken pipe will happen.
How to avoid Broken pipe errors?
If we do not care about properly catching SIGPIPE and have to get things running rapidly, insert the following snippet of code to the top of the Python program.
Syntax:
Explanation:
In the above snippet of code, we have redirected the SIGPIPE signals to the default SIG_DFL, which the system generally ignore.
However, it is advised to beware of the Python manual on the signal library to warn against this handling SIGPIPE.
Properly catching IOError to avoid Broken pipe error
As a Broken pipe error is an IOError error, we can place a try/catch block in order to catch it, as shown in the following snippet of code:
Syntax:
Explanation:
In the above snippet of code, we have imported the sys and errno module and placed the try/catch block in order to catch the raised exception and handle it.
A possible solution for Broken pipe error in the multi-process program
In programs that utilize worker processes to speed up processing and make utilization of multi-core CPUs, we can attempt to reduce the count of the worker processes to check whether the error remains or not.
A large number of worker processes may conflict with each other while trying to take control of resources of the system or the permission in order to write into the disk.
@alykhantejani
-
The code link was :https://github.com/andreasveit/triplet-network-pytorch/blob/master/train.py
-
The error ocured in train.py — 136
-
The error was:
runfile(‘G:/researchWork2/pytorch/triplet-network-pytorch-master/train.py’, wdir=’G:/researchWork2/pytorch/triplet-network-pytorch-master’)
Reloaded modules: triplet_mnist_loader, triplet_image_loader, tripletnet
Number of params: 21840
Traceback (most recent call last):
File «», line 1, in
runfile(‘G:/researchWork2/pytorch/triplet-network-pytorch-master/train.py’, wdir=’G:/researchWork2/pytorch/triplet-network-pytorch-master’)
File «D:Anaconda3libsite-packagesspyderutilssitesitecustomize.py», line 880, in runfile
execfile(filename, namespace)
File «D:Anaconda3libsite-packagesspyderutilssitesitecustomize.py», line 102, in execfile
exec(compile(f.read(), filename, ‘exec’), namespace)
File «G:/researchWork2/pytorch/triplet-network-pytorch-master/train.py», line 258, in
main()
File «G:/researchWork2/pytorch/triplet-network-pytorch-master/train.py», line 116, in main
train(train_loader, tnet, criterion, optimizer, epoch)
File «G:/researchWork2/pytorch/triplet-network-pytorch-master/train.py», line 137, in train
for batch_idx, (data1, data2) in enumerate(train_loader):
File «D:Anaconda3libsite-packagestorchutilsdatadataloader.py», line 303, in iter
return DataLoaderIter(self)
File «D:Anaconda3libsite-packagestorchutilsdatadataloader.py», line 162, in init
w.start()
File «D:Anaconda3libmultiprocessingprocess.py», line 105, in start
self._popen = self._Popen(self)
File «D:Anaconda3libmultiprocessingcontext.py», line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File «D:Anaconda3libmultiprocessingcontext.py», line 322, in _Popen
return Popen(process_obj)
File «D:Anaconda3libmultiprocessingpopen_spawn_win32.py», line 65, in init
reduction.dump(process_obj, to_child)
File «D:Anaconda3libmultiprocessingreduction.py», line 60, in dump
ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe
- Some part of train related codes as follows:
def train(train_loader, tnet, criterion, optimizer, epoch):
losses = AverageMeter()
accs = AverageMeter()
emb_norms = AverageMeter()
switch to train mode
tnet.train()
for batch_idx, (data1, data2, data3) in enumerate(train_loader):
if args.cuda:
data1, data2, data3 = data1.cuda(), data2.cuda(), data3.cuda()
data1, data2, data3 = Variable(data1), Variable(data2), Variable(data3)
# compute output
dista, distb, embedded_x, embedded_y, embedded_z = tnet(data1, data2, data3)
# 1 means, dista should be larger than distb
target = torch.FloatTensor(dista.size()).fill_(1)
if args.cuda:
target = target.cuda()
target = Variable(target)
loss_triplet = criterion(dista, distb, target)
loss_embedd = embedded_x.norm(2) + embedded_y.norm(2) + embedded_z.norm(2)
loss = loss_triplet + 0.001 * loss_embedd
# measure accuracy and record loss
acc = accuracy(dista, distb)
losses.update(loss_triplet.data[0], data1.size(0))
accs.update(acc, data1.size(0))
emb_norms.update(loss_embedd.data[0]/3, data1.size(0))
# compute gradient and do optimizer step
optimizer.zero_grad()
loss.backward()
optimizer.step()
if batch_idx % args.log_interval == 0:
print('Train Epoch: {} [{}/{}]t'
'Loss: {:.4f} ({:.4f}) t'
'Acc: {:.2f}% ({:.2f}%) t'
'Emb_Norm: {:.2f} ({:.2f})'.format(
epoch, batch_idx * len(data1), len(train_loader.dataset),
losses.val, losses.avg,
100. * accs.val, 100. * accs.avg, emb_norms.val, emb_norms.avg))
log avg values to somewhere
plotter.plot(‘acc’, ‘train’, epoch, accs.avg)
plotter.plot(‘loss’, ‘train’, epoch, losses.avg)
plotter.plot(’emb_norms’, ‘train’, epoch, emb_norms.avg)
Thank you so much.
BrokenPipeError at /ru/api/v1/my_url/
[Errno 32] Broken pipe
Request Method: POST
Request URL: https://apihrc.wienerdeming.com/ru/api/v1/my_url/
Django Version: 2.0
Python Executable: /my_project/.venv/bin/uwsgi
Python Version: 3.6.1
Python Path: [‘.’, », ‘/my_project/.venv/lib64/python36.zip’, ‘/my_project/.venv/lib64/python3.6’, ‘/my_project/.venv/lib64/python3.6/lib-dynload’, ‘/usr/lib64/python3.6’, ‘/usr/lib/python3.6’, ‘/my_project/.venv/lib/python3.6/site-packages’, ‘/my_project’, ‘/my_project/apps’]
Server time: Пн, 30 Июл 2018 16:02:43 +0500
Installed Applications:
[‘django.contrib.admin’,
‘django.contrib.auth’,
‘django.contrib.contenttypes’,
‘django.contrib.sessions’,
‘django.contrib.messages’,
‘django.contrib.staticfiles’,
‘django.contrib.gis’,
‘easy_thumbnails’,
‘django_filters’,
‘rest_framework’,
‘dynamic_rest’,
‘rest_framework.authtoken’,
‘easy_pdf’,
‘corsheaders’,
‘parler’,
*’my_apps’
]
Installed Middleware:
[‘raven.contrib.django.middleware.SentryMiddleware’,
‘django.middleware.security.SecurityMiddleware’,
‘django.contrib.sessions.middleware.SessionMiddleware’,
‘django.middleware.locale.LocaleMiddleware’,
‘corsheaders.middleware.CorsMiddleware’,
‘django.middleware.common.CommonMiddleware’,
‘django.middleware.csrf.CsrfViewMiddleware’,
‘django.contrib.auth.middleware.AuthenticationMiddleware’,
‘django.contrib.messages.middleware.MessageMiddleware’,
‘django.middleware.clickjacking.XFrameOptionsMiddleware’,
‘system.utils.middleware.RequestLogsMiddleware’]
Traceback:
File «/my_project/.venv/lib/python3.6/site-packages/django/core/handlers/exception.py» in inner
35. response = get_response(request)
File «/my_project/.venv/lib/python3.6/site-packages/django/core/handlers/base.py» in _get_response
128. response = self.process_exception_by_middleware(e, request)
File «/my_project/.venv/lib/python3.6/site-packages/django/core/handlers/base.py» in _get_response
126. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File «/my_project/.venv/lib/python3.6/site-packages/django/views/decorators/csrf.py» in wrapped_view
54. return view_func(*args, **kwargs)
File «/my_project/.venv/lib/python3.6/site-packages/rest_framework/viewsets.py» in view
95. return self.dispatch(request, *args, **kwargs)
File «/my_project/.venv/lib/python3.6/site-packages/rest_framework/views.py» in dispatch
494. response = self.handle_exception(exc)
File «/my_project/.venv/lib/python3.6/site-packages/rest_framework/views.py» in handle_exception
454. self.raise_uncaught_exception(exc)
File «/my_project/.venv/lib/python3.6/site-packages/rest_framework/views.py» in dispatch
491. response = handler(request, *args, **kwargs)
File «/my_project/.venv/lib/python3.6/site-packages/rest_framework/mixins.py» in create
21. self.perform_create(serializer)
File «/my_project/.venv/lib/python3.6/site-packages/rest_framework/mixins.py» in perform_create
26. serializer.save()
File «/my_project/.venv/lib/python3.6/site-packages/rest_framework/serializers.py» in save
214. self.instance = self.create(validated_data)
File «/usr/lib64/python3.6/contextlib.py» in inner
53. return func(*args, **kwds)
File «/my_project/apps/applications/serializers/application_action.py» in create
83. self._submit_to_client(application, validated_data.get(‘comment’, «»))
File «/my_project/apps/applications/serializers/application_action.py» in _submit_to_client
113. send = gmail.messages.send_message(message=message)
File «/my_project/apps/mailboxes/utils/gmail/messages.py» in send_message
129. message = self.service.users().messages().send(userId=’me’, body=message).execute()
File «/my_project/.venv/lib/python3.6/site-packages/googleapiclient/_helpers.py» in positional_wrapper
130. return wrapped(*args, **kwargs)
File «/my_project/.venv/lib/python3.6/site-packages/googleapiclient/http.py» in execute
835. method=str(self.method), body=self.body, headers=self.headers)
File «/my_project/.venv/lib/python3.6/site-packages/googleapiclient/http.py» in _retry_request
179. raise exception
File «/my_project/.venv/lib/python3.6/site-packages/googleapiclient/http.py» in _retry_request
162. resp, content = http.request(uri, method, *args, **kwargs)
File «/my_project/.venv/lib/python3.6/site-packages/oauth2client/transport.py» in new_request
175. redirections, connection_type)
File «/my_project/.venv/lib/python3.6/site-packages/oauth2client/transport.py» in request
282. connection_type=connection_type)
File «/my_project/.venv/lib/python3.6/site-packages/httplib2/__init__.py» in request
1322. (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File «/my_project/.venv/lib/python3.6/site-packages/httplib2/__init__.py» in _request
1072. (response, content) = self._conn_request(conn, request_uri, method, body, headers)
File «/my_project/.venv/lib/python3.6/site-packages/httplib2/__init__.py» in _conn_request
996. conn.request(method, request_uri, body, headers)
File «/usr/lib64/python3.6/http/client.py» in request
1239. self._send_request(method, url, body, headers, encode_chunked)
File «/usr/lib64/python3.6/http/client.py» in _send_request
1285. self.endheaders(body, encode_chunked=encode_chunked)
File «/usr/lib64/python3.6/http/client.py» in endheaders
1234. self._send_output(message_body, encode_chunked=encode_chunked)
File «/usr/lib64/python3.6/http/client.py» in _send_output
1065. self.send(chunk)
File «/usr/lib64/python3.6/http/client.py» in send
986. self.sock.sendall(data)
File «/usr/lib64/python3.6/ssl.py» in sendall
965. v = self.send(data[count:])
File «/usr/lib64/python3.6/ssl.py» in send
935. return self._sslobj.write(data)
File «/usr/lib64/python3.6/ssl.py» in write
636. return self._sslobj.write(data)
Exception Type: BrokenPipeError at /ru/api/v1/my_url/
Exception Value: [Errno 32] Broken pipe
Request information:
USER: admin2
This article illustrates an error resolution for a Python prompt [Errno 32] that causes a thread crash. Share with you for your reference. Specific methods are as follows:
1. Error phenomena
The HTTP service implemented by ThreadingHTTPServer, if the client actively disconnects before the server returns, the server will report an error [Errno 32] Broken pipe and cause the processing thread to crash.
Let’s start with an example,python version: 2.7
The sample code
#!/usr/bin/env python
#!coding=utf-8
import os
import time
import socket
import threading
from BaseHTTPServer import HTTPServer ,BaseHTTPRequestHandler
from SocketServer import ThreadingMixIn
class RequestHandler(BaseHTTPRequestHandler):
def do_GET(self):
"""
To deal with get request
"""
query=self.path
print "query: %s thread=%s" % (query, str(threading.current_thread()))
#ret_str="<html>" + self.path + "<br>" + str(self.server) + "<br>" + str(self.responses) + "</html>"
ret_str="<html>" + self.path + "<br>" + str(self.server) + "</html>"
time.sleep(5)
try:
self.send_response(200)
self.send_header('Content-type','text/html')
self.end_headers()
self.wfile.write(ret_str)
except socket.error, e:
print "socket.error : Connection broke. Aborting" + str(e)
self.wfile._sock.close() # close socket
self.wfile._sock=None
return False
print "success prod query :%s" % (query)
return True
# multithreading
class ThreadingHTTPServer(ThreadingMixIn,HTTPServer):
pass
if __name__ == '__main__':
serveraddr = ('',9001)
ser = ThreadingHTTPServer(serveraddr,RequestHandler)
ser.serve_forever()
sys.exit(0)
Run the service
. / thread_http_server_error. Py
Curl for the first time, waiting to return
[~]$curl -s 'http://10.232.41.142:9001/hello1 '
<html>/hello1<br><__main__.ThreadingHTTPServer instance at 0x37483b0></html>[~]$
At this point, the server-side output log is as follows:
$./thread_http_server_error.py
query: /hello1 thread=
search041142.sqa.cm4.tbsite.net � - [15/May/2014 15:02:27] " GET /hello1 HTTP/1.1 " 200 -
success prod query :/hello1
Curl the second time, without waiting to return, CTRL +C to simulate client disconnection
[~]$curl -s 'http://10.232.41.142:9001/hello2 '
[~]$ ctrl+CAt this point, the server-side output log is as follows:
query: /hello2 thread=
search041142.sqa.cm4.tbsite.net � - [15/May/2014 15:33:10] " GET /hello2 HTTP/1.1 " 200 -
socket.error : Connection broke. Aborting[Errno 32] Broken pipe
-- -- -- -- -- -- - -
Exception happened during processing of request from ('10.232.41.142 ' , 48769)
Traceback (most recent call last):
File " /home/wuzhu/tools/python_2_7_1/lib/python2.7/SocketServer.py " , line 582, in process_request_thread
self.finish_request(request, client_address)
File " /home/wuzhu/tools/python_2_7_1/lib/python2.7/SocketServer.py " , line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File " /home/wuzhu/tools/python_2_7_1/lib/python2.7/SocketServer.py " , line 639, in __init__
self.handle()
File " /home/wuzhu/tools/python_2_7_1/lib/python2.7/BaseHTTPServer.py " , line 337, in handle
self.handle_one_request()
File " /home/wuzhu/tools/python_2_7_1/lib/python2.7/BaseHTTPServer.py " , line 326, in handle_one_request
self.wfile.flush() #actually send the response if not already done.
File " /home/wuzhu/tools/python_2_7_1/lib/python2.7/socket.py " , line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
AttributeError: 'NoneType' object has no attribute 'sendall'
2. Cause analysis
The reason for «[Errno 32] Broken pipe» is quite clear. Since the client actively disconnected the server before it returned, the server received SIGPIPE error when it wrote socket. Although we also handled the exception in our program by closing the handler’s wfile._sock object, the member function handle_one_request of the BaseHTTPRequestHandler class in the python library basehttpserver.py calls wfile.flush directly without determining whether the wfile is close.
def handle_one_request(self):
"""Handle a single HTTP request.
You normally don't need to override this method; see the class
__doc__ string for information on how to handle specific HTTP
commands such as GET and POST.
"""
try:
self.raw_requestline = self.rfile.readline(65537)
if len(self.raw_requestline) > 65536:
self.requestline = ''
self.request_version = ''
self.command = ''
self.send_error(414)
return
if not self.raw_requestline:
self.close_connection = 1
return
if not self.parse_request():
# An error code has been sent, just exit
return
mname = 'do_' + self.command
if not hasattr(self, mname):
self.send_error(501, "Unsupported method (%r)" % self.command)
return
method = getattr(self, mname)
method()
# No judgment wfile Whether they have been close Just call directly flush()
self.wfile.flush() #actually send the response if not already done.
except socket.timeout, e:
#a read or a write timed out. Discard this connection
self.log_error("Request timed out: %r", e)
self.close_connection = 1
return
3. Solutions
Simply override the member function handle_one_reques () of RequestHandler’s base class BaseHTTPRequestHandler and add whether the wfile is close before calling wfile.flush().
#!/usr/bin/env python
#!coding=utf-8
import os
import time
import socket
import threading
from BaseHTTPServer import HTTPServer ,BaseHTTPRequestHandler
from SocketServer import ThreadingMixIn
class RequestHandler(BaseHTTPRequestHandler):
def handle_one_request(self):
"""Handle a single HTTP request.
You normally don't need to override this method; see the class
__doc__ string for information on how to handle specific HTTP
commands such as GET and POST.
"""
try:
self.raw_requestline = self.rfile.readline(65537)
if len(self.raw_requestline) > 65536:
self.requestline = ''
self.request_version = ''
self.command = ''
self.send_error(414)
return
if not self.raw_requestline:
self.close_connection = 1
return
if not self.parse_request():
# An error code has been sent, just exit
return
mname = 'do_' + self.command
if not hasattr(self, mname):
self.send_error(501, "Unsupported method (%r)" % self.command)
return
method = getattr(self, mname)
print "before call do_Get"
method()
# increase debug info and wfile To determine whether or not close
print "after call do_Get"
if not self.wfile.closed:
self.wfile.flush() #actually send the response if not already done.
print "after wfile.flush()"
except socket.timeout, e:
#a read or a write timed out. Discard this connection
self.log_error("Request timed out: %r", e)
self.close_connection = 1
return
def do_GET(self):
"""
To deal with get request
"""
query=self.path
print "query: %s thread=%s" % (query, str(threading.current_thread()))
ret_str="<html>" + self.path + "<br>" + str(self.server) + "</html>"
time.sleep(5)
try:
self.send_response(200)
self.send_header('Content-type','text/html')
self.end_headers()
self.wfile.write(ret_str)
except socket.error, e:
print "socket.error : Connection broke. Aborting" + str(e)
self.wfile._sock.close()
self.wfile._sock=None
return False
print "success prod query :%s" % (query)
return True
# multithreading
class ThreadingHTTPServer(ThreadingMixIn,HTTPServer):
pass
if __name__ == '__main__':
serveraddr = ('',9001)
ser = ThreadingHTTPServer(serveraddr,RequestHandler)
ser.serve_forever()
sys.exit(0)
Run the service
. / thread_http_server. Py
Curl, without waiting to return, CTRL +C to simulate client disconnection
[~]$curl -s 'http://10.232.41.142:9001/hello2'
[~]$ ctrl+CAt this point, the server-side output log is as follows:
$./thread_http_server.pybefore call do_Get
query: /hello2 thread=<Thread(Thread-1, started 1103210816)>
search041142.sqa.cm4.tbsite.net - - [15/May/2014 15:54:09] "GET /hello2 HTTP/1.1" 200 -
socket.error : Connection broke. Aborting[Errno 32] Broken pipe
after call do_Get
after wfile.flush()I hope this article has helped you with your Python programming.
Have you ever seen a socket.error: [Errno 32] Broken pipe message when running a Python Web server and wondered what that means?
The rule is that when a process tries to write to a socket that has already received an RST packet, the SIGPIPE signal is sent to that process which causes the Broken pipe socket.error exception.
Here are two scenarios that you can try that cause SIGPIPE signal to be fired.
1. Server may send an RST packet to a client to
abort the socket connection but the client ignores the packet and
continues to write to the socket.
To test that behavior install Cynic, run it
$ cynic INFO [2012-06-08 05:06:37,040] server: Starting 'HTTPHtmlResponse' on port 2000 INFO [2012-06-08 05:06:37,040] server: Starting 'HTTPJsonResponse' on port 2001 INFO [2012-06-08 05:06:37,040] server: Starting 'HTTPNoBodyResponse' on port 2002 INFO [2012-06-08 05:06:37,040] server: Starting 'HTTPSlowResponse' on port 2003 INFO [2012-06-08 05:06:37,040] server: Starting 'RSTResponse' on port 2020 INFO [2012-06-08 05:06:37,040] server: Starting 'RandomDataResponse' on port 2021 INFO [2012-06-08 05:06:37,040] server: Starting 'NoResponse' on port 2022 INFO [2012-06-08 05:06:37,041] server: Starting 'LogRecordHandler' on port /tmp/_cynic.sock
and then run the client1.py:
import socket
# connect to Cynic's RSTResponse service on port 2020
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('', 2020))
# first read gets an RST packet
try:
s.recv(1024)
except socket.error as e:
print e
print
# write after getting the RST causes SIGPIPE signal
# to be sent to this process which causes a socket.error
# exception
s.send('hello')
and see what happens:
$ python ./client1.py
[Errno 104] Connection reset by peer
Traceback (most recent call last):
File "./client1.py", line 17, in
s.send('hello')
socket.error: [Errno 32] Broken pipe2. Server can send an RST to the client’s SYN
request to indicate that there is no process wating for connections on
the host at the specified port, but the client tries to write to the
socket anyway.
To test it run the client2.py:
import socket
# connect to the port that nobody is listening on
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# gets us an RST packet
try:
s.connect(('', 30301))
except socket.error as e:
print e
print
# write after getting RST causes SIGPIPE signal
# to be sent to this process which causes an exception
s.send('hello')
Here is the output:
$ python ./client2.py
[Errno 111] Connection refused
Traceback (most recent call last):
File "./sigpipe2.py", line 15, in
s.send('hello')
socket.error: [Errno 32] Broken pipeI hope that clarifies a SIGPIPE’s nature a little bit.