Request failed error reading the headers - Исправление ошибок и поиск оптимальных решений проблем

Recently I moved my servers from one provider to the other and started to get this messages in apache error log:
«request failed: error reading the headers»

Example from the error and corresponding access apache logs:

ApacheServer$ cat  error_log
ApacheServer [Tue Jan 20 11:07:44 2015] [error] [client x.x.x.x] request failed: error reading the headers

ApacheServer$ cat  access_log
x.x.x.x - - [20/Jan/2015:11:06:44 +0200] "GET SomeRandomRequest HTTP/1.1" 400 226 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729; BRI/2; MAARJS)" "-"

I must admit that nothing is changed besides php version (from 5.3 to 5.4).

I don’t see any relation to some specific browser or some specific request, it is totally random.
Also it can not be the switch problem because I have the same problem on my other apache servers that are not sharing same switch.

Till now I tried to:
— update network card driver
— raise RX from 256 to 4096 with: ethtool -G eth1 rx 4096
— bond second network card to the interface
— google this problem many times but didn’t find the solution or same issue.

Server version: Apache/2.2.15-39 (Unix)
centos 6.5

Jenny D

27.5k21 gold badges74 silver badges112 bronze badges

asked Jan 20, 2015 at 13:56

This is probably a client problem, though why you didn’t get it before I could not say.

The error is logged when a request is received by Apache that doesn’t complete before the end of the headers section. So as a minimum Apache has receive the request line e.g. GET / HTTP/1.0 but has been sent a valid set of headers after this.

So it could either be no headers at all, or a bunch of headers without the blank line that indicates the boundary between the headers and the body of the request.

I guess it’s possible the default PHP timeout may have been changed between your two installations, so that is worth checking, but really this is an indication of a bad (or slow) client making requests to your web server.

answered Oct 1, 2016 at 7:37

UnbelieverUnbeliever

2,3161 gold badge9 silver badges18 bronze badges

Источник

Содержание

request failed: error reading the headers
testtubebaby
Apache stopped working permanently after error «Server ran out of threads to serve requests.»
500 internal server — AH02429: Response header name
4 Answers 4
request failed: error reading the headers
testtubebaby
Сброс соединения по пиру: mod_fcgid: ошибка чтения данных с сервера FastCGI
13 ответов

testtubebaby

Member

I have setup cpanel server recently. There is no domain hosted on it and I see the following errors in /usr/local/apache/logs/error_log

/SOFTWARE-DOWNLOADS]# tail -f /usr/local/apache/logs/error_log | grep headers
[Fri May 23 03:56:14 2008] [error] [client 88.240.242.204] request failed: error reading the headers
[Fri May 23 03:56:14 2008] [error] [client 78.180.199.162] request failed: error reading the headers
[Fri May 23 03:56:14 2008] [error] [client 72.201.27.34] request failed: error reading the headers
[Fri May 23 03:56:14 2008] [error] [client 67.164.18.226] request failed: error reading the headers
[Fri May 23 03:56:14 2008] [error] [client 70.130.188.105] request failed: error reading the headers

In WHM apache status I see the folllowing:

Srv PID Acc M CPU SS Req Conn Child Slot Host VHost Request
0-2 18618 2/5748/22036 W 3.29 1 0 0.7 2.02 7.75 78.183.208.205 server01.myservername POST http://yahoo.com HTTP/1.1
1-2 18619 0/5796/22256 W 3.30 3 0 0.0 2.04 7.82 127.0.0.1 server01.myservername GET /whm-server-status/ HTTP/1.0
2-2 18620 2/5682/21814 W 3.16 2 0 0.7 2.00 7.68 88.240.204.170 server01.myservername POST http://yahoo.com HTTP/1.1
3-2 18621 0/5642/21941 _ 3.19 1 0 0.0 1.98 7.72 72.201.27.34 server01.myservername POST http://yahoo.com HTTP/1.1
4-2 18622 2/5782/21926 W 3.28 3 0 0.7 2.03 7.71 85.106.233.35 server01.myservername POST http://yahoo.com HTTP/1.1
5-2 18642 2/5608/21906 W 3.18 2 0 0.7 1.97 7.71 78.180.199.162 server01.myservername POST http://yahoo.com HTTP/1.1
6-2 18658 0/5657/21941 _ 3.18 1 0 0.0 2.00 7.72 88.234.47.182 server01.myservername POST http://yahoo.com HTTP/1.1

Is this some kind of attack? There are tons of such requests.

Источник

Apache stopped working permanently after error «Server ran out of threads to serve requests.»

I have been working with Apache for more than 8 years, but now I’m facing a severe problem and I couldn’t find a solution on internet so far.

I’m running Apache 2.2.15 32bit on a Windows 2008 R2 64bit with Service Pack 1. I always used a 32bit Apache because my CGI codes (exe files) are 32bit.

The computer has a 3.10 GHz processor and 16GB RAM. The server load is always very low. The access log showed a normal request amount when the problem happened.

Since the installation many months ago, everything was running correctly, but now my Apache stopped working and I don’t know how to fix this problem.

It has been many weeks I don’t change any configuration, but suddenly the Apache stopped serving the HTTP responses. All requests get timed out. In a practical sense, my site is offline.

By using Windows Event Viewer, I have found an error log when Apache stopped working. It’s the error event:

It’s what Apache error log recorded near to the time of error above (6/28/2016 8:08:33). It’s possible to see the error message «Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting». After this message, Apache never worked again. (Previous Apache error log messages were recorded very earlier and are not related.)

After the error above, restarting Windows and Apache service didn’t solve the problem (same error message). After reinstalling Apache and booting the machine, the problem was still there (same error message). So, I increased the «MaxRequestsPerChild» parameter to 300, but all worker threads were consumed in 12 seconds. So, I increased the «MaxRequestsPerChild» parameter to 500 and rebooted the machine, but all worker threads were consumed in 9 seconds as the log shows below.

Below you can see my complete original httpd.conf. It had been working correctly for months. (I removed just all the comment lines to make it smaller and more readable, and also masked some minimal sensible data with «xxxxx»):

As shown above, my Apache stops responding almost instantly after launch.

My site doesn’t connect to databases. It just serves dynamic pages by using CGI.

Is there a way to clean up the system to make Apache work again as it used to work before this mysterious crash?

Could anyone give me any advice to solve this issue or at least to get a workaround?

Источник

OS Ubuntu Ubuntu 16.04.2 LTS Apache/2.4.18 Perl v5.22.1

So I have perl code that reads a mysql database, and then creates a dynamic webpage. The code’s been working great for years. A week ago, my client calls and says they are getting a 500 internal server message. I take a look and everything seems fine, but when I try to load the page, I get 500 error. In the apache error log, there’s this error:

[Tue May 30 22:16:13.144199 2017] [http:error] [pid 12487] [client 99.99.99.99:55628] AH02429: Response header name ‘

Has anyone seen this, or have any idea what is causing it?

4 Answers 4

You have a key header with invalids characters, it wasn’t a problem untill security fix CVE-2016-8743.

In my case wasApache + PHP and a whitespace before «:» like «X-CUSTOM-KEY :» and I haven’t found other solution than changing the header.

As @GroGz says, this issue is almost certainly caused (or rather exposed) by the fix to CVE-2016-8743 — the parsing of HTTP headers was made much more strict in an update to Apache released on 20 December 2016 (details at https://httpd.apache.org/security/vulnerabilities_24.html). It’s likely your Perl script uses the module CGI::Carp which includes a «warningsToBrowser» subroutine. This subroutine puts warnings triggered by code issues in HTML comments embedded in the output of your program, rather than just logging them to the HTTPD logs. This subroutine is triggered like so:

If you trigger it before the HTTP headers have been sent in their entirety you will see an error similar to the one you describe.

The easy fix is to search your code for any occurrence of:

Источник

testtubebaby

Member

I have setup cpanel server recently. There is no domain hosted on it and I see the following errors in /usr/local/apache/logs/error_log

In WHM apache status I see the folllowing:

Is this some kind of attack? There are tons of such requests.

Источник

Сброс соединения по пиру: mod_fcgid: ошибка чтения данных с сервера FastCGI

У меня возникла проблема с PHP, где мое приложение пытается запустить файл резервной копии php и вдруг получает код HTTP Error 500. Я проверил журналы и это то, что он говорит.

[Вт 28 авг 14:17:28 2012] [предупреждение] [клиент xxxx] (104) Соединение reset by peer: mod_fcgid: ошибка чтения данных с сервера FastCGI, референт: http://example.com/backup/backup.php
[Вт 28 авг 14:17:28 2012] [ошибка] [клиент xxxx] Преждевременный конец заголовков script: backup.php, referer: http://example.com/backup/backup.php

Кто-нибудь знает, как это исправить? Я действительно застрял здесь и не могу найти решение в Интернете.

Надеюсь, что кто-то может поделиться своими знаниями.

13 ответов

Мне удалось решить эту проблему, добавив FcgidBusyTimeout. На всякий случай, если у кого-то есть аналогичная проблема со мной.

Вот мои настройки на моем apache.conf:

У меня были очень похожие ошибки в файлах журнала Apache2:

После проверки сценариев оболочки и настроек Apache2 я понял, что/var/www/не имеет соответствующих разрешений. Таким образом, скрипты FCGId Wrapper не могли быть прочитаны вообще.

Для моего сценария chmod -o+rx /var/www требовалось, конечно, так как используемые пользователи SuExec не входят в группу пользователей www-data , и, конечно, они не должны быть членами по соображениям безопасности.

если вы хотите установить версию PHP ./configure , выдержка из документа php.net:

Если это включено, модуль CGI будет построен с поддержкой FastCGI. Доступно с PHP 4.3.0

Начиная с PHP 5.3.0, этот аргумент больше не существует и включен вместо —enable-cgi. После компиляции ./php-cgi -v должен выглядеть следующим образом:

В версиях CentOS suexec компилируется для запуска только в /var/www. Если вы попытаетесь установить DocumentRoot где-то в другом месте, вам придется перекомпилировать его — ошибка в журнале apache: (104) Соединение reset by peer: mod_fcgid: ошибка чтения данных с сервера FastCGI Преждевременный конец заголовков script: php5.fcgi

У меня была та же проблема с длинными сценариями с сообщениями об ошибках «Недопустимый конец заголовков script: index.php» и «Connection reset by peer: mod_fcgid: ошибка чтения данных с сервера FastCGI» в error_log. После нескольких часов тестирования это помогает мне (CentOS 6, PHP-FPM 7, Plesk 12.5.30):

отредактируйте файл конфигурации:

Установите более высокое время работы. В моем случае 600 секунд

создайте новую запись:

адаптировать следующие записи:

Я столкнулся с этим при отладке связанной с virtualmin/apache ошибки.

В моем случае я запускаю virtualmin и имел в своей виртуальной машине php.ini safe_mode = On.

В моем журнале ошибок виртуальной машины я получал соединение fcgi reset через peer: mod_fcgid: ошибка чтения данных с сервера FastCGI

В моем главном журнале ошибок apache я получал: PHP Неустранимая ошибка: директива «safe_mode» больше не доступна в PHP в Unknown on line 0

В моем случае я просто устанавливаю safe_mode = Off в моем php.ini и перезапущенном apache.

У меня возникла эта проблема и я понял, что файл cgi-bin/php-fcgi не имеет прав выполнения.

Он имел режим 644, в то время как он должен иметь режим 755.

Настройка правильного режима была невозможна (вероятно, потому, что файл был открыт или sth), поэтому я скопировал этот файл из другого каталога домена, где у него были уже установленные права, и что все исправлено.

Знаменитый Moodle «replace.php» script также может генерировать эту ситуацию. Для меня это потребовало времени для запуска, а затем с сообщением 500 в браузере, а также с указанным выше сообщением об ошибке в моем файле журнала ошибок apache.

Я проследил @james-мудрый ответ: FcgidBusy легко читается в документации Apache. Я попробовал это: удвоил количество времени, которое apache дал бы моему script, вставив следующую строку в /etc/apache 2/mods-available/fcgid.conf

Затем я перезапустил Apache и снова попытался запустить my replace.php script.

К счастью, на этот раз экземпляр script завершился, поэтому для моих целей это послужило решением.

Источник

Currently we have an Apache 2.2.3 server with mod_ssl 2.2.3 running Django, with users authenticating by using a x509 certificate.

So far the system is running perfectly except for a single user, who when trying to upload a file receives 400 Bad Request error, and the contents of the ssl_error_log regarding this operation are:

[<date>] [error] [client <client ip>] request failed: error reading the headers, referer: <referrer url>

The contents of the ssl_access_log are:

<client ip> — — [<date>] «POST <target page> HTTP/1.1» 400 321

Also, the user’s browser is Firefox as far as I know.

I am completely unable to reproduce this bug and so far none of the other users have experienced it. Could you point out some reasons for this to happen?

asked Jun 18, 2012 at 13:06

I’ve experienced connectivity that stops the upstream after an X amount of bytes is sent. X was a pretty low value, as in enough to request some simple pages, but not to deal with ajax requests much less upload files. As far as I recall, this connectivity problem occurred only when tethering (from a specific Android phone, but I didnt even test other phones).

So if the upstream gets interrupted and the upload stalls, it makes sense apache would return this error, according to this post: «Apache waits a time equal to the Timeout directive (defaults to 5 minutes if not defined) for a response from the client. It is likely Apache is waiting for the CRLF that indicates the end of the headers, yet it is never received..»

answered Oct 3, 2013 at 22:44

GaiaGaia

2,8021 gold badge39 silver badges59 bronze badges

Источник

: February 26 2004, 02:00

This post started out as a question, but in writing it out I figured out the problem and thought it may be of use to anyone who has the same problem since it was hard to come by information on it.

The situation: running Apache httpd 2.0 and using mod_rewrite to play low-cost load balancer to a farm of web servers. The LB machine is Redhat 9.0 with all but ports 22 and 80 firewalled. The servers on the web farm are Apache 1.3.x running, among other things, FastCGI and mod_perl.

The problem: the web servers in the far are dropping errors into the logs, once or twice a minute (but probably will be more by morning load) to the tune of «request failed: error reading the headers«.

It seems to be a very select problem — there are few relevant mentions of this one usenet or even the web, and many are reports of worms or the posts were never followed up on. I was ready to give up and blame httpd 2.0 or mod_perl. But…

I did some checking and creative grepping on my error and access logs on both the LB and the webheads and found something interesting. The errant requests were being passed from the httpd 2.0 front end, but the errors weren’t directly caused by it.

Instead I found that these are errors cause by someone issuing a «proxy request» to my site; but since the load balancer is not (anymore) configured as a proxy, it passes those «CONNECT // HTTP/1.1» requests to the webheads. The webheads in turn either don’t do proxying or have it disabled and thus throw errors: «request failed: error reading the headers».

This can be proved by running tail -f /var/log/httpd/error_log | grep headers in one window and tail -f /var/log/httpd/access_log | grep CONNECT in another — whenever an «error reading the headers» message appears, a CONNECT message will also appear in the access log and they both will have the same timestamp.

So in the end, in cases such as this, these errors are safe to be ignored. But for the sake of the Internet and you pocketbook, make sure your install of Apache isn’t configured as an open proxy

Источник

When sending a HTTP request to Apache it might happen that the response is HTTP 400.

<!DOCTYPE HTML PUBLIC ”-//IETF//DTD HTML 2.0//EN”>n<html><head>n<title>400 Bad Request</title>n</head><body>n<h1>Bad Request</h1>n<p>Your browser sent a request that this server could not understand.<br />nSize of a request header field exceeds server limit.</p>n</body></html>n”

The error message is:

Size of a request header field exceeds server limit.

Root cause

The browser is sending a large HTTP header. Apache fails to process the HTTP request because, for instance, the request includes authentication information in the form of an access token. In my case, the authorization header is blowing up the request and already takes > 8Kb of space.

The http request contains a large header because the Authorization header is already larger than 8kb. Large is relative and depends on the Apache (or any web server) configuration. In current Apache version, everything that is larger than 8kb is considered too large.

In the Apache log the error is logged too.

[timestamp] [core:debug] [pid 10:tid 140312387237632] protocol.c(1022): (28)No space left on device: [client 172.21.0.1:63472] Failed to read request header line Authorization: Bearer eyJhbGciOiMjg2OXRyaWFsIiwic[…]

[timestamp] [core:info] [pid 10:tid 140312387237632] [client 172.21.0.1:63472] AH00561: Request header exceeds LimitRequestFieldSize: Authorization

[timestamp] [core:debug] [pid 10:tid 140312387237632] protocol.c(1375): [client 172.21.0.1:63472] AH00567: request failed: error reading the headers

The Apache log contains the error ID: AH00561

Why Apache reports this as HTTP 400 and not as HTTP 413? I have no idea. Maybe because the error is in the Authorization field? I know 413 when the cookies are too large.

Solution

With the error ID AH00561 known, it is easy to find the parameter that causes the error:

Parameter: LimitRequestFieldSize

Apache documentation shows an explanation for the error: “The max size of the http header permitted by default is 8kb.”

To increase the limit, adjust the parameter LimitRequestFieldSize for the virtual host or location in the Apache configuration.

Example

Increase the max header size to 20Kb.

<VirtualHost *:80>
  ProxyRequests Off
  LimitRequestFieldSize 200000
  <Location /wf >
    ProxyPass http://backend/wf
    ProxyPassReverse http://backend/wf
    Order allow,deny
    Allow from all
  </Location>
</VirtualHost>

This is another example that a good error message should contain a hint why something failed and the log should contain an error message ID that is unique. Both make it easy to find a solution.

Источник

Symptoms

When trying to connect to VisualSVN Server using Subversion client, a
user may receive the following error message:

svn: E175002: Unable to connect to a repository at URL ‘https://svn.example.com/svn/repo’
svn: E175002: Unexpected HTTP status 400 ‘Bad Request’ on ‘/svn/repo’

The error on the client is accompanied by the following Error Level
event in the VisualSVN Server event log:

request failed: error reading the headers

This article relates to the particular issue that can occur only when
VisualSVN Server is configured to use Integrated Windows
Authentication.

Technical background

The above errors indicate that HTTP request header received by VisualSVN
Server exceeds the header size limit. By default, request header size
limit is set to 16384 bytes in VisualSVN Server.

Kerberos authentication method is always selected when a user
authenticates to VisualSVN Server using Negotiate (SPNEGO) protocol and
the connecting client supports Kerberos. In such case, the client sends
special Kerberos access token as an HTTP request header. The error
occurs when HTTP request header containing the access token becomes too
large and exceeds the limit.

Kerberos token can become too large when Active Directory accounts
migration is in process and has not been fully completed yet. Kerberos
token includes SID history to retain access control and security
settings in the process of migration. Therefore, Kerberos token can
exceed HTTP header size limit if SID history is large. For more
information about SID history, read the Microsoft TechNet article

«Using SID History to Preserve Resource Access».

Resolution

If the problem occurs in the middle of Active Directory accounts
migration or the migration has not been properly completed, it is
necessary to complete the migration and clear the SID history. You can
find articles describing the problem and its resolution on
TechNet Blogs:

Using PowerShell to resolve Token Size issues caused by SID history,
Do Over: SID History One-Liner.

Workaround

In order to workaround the issue, it is possible to increase the HTTP
request header limit set in VisualSVN Server.

Increasing the HTTP request header size limit might affect performance
of VisualSVN Server and make it vulnerable to DoS attacks. Therefore,
this workaround is intended only for temporary mitigation!

Follow these steps to increase the HTTP request header limit:

Start Notepad as administrator and open the file:
%VISUALSVN_SERVER%confhttpd-custom.conf.
Enter the following line and save the file:
```
LimitRequestFieldSize 32760
```
Restart VisualSVN Server.

Источник

Troubleshooting

Problem

Request to IBM HTTP Server fails with Response code 400.

Symptom

Response from the browser could be shown like this:
Bad Request
Your browser sent a request that this server could not understand.
Size of a request header field exceeds server limit.

IBM HTTP Server Error.log shows the following message:
«request failed: error reading the headers»

Cause

This is normally caused by having a very large Cookie, so a request header field exceeded the limit set for Web Server.
For IBM® HTTP Server, this limit is set by LimitRequestFieldSize directive (default 8K). The LimitRequestFieldSize directive allows the Web server administrator to reduce or increase the limit on the allowed size of an HTTP request header field.
SPNEGO authentication headers can be up to 12392 bytes. This directive gives the server administrator greater control over abnormal client request behavior, which may be useful for avoiding some forms of denial-of-service attacks.

Diagnosing The Problem

To assist with diagnose of the problem you can add the following to the LogFormat directive in the httpd.conf:
error-note: %{error-notes}n

Resolving The Problem

Increase the value for the directive LimitRequestFieldSize in the httpd.conf:

LimitRequestFieldSize 12288 or 16384

[{«Product»:{«code»:»SSEQTJ»,»label»:»IBM HTTP Server»},»Business Unit»:{«code»:»BU059″,»label»:»IBM Software w/o TPS»},»Component»:»Base Server»,»Platform»:[{«code»:»PF002″,»label»:»AIX»},{«code»:»PF010″,»label»:»HP-UX»},{«code»:»PF016″,»label»:»Linux»},{«code»:»PF035″,»label»:»z/OS»},{«code»:»PF012″,»label»:»IBM i»},{«code»:»PF027″,»label»:»Solaris»},{«code»:»PF033″,»label»:»Windows»}],»Version»:»7.0″,»Edition»:»»,»Line of Business»:{«code»:»LOB45″,»label»:»Automation»}}]

Источник