Pure of heart, life is full of sweet and joy. — Leo Tolstoy
Socket hang up first appeared in a service load test and was later resolved. Recently, this problem was reported again when node.js service migrated to K8S container. After checking the cause, it was found that the container’S CPU and memory size were limited. Here is a summary of what Socket hang up is, when it happens, and how to solve it.
About the author: May Jun, Nodejs Developer, moOCs certified author, love technology, like sharing, welcome to pay attention to Nodejs technology stack and Github open source project www.nodejs.red
What is a Socket hang up
What is a Socket hang up?
Socket hang up means that the socket (link) is hung up. No matter which language you use, you should have encountered it more or less, but have you ever thought about why? For example, in Node.js, the system provides a default timeout of 2 minutes for the HTTP server. If a request exceeds this time, the HTTP server will close the request. When a client tries to return a request and finds that the socket has been «hung up», it sends a socket hang up error.
To understand a problem, or to practice more, the following is a small demo of the problem and then combine with node.js HTTP related source code to further understand Socket hang up? Also recommend that you look at the stack overflow of the universal also has a discussion on the issue above stackoverflow.com/questions/1… .
Replicating Socket hang up
The service side
To start an HTTP service, define the /timeout interface to delay the response for 3 minutes
const http = require('http');
const port = 3020;
const server = http.createServer((request, response) = {
console.log('request url: ', request.url);
if (request.url === '/timeout') {
setTimeout(function() {
response.end('OK! ');
}, 1000 * 60 * 3)
}
}).listen(port);
console.log('server listening on port ', port);
Copy the code
The client
const http = require('http');
const opts = {
hostname: '127.0.0.1'.port: 3020.path: '/timeout'.method: 'GET'}; http.get(opts, (res) = {let rawData = ' ';
res.on('data', (chunk) = { rawData += chunk; });
res.on('end', () = {try {
console.log(rawData);
} catch (e) {
console.error(e.message); }}); }).on('error', err = {
console.error(err);
});
Copy the code
After starting the server and then starting the client about 2 minutes later or directly killing the server, the following error is reported. You can see the corresponding error stack
Error: socket hang up
at connResetException (internal/errors.js:570:14)
at Socket.socketOnEnd (_http_client.js:440:23)
at Socket.emit (events.js:215:7)
at endReadableNT (_stream_readable.js:1183:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21) {
code: 'ECONNRESET'
}
Copy the code
The node.js HTTP client source code does not receive any response, so it is considered that the socket has ended. A connResetException(‘socket hang up’) error is therefore emitted at L440.
// https://github.com/nodejs/node/blob/v12.x/lib/_http_client.js#L440
function socketOnEnd() {
const socket = this;
const req = this._httpMessage;
const parser = this.parser;
if(! req.res ! req.socket._hadError) {// If we don't have a response then we know that the socket
// ended prematurely and we need to emit an error on the request.
req.socket._hadError = true;
req.emit('error', connResetException('socket hang up'));
}
if (parser) {
parser.finish();
freeParser(parser, req, socket);
}
socket.destroy();
}
Copy the code
Socket hang up Solution
1. Set the TIMEOUT period of the HTTP Server socket
By default, the timeout value of the server is 2 minutes. If it does, the socket will automatically destroy itself. You can use the server.setTimeout(msecs) method to set the timeout to a larger value. Passing a 0 will turn off the timeout mechanism
// https://github.com/nodejs/node/blob/v12.x/lib/_http_server.js#L348
function Server(options, requestListener) {
// ...
this.timeout = kDefaultHttpServerTimeout; // The default value is 2 x 60 x 1000
this.keepAliveTimeout = 5000;
this.maxHeadersCount = null;
this.headersTimeout = 40 * 1000; // 40 seconds
}
Object.setPrototypeOf(Server.prototype, net.Server.prototype);
Object.setPrototypeOf(Server, net.Server);
Server.prototype.setTimeout = function setTimeout(msecs, callback) {
this.timeout = msecs;
if (callback)
this.on('timeout', callback);
return this;
};
Copy the code
The modified code looks like this:
const server = http.createServer((request, response) = {
console.log('request url: ', request.url);
if (request.url === '/timeout') {
setTimeout(function() {
response.end('OK! ');
}, 1000 * 60 * 3)
}
}).listen(port);
server.setTimeout(0); // Set the timeout period
Copy the code
If you do not set setTimeout, you can also catch such errors on the HTTP client, put them in the queue and initiate retry. If the probability of such errors is high, you should check whether the corresponding service has abnormal problems such as slow processing.
ECONNRESET VS ETIMEDOUT
Note the difference between ECONNRESET and ETIMEDOUT
ECONNRESET is read timeout. {«code»:»ECONNRESET»} error occurs when the server is too slow to respond properly, such as the socket hang up example described above.
ETIMEDOUT refers to the timeout that occurs when a client initiates a connection with a remote server. Here is an example of a request from the Request module.
const request = require('request');
request({
url: 'http://127.0.0.1:3020/timeout'.timeout: 5000,
}, (err, response, body) = {
console.log(err, body);
});
Copy the code
In the above example, {code: ‘ETIMEDOUT’} error is reported after approximately 5 seconds, and the stack is as follows:
Error: ETIMEDOUT
at Timeout._onTimeout (/Users/test/node_modules/request/request.js:677:15)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7) {
code: 'ETIMEDOUT'
}
Copy the code
- Version: 8.0.0
- Platform: OS X
- Subsystem: http
With the following
var http = require('http'); var server = http.createServer((req, res) => { console.log('got request'); res.write('efgh') res.end(); }); server.listen(8080, function() { makeRequest(() => { server.close(); }); }); function makeRequest(cb) { var req = http.request('http://localhost:8080', function(res) { console.log('got response') res.resume(); cb(); }) req.end('abcd'); }
I get this error
events.js:182
throw er; // Unhandled 'error' event
^
Error: socket hang up
at createHangUpError (_http_client.js:343:15)
at Socket.socketOnEnd (_http_client.js:435:23)
at emitNone (events.js:110:20)
at Socket.emit (events.js:207:7)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:102:11)
at process._tickCallback (internal/process/next_tick.js:161:9)
The same code works fine on Node 7 and 6. The key to reproducing seems to be that req.end()
is called with data and res.write()
is called with data before calling res.end()
.
Simplified scenario:
var http = require('http') http.createServer((req, res) => { }).listen(25000, () => { http.request('http://localhost:25000', (res) => { }).end(' ') })
It fails in more platforms and more versions.
if I use the same client and a remote web end-point for server, it works fine.
As @martinkuba pointed out, if end is called without data, no exception is seen.
Also if the server is responding atomically, no exception is seen.
Even if server delays responding, if it is atomic exception is seen:
http.createServer((req, res) => { setTimeout(() => res.end(), 1000); })
Version: 8.0.0
Platform: Ubuntu 16.04 LTS
Subsystem: http
Using @martinkuba server code I have the same error message on Ubuntu.
This is because of wrong method
being used? Since here https://github.com/nodejs/node/blob/master/lib/_http_client.js#L432 when the method is GET
the req.res
object is null
.
If we change @martinkuba’s line here:
var req = http.request('http://localhost:8080', function(res) {
to e.g. POST
that makes useChunkedEncodingByDefault
true.
var req = http.request({host: 'localhost', port: 8080, method: 'POST'}, function(res) {
It works: https://runkit.com/diorahman/post-13461
Not sure if we should throw a meaningful error while doing req.end(<PAYLOAD>)
with the client request when the method is not right.
ClientRequest.end specification does not impose any restrictions on the payload based on the method — so it looks like something else.
[refack fixed formating]
@diorahman — thanks for the info. I went though the history and came to an inference that malformed GET request can be rejected
However, I see inconsistencies in the client behavior based on the nature of client, nature of server and behavior of the server with respect to handling the parse error:
#cat q.js
'use strict' const http = require('http') const net = require('net') const server = http.createServer((req, res) => { if (process.argv[2] === 'handle') { server.on('clientError', function(err) { console.log('parser error', err.code) }) } setTimeout(() => {res.end('hello malformed client!'), 10}) }) server.listen(25000, () => { const malformdata = 'me too' if (process.argv[3] === 'http') { // HTTP CLIENT http.request({host: 'localhost', port: 25000, method:'GET', path: '/'}, (res) => { let data = '' res.on('data', function(d) { data += d }) res.on('end', function() { console.log(data) server.close() }) }).end(malformdata) } else { // TCP client const input = `GET / HTTP/1.1rnHost: localhost:25000rnConnection: closernrn${malformdata}` const client = net.createConnection({host:'localhost', port: 25000}, () => { client.write(input) }) client.on('error', (e) => { console.log(e) }) client.on('data', (data) => { console.log('response: ' + data.toString()) server.close() }) } })
#node q.js handle http
parser error HPE_INVALID_METHOD
hello malformed client!
#node q.js handle tcp
parser error HPE_INVALID_METHOD
response: HTTP/1.1 200 OK
Date: Fri, 09 Jun 2017 06:07:53 GMT
Connection: close
Content-Length: 23
hello malformed client!
#node q.js nohandle http
events.js:182
throw er; // Unhandled 'error' event
^
Error: socket hang up
at createHangUpError (_http_client.js:343:15)
at Socket.socketOnEnd (_http_client.js:435:23)
at emitNone (events.js:110:20)
at Socket.emit (events.js:207:7)
at endReadableNT (_stream_readable.js:1045:12)
at _combinedTickCallback (internal/process/next_tick.js:102:11)
at process._tickCallback (internal/process/next_tick.js:161:9)
#node q.js nohandle tcp
^C // HANG
#
Given that node can be acting as HTTP client in many of the distributed computing workloads, or delegating requests to remote end points, I guess the http / net client library need handle this cleanly.
I would like either:
- Server intercept the parse error, end response with closest HTTP error code mapping to parse error
- Client intercepting malformed request, refuse to contact server, instead throw an error
@nodejs/http
@nodejs/streams
Also @refack @gibfahn
cc/ @nodejs/http @nodejs/streams
I am a bit lost on what is the problem here. There have been different threads that points me to separate issues. Is this an http regression, or a stream regression?
Regarding the «socket hang up», I think that error is correct, the server is closing the connection abruptly, and then it’s normal to have that error.
So maybe it was an HTTP change, and I think the current behavior is correct.
@gireeshpunathil can you add the links of the relative PRs?
@mcollina — I don’t have references to PRs, please treat my previous comment as an absolute problem statement.
I don’t claim it a regression either, as I can reproduce this in earlier versions as well.
If socket ‘hang up’ is the right client behavior, we have these situations:
- If server design is to close connection abruptly on malformed requests, then it has to be consistent. In this case,
- It sends proper response, if we call res.end() from server as soon as it hits the request
- It sends proper response, if we server has a client error handler.
- In either case, it does not provide a meaningful message to the client, or the clien’t caller.
- If a TCP client is enployed, we get a ‘hang’ instead of a ‘hang up’
(2) means that there is inconsistency at the client side as well.
Now that the clients are net and http, the entity which connect them is ‘stream’. And hence I suspect this has bearing on http as well as streams.
Bottom line is that an error of type Unhandled 'error' event with Error: socket hang up
does not qualify for and HTTP response on bad data, it could be proper for a low level network issue where the protocol layer dont have sufficient control.
We could very well reject this as noises through ‘hacks’ but then this behavior poses implications to web apps that forward remote requests that are not validated.
I still do not understand the problem.
The http client does something more to handle the error condition, here is a fixed version of your script, it should produce similar behavior:
'use strict' const http = require('http') const net = require('net') const server = http.createServer((req, res) => { if (process.argv[2] === 'handle') { server.on('clientError', function(err, sock) { console.log('parser error', err.code) console.log('destroying socket') sock.destroy() // the socket will not be destroyed automatically if you specify clientError }) } setTimeout(() => {res.end('hello malformed client!'), 10}) }) server.listen(25000, () => { const malformdata = 'me too' if (process.argv[3] === 'http') { // HTTP CLIENT http.request({host: 'localhost', port: 25000, method:'GET', path: '/'}, (res) => { let data = '' res.on('data', function(d) { data += d }) res.on('end', function() { console.log(data) server.close() }) }) .on('error', (err) => { console.log('error on request', err) server.close() }) .end(malformdata) } else { // TCP client const input = `GET / HTTP/1.1rnHost: localhost:25000rnConnection: closernrn${malformdata}` const client = net.createConnection({host:'localhost', port: 25000, allowHalfOpen: true }) client.end(input) client.on('error', (e) => { console.log('tcp', e) server.close() }) client.on('data', (data) => { console.log('response: ' + data.toString()) }) client.on('end', () => { console.log('client ended') server.close() }) } }) setTimeout(function () { console.log('normal termination') }, 1000)
We could very well reject this as noises through ‘hacks’ but then this behavior poses implications to web apps that forward remote requests that are not validated.
Does this mean you would like to accept requests that are not fully validated, in the example GET
requests with a body?
I would like to point out that this was a regression from my point of view. My original repro works fine on previous versions of Node.
@mcollina — thanks for this. Yes, this produces a consistent behavior.
So bad input is presented to the (node) server in the form of a ‘clientError’ event, and server is free to chose next action on the connection — I think that makes the server robust.
I will review the client side code and get back
@martinkuba that HTTP request is invalid, as there should be no body for GET requests.
That causes the request to abort, and it will emit an 'error'
event.
Yes, it is a change of behavior from 6 and 7, but I think it is better in this way. The request is invalid, and the default behavior in the HTTP server is to destroy the socket without replying. If you want to handle it differently, you should specify a 'clientError'
event handler.
cc @nodejs/http @jasnell @mscdex
Technically, HTTP allows GET requests with a body but it’s useless in practice.
Because GET is idempotent, proxies and servers are not allowed to attach meaning to it; all a conforming recipient can do is ignore the body (ref).
Node 8’s behavior is not wrong because it’s not a reasonable thing to do in the first place. I’ll close this out.
To add some more context, in Node 6 and 7 the following has the same behavior of Node 8:
var http = require('http'); var server = http.createServer((req, res) => { console.log('got request'); setTimeout(function () { res.write('efgh') res.end(); }, 100) }); server.listen(8082, function() { makeRequest(() => { server.close(); }); }); function makeRequest(cb) { var req = http.request('http://localhost:8082', function(res) { console.log('got response') res.resume(); cb(); }) req.end('abcd'); }
Node 8 made it consistent in all cases.
@mcollina I have some clients with this issue. I don’t have control over the client, so I’d like to provide a work on the server using ‘clientError’ event handler as you suggested. However, I’m struggling to get it working. How do I get the original request? I know I can use socket.end
to reply but I don’t see how to get the request…
https.createServer(options, (req, res) => {
// handle requests
})
.listen(port)
.on('clientError', (err, socket) => {
console.log('client error ->', err);
// also handle requests somehow
socket.end('HTTP/1.1 200 stuffrnrn');
});
I’m also getting an error (I’m using HTTPS):
client error -> Error: 140737126245312:error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request:../deps/openssl/openssl/ssl/s23_srvr.c:394:
at Error (native)
@amejiarosario what you get is not a request. You cannot get it. The only thing you can do in 'clientError'
is sending a specialized 5xx code, or customize the error.
for fellows still getting this error, i used longjohn which would print long stack traces, it helped me find the root cause of the error.
When a socket hang up is thrown, one of two things happens:
When you’re a customer,
When you send a request to a distant server as a client and don’t get a response in a timely manner. This error is caused by the end of your socket. You should catch this error and decide what to do with it, such as retrying the request or queueing it for later.
If you’re a server or proxy,
When you, as a server, possibly a proxy server, get a request from a client and begin acting on it (or relaying the request to the upstream server), the client decides to cancel/abort the request before you have finished preparing the response.
When a customer cancels a request, this stack trace depicts what happened.
Trace: { [Error: socket hang up] code: 'ECONNRESET' } at ClientRequest.proxyError (your_server_code_error_handler.js:137:15) at ClientRequest.emit (events.js:117:20) at Socket.socketCloseListener (http.js:1526:9) at Socket.emit (events.js:95:17) at TCP.close (net.js:465:12)
Line http.js:1526:9points to the same socketCloseListener mentioned by @Blender, particularly:
// This socket error fired before we started to // receive a response. The error needs to // fire on the request. req.emit('error', createHangUpError()); ... function createHangUpError() { var error = new Error('socket hang up'); error.code = 'ECONNRESET'; return error; }
If the client is a browser user, this is a common scenario. When a request for a resource/page takes a long time to load, visitors simply refresh the page. As a result of this action, the previous request is cancelled, resulting in this error on your server.
Because this issue is the result of a client’s request, they should not expect to receive an error notice. As a result, there’s no reason to consider this error to be important. Simply disregard it. This is bolstered by the fact that on such an error, the res socket that your client was listening to is destroyed, despite the fact that it is still editable.
console.log(res.socket.destroyed); //true
So, no point to send anything, except explicitly closing the response object:
res.end();
However, if you are a proxy server that has already transmitted the request to the upstream, you should abort your internal request to the upstream, signalling that you are uninterested in the response, which will alert the upstream server to maybe halt an expensive activity.
There are two cases when socket hang up
gets thrown:
When you are a client
When you, as a client, send a request to a remote server, and receive no timely response. Your socket is ended which throws this error. You should catch this error and decide how to handle it: whether retry the request, queue it for later, etc.
When you are a server/proxy
When you, as a server, perhaps a proxy server, receive a request from a client, then start acting upon it (or relay the request to the upstream server), and before you have prepared the response, the client decides to cancel/abort the request.
This stack trace shows what happens when a client cancels the request.
Trace: { [Error: socket hang up] code: 'ECONNRESET' }
at ClientRequest.proxyError (your_server_code_error_handler.js:137:15)
at ClientRequest.emit (events.js:117:20)
at Socket.socketCloseListener (http.js:1526:9)
at Socket.emit (events.js:95:17)
at TCP.close (net.js:465:12)
Line http.js:1526:9
points to the same socketCloseListener
mentioned by @Blender, particularly:
// This socket error fired before we started to
// receive a response. The error needs to
// fire on the request.
req.emit('error', createHangUpError());
...
function createHangUpError() {
var error = new Error('socket hang up');
error.code = 'ECONNRESET';
return error;
}
This is a typical case if the client is a user in the browser. The request to load some resource/page takes long, and users simply refresh the page. Such action causes the previous request to get aborted which on your server side throws this error.
Since this error is caused by the wish of a client, they don’t expect to receive any error message. So, no need to consider this error as critical. Just ignore it. This is encouraged by the fact that on such error the res
socket that your client listened to is, though still writable, destroyed.
console.log(res.socket.destroyed); //true
So, no point to send anything, except explicitly closing the response object:
res.end();
However, what you should do for sure if you are a proxy server which has already relayed the request to the upstream, is to abort your internal request to the upstream, indicating your lack of interest in the response, which in turn will tell the upstream server to, perhaps, stop an expensive operation.
~ Answered on 2015-01-08 07:37:30
There are two cases when socket hang up
gets thrown:
When you are a client
When you, as a client, send a request to a remote server, and receive no timely response. Your socket is ended which throws this error. You should catch this error and decide how to handle it: whether retry the request, queue it for later, etc.
When you are a server/proxy
When you, as a server, perhaps a proxy server, receive a request from a client, then start acting upon it (or relay the request to the upstream server), and before you have prepared the response, the client decides to cancel/abort the request.
This stack trace shows what happens when a client cancels the request.
Trace: { [Error: socket hang up] code: 'ECONNRESET' }
at ClientRequest.proxyError (your_server_code_error_handler.js:137:15)
at ClientRequest.emit (events.js:117:20)
at Socket.socketCloseListener (http.js:1526:9)
at Socket.emit (events.js:95:17)
at TCP.close (net.js:465:12)
Line http.js:1526:9
points to the same socketCloseListener
mentioned by @Blender, particularly:
// This socket error fired before we started to
// receive a response. The error needs to
// fire on the request.
req.emit('error', createHangUpError());
...
function createHangUpError() {
var error = new Error('socket hang up');
error.code = 'ECONNRESET';
return error;
}
This is a typical case if the client is a user in the browser. The request to load some resource/page takes long, and users simply refresh the page. Such action causes the previous request to get aborted which on your server side throws this error.
Since this error is caused by the wish of a client, they don’t expect to receive any error message. So, no need to consider this error as critical. Just ignore it. This is encouraged by the fact that on such error the res
socket that your client listened to is, though still writable, destroyed.
console.log(res.socket.destroyed); //true
So, no point to send anything, except explicitly closing the response object:
res.end();
However, what you should do for sure if you are a proxy server which has already relayed the request to the upstream, is to abort your internal request to the upstream, indicating your lack of interest in the response, which in turn will tell the upstream server to, perhaps, stop an expensive operation.