Node js error socket hang up

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Pure of heart, life is full of sweet and joy. — Leo Tolstoy

Socket hang up first appeared in a service load test and was later resolved. Recently, this problem was reported again when node.js service migrated to K8S container. After checking the cause, it was found that the container’S CPU and memory size were limited. Here is a summary of what Socket hang up is, when it happens, and how to solve it.

About the author: May Jun, Nodejs Developer, moOCs certified author, love technology, like sharing, welcome to pay attention to Nodejs technology stack and Github open source project www.nodejs.red

What is a Socket hang up

What is a Socket hang up?

Socket hang up means that the socket (link) is hung up. No matter which language you use, you should have encountered it more or less, but have you ever thought about why? For example, in Node.js, the system provides a default timeout of 2 minutes for the HTTP server. If a request exceeds this time, the HTTP server will close the request. When a client tries to return a request and finds that the socket has been «hung up», it sends a socket hang up error.

To understand a problem, or to practice more, the following is a small demo of the problem and then combine with node.js HTTP related source code to further understand Socket hang up? Also recommend that you look at the stack overflow of the universal also has a discussion on the issue above stackoverflow.com/questions/1… .

Replicating Socket hang up

The service side

To start an HTTP service, define the /timeout interface to delay the response for 3 minutes

const http = require('http');
const port = 3020;

const server = http.createServer((request, response) =  {
    console.log('request url: ', request.url);

    if (request.url === '/timeout') {
        setTimeout(function() {
            response.end('OK! ');
        }, 1000 * 60 * 3)
    }
}).listen(port);

console.log('server listening on port ', port);
Copy the code

The client

const http = require('http');
const opts = {
  hostname: '127.0.0.1'.port: 3020.path: '/timeout'.method: 'GET'}; http.get(opts, (res) = {let rawData = ' ';
  res.on('data', (chunk) = { rawData += chunk; });
  res.on('end', () =  {try {
      console.log(rawData);
    } catch (e) {
      console.error(e.message); }}); }).on('error', err = {
  console.error(err);
});
Copy the code

After starting the server and then starting the client about 2 minutes later or directly killing the server, the following error is reported. You can see the corresponding error stack

Error: socket hang up
    at connResetException (internal/errors.js:570:14)
    at Socket.socketOnEnd (_http_client.js:440:23)
    at Socket.emit (events.js:215:7)
    at endReadableNT (_stream_readable.js:1183:12)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  code: 'ECONNRESET'
}
Copy the code

The node.js HTTP client source code does not receive any response, so it is considered that the socket has ended. A connResetException(‘socket hang up’) error is therefore emitted at L440.

// https://github.com/nodejs/node/blob/v12.x/lib/_http_client.js#L440

function socketOnEnd() {
  const socket = this;
  const req = this._httpMessage;
  const parser = this.parser;

  if(! req.res  ! req.socket._hadError) {// If we don't have a response then we know that the socket
    // ended prematurely and we need to emit an error on the request.
    req.socket._hadError = true;
    req.emit('error', connResetException('socket hang up'));
  }
  if (parser) {
    parser.finish();
    freeParser(parser, req, socket);
  }
  socket.destroy();
}
Copy the code

Socket hang up Solution

1. Set the TIMEOUT period of the HTTP Server socket

By default, the timeout value of the server is 2 minutes. If it does, the socket will automatically destroy itself. You can use the server.setTimeout(msecs) method to set the timeout to a larger value. Passing a 0 will turn off the timeout mechanism

// https://github.com/nodejs/node/blob/v12.x/lib/_http_server.js#L348
function Server(options, requestListener) {
  // ...

  this.timeout = kDefaultHttpServerTimeout; // The default value is 2 x 60 x 1000
  this.keepAliveTimeout = 5000;
  this.maxHeadersCount = null;
  this.headersTimeout = 40 * 1000; // 40 seconds
}
Object.setPrototypeOf(Server.prototype, net.Server.prototype);
Object.setPrototypeOf(Server, net.Server);


Server.prototype.setTimeout = function setTimeout(msecs, callback) {
  this.timeout = msecs;
  if (callback)
    this.on('timeout', callback);
  return this;
};
Copy the code

The modified code looks like this:

const server = http.createServer((request, response) =  {
    console.log('request url: ', request.url);

    if (request.url === '/timeout') {
        setTimeout(function() {
            response.end('OK! ');
        }, 1000 * 60 * 3)
    }
}).listen(port);

server.setTimeout(0); // Set the timeout period
Copy the code

If you do not set setTimeout, you can also catch such errors on the HTTP client, put them in the queue and initiate retry. If the probability of such errors is high, you should check whether the corresponding service has abnormal problems such as slow processing.

ECONNRESET VS ETIMEDOUT

Note the difference between ECONNRESET and ETIMEDOUT

ECONNRESET is read timeout. {«code»:»ECONNRESET»} error occurs when the server is too slow to respond properly, such as the socket hang up example described above.

ETIMEDOUT refers to the timeout that occurs when a client initiates a connection with a remote server. Here is an example of a request from the Request module.

const request = require('request');

request({
  url: 'http://127.0.0.1:3020/timeout'.timeout: 5000,
}, (err, response, body) = {
  console.log(err, body);
});
Copy the code

In the above example, {code: ‘ETIMEDOUT’} error is reported after approximately 5 seconds, and the stack is as follows:

Error: ETIMEDOUT
    at Timeout._onTimeout (/Users/test/node_modules/request/request.js:677:15)
    at listOnTimeout (internal/timers.js:531:17)
    at processTimers (internal/timers.js:475:7) {
  code: 'ETIMEDOUT'
}
Copy the code

@martinkuba

  • Version: 8.0.0
  • Platform: OS X
  • Subsystem: http

With the following

var http = require('http');

var server = http.createServer((req, res) => {
  console.log('got request');
  res.write('efgh')
  res.end();
});

server.listen(8080, function() {
  makeRequest(() => {
    server.close();
  });
});

function makeRequest(cb) {
  var req = http.request('http://localhost:8080', function(res) {
    console.log('got response')
    res.resume();
    cb();
  })
  req.end('abcd');
}

I get this error

events.js:182
      throw er; // Unhandled 'error' event
      ^

Error: socket hang up
    at createHangUpError (_http_client.js:343:15)
    at Socket.socketOnEnd (_http_client.js:435:23)
    at emitNone (events.js:110:20)
    at Socket.emit (events.js:207:7)
    at endReadableNT (_stream_readable.js:1045:12)
    at _combinedTickCallback (internal/process/next_tick.js:102:11)
    at process._tickCallback (internal/process/next_tick.js:161:9)

The same code works fine on Node 7 and 6. The key to reproducing seems to be that req.end() is called with data and res.write() is called with data before calling res.end().

@gireeshpunathil

Simplified scenario:

var http = require('http')
http.createServer((req, res) => {
}).listen(25000, () => {
  http.request('http://localhost:25000', (res) => {
  }).end(' ')
})

It fails in more platforms and more versions.
if I use the same client and a remote web end-point for server, it works fine.
As @martinkuba pointed out, if end is called without data, no exception is seen.
Also if the server is responding atomically, no exception is seen.
Even if server delays responding, if it is atomic exception is seen:

http.createServer((req, res) => {
  setTimeout(() => res.end(), 1000);
})

@Michael-Antczak

Version: 8.0.0
Platform: Ubuntu 16.04 LTS
Subsystem: http

Using @martinkuba server code I have the same error message on Ubuntu.

@diorahman

This is because of wrong method being used? Since here https://github.com/nodejs/node/blob/master/lib/_http_client.js#L432 when the method is GET the req.res object is null.

If we change @martinkuba’s line here:

var req = http.request('http://localhost:8080', function(res) {

to e.g. POST that makes useChunkedEncodingByDefault true.

var req = http.request({host: 'localhost', port: 8080, method: 'POST'}, function(res) {

It works: https://runkit.com/diorahman/post-13461

Not sure if we should throw a meaningful error while doing req.end(<PAYLOAD>) with the client request when the method is not right.

@gireeshpunathil

ClientRequest.end specification does not impose any restrictions on the payload based on the method — so it looks like something else.

[refack fixed formating]

@diorahman

@gireeshpunathil

@diorahman — thanks for the info. I went though the history and came to an inference that malformed GET request can be rejected

However, I see inconsistencies in the client behavior based on the nature of client, nature of server and behavior of the server with respect to handling the parse error:

#cat q.js

'use strict'
const http = require('http')
const net = require('net')
const server = http.createServer((req, res) => {
  if (process.argv[2] === 'handle') {
    server.on('clientError', function(err) {
      console.log('parser error', err.code)
    })
  }
  setTimeout(() => {res.end('hello malformed client!'), 10})
})
server.listen(25000, () => {
  const malformdata = 'me too'
  if (process.argv[3] === 'http') {
    // HTTP CLIENT
    http.request({host: 'localhost', port: 25000, method:'GET', path: '/'}, (res) => {
      let data = ''
      res.on('data', function(d) {
        data += d
      })
      res.on('end', function() {
        console.log(data)
        server.close()
      })
    }).end(malformdata)
  } else {
    // TCP client
    const input = `GET / HTTP/1.1rnHost: localhost:25000rnConnection: closernrn${malformdata}`
    const client = net.createConnection({host:'localhost', port: 25000}, () => {
      client.write(input)
    })
    client.on('error', (e) => {
      console.log(e)
    })
    client.on('data', (data) => {
      console.log('response: ' + data.toString())
      server.close()
    })
  }
})
#node q.js handle http
parser error HPE_INVALID_METHOD
hello malformed client!

#node q.js handle tcp
parser error HPE_INVALID_METHOD
response: HTTP/1.1 200 OK
Date: Fri, 09 Jun 2017 06:07:53 GMT
Connection: close
Content-Length: 23

hello malformed client!
#node q.js nohandle http
events.js:182
      throw er; // Unhandled 'error' event
      ^

Error: socket hang up
    at createHangUpError (_http_client.js:343:15)
    at Socket.socketOnEnd (_http_client.js:435:23)
    at emitNone (events.js:110:20)
    at Socket.emit (events.js:207:7)
    at endReadableNT (_stream_readable.js:1045:12)
    at _combinedTickCallback (internal/process/next_tick.js:102:11)
    at process._tickCallback (internal/process/next_tick.js:161:9)

#node q.js nohandle tcp
^C // HANG
#

Given that node can be acting as HTTP client in many of the distributed computing workloads, or delegating requests to remote end points, I guess the http / net client library need handle this cleanly.

I would like either:

  • Server intercept the parse error, end response with closest HTTP error code mapping to parse error
  • Client intercepting malformed request, refuse to contact server, instead throw an error

@nodejs/http
@nodejs/streams
Also @refack @gibfahn

@gibfahn

cc/ @nodejs/http @nodejs/streams

@mcollina

I am a bit lost on what is the problem here. There have been different threads that points me to separate issues. Is this an http regression, or a stream regression?

Regarding the «socket hang up», I think that error is correct, the server is closing the connection abruptly, and then it’s normal to have that error.

So maybe it was an HTTP change, and I think the current behavior is correct.

@gireeshpunathil can you add the links of the relative PRs?

@gireeshpunathil

@mcollina — I don’t have references to PRs, please treat my previous comment as an absolute problem statement.

I don’t claim it a regression either, as I can reproduce this in earlier versions as well.

If socket ‘hang up’ is the right client behavior, we have these situations:

  1. If server design is to close connection abruptly on malformed requests, then it has to be consistent. In this case,
    • It sends proper response, if we call res.end() from server as soon as it hits the request
    • It sends proper response, if we server has a client error handler.
    • In either case, it does not provide a meaningful message to the client, or the clien’t caller.
  2. If a TCP client is enployed, we get a ‘hang’ instead of a ‘hang up’

(2) means that there is inconsistency at the client side as well.

Now that the clients are net and http, the entity which connect them is ‘stream’. And hence I suspect this has bearing on http as well as streams.

Bottom line is that an error of type Unhandled 'error' event with Error: socket hang up does not qualify for and HTTP response on bad data, it could be proper for a low level network issue where the protocol layer dont have sufficient control.

We could very well reject this as noises through ‘hacks’ but then this behavior poses implications to web apps that forward remote requests that are not validated.

@mcollina

I still do not understand the problem.

The http client does something more to handle the error condition, here is a fixed version of your script, it should produce similar behavior:

'use strict'
const http = require('http')
const net = require('net')
const server = http.createServer((req, res) => {
  if (process.argv[2] === 'handle') {
    server.on('clientError', function(err, sock) {
      console.log('parser error', err.code)
      console.log('destroying socket')
      sock.destroy() // the socket will not be destroyed automatically if you specify clientError
    })
  }
  setTimeout(() => {res.end('hello malformed client!'), 10})
})
server.listen(25000, () => {
  const malformdata = 'me too'
  if (process.argv[3] === 'http') {
    // HTTP CLIENT
    http.request({host: 'localhost', port: 25000, method:'GET', path: '/'}, (res) => {
      let data = ''
      res.on('data', function(d) {
        data += d
      })
      res.on('end', function() {
        console.log(data)
        server.close()
      })
    })
    .on('error', (err) => {
      console.log('error on request', err)
      server.close()
    })
    .end(malformdata)
  } else {
    // TCP client
    const input = `GET / HTTP/1.1rnHost: localhost:25000rnConnection: closernrn${malformdata}`
    const client = net.createConnection({host:'localhost', port: 25000, allowHalfOpen: true })
    client.end(input)
    client.on('error', (e) => {
      console.log('tcp', e)
      server.close()
    })
    client.on('data', (data) => {
      console.log('response: ' + data.toString())
    })
    client.on('end', () => {
      console.log('client ended')
      server.close()
    })
  }
})

setTimeout(function () {
  console.log('normal termination')
}, 1000)

@mcollina

We could very well reject this as noises through ‘hacks’ but then this behavior poses implications to web apps that forward remote requests that are not validated.

Does this mean you would like to accept requests that are not fully validated, in the example GET requests with a body?

@martinkuba

I would like to point out that this was a regression from my point of view. My original repro works fine on previous versions of Node.

@gireeshpunathil

@mcollina — thanks for this. Yes, this produces a consistent behavior.
So bad input is presented to the (node) server in the form of a ‘clientError’ event, and server is free to chose next action on the connection — I think that makes the server robust.

I will review the client side code and get back

@mcollina

@martinkuba that HTTP request is invalid, as there should be no body for GET requests.
That causes the request to abort, and it will emit an 'error'  event.

Yes, it is a change of behavior from 6 and 7, but I think it is better in this way. The request is invalid, and the default behavior in the HTTP server is to destroy the socket without replying. If you want to handle it differently, you should specify a 'clientError' event handler.

cc @nodejs/http @jasnell @mscdex

@bnoordhuis

Technically, HTTP allows GET requests with a body but it’s useless in practice.

Because GET is idempotent, proxies and servers are not allowed to attach meaning to it; all a conforming recipient can do is ignore the body (ref).

Node 8’s behavior is not wrong because it’s not a reasonable thing to do in the first place. I’ll close this out.

@mcollina

To add some more context, in Node 6 and 7 the following has the same behavior of Node 8:

var http = require('http');

var server = http.createServer((req, res) => {
  console.log('got request');
  setTimeout(function () {
    res.write('efgh')
    res.end();
  }, 100)
});

server.listen(8082, function() {
  makeRequest(() => {
    server.close();
  });
});

function makeRequest(cb) {
  var req = http.request('http://localhost:8082', function(res) {
    console.log('got response')
    res.resume();
    cb();
  })
  req.end('abcd');
}

Node 8 made it consistent in all cases.

@Yugloocamai

@amejiarosario

@mcollina I have some clients with this issue. I don’t have control over the client, so I’d like to provide a work on the server using ‘clientError’ event handler as you suggested. However, I’m struggling to get it working. How do I get the original request? I know I can use socket.end to reply but I don’t see how to get the request…

      https.createServer(options, (req, res) => {
        // handle requests
      })
      .listen(port)
      .on('clientError', (err, socket) => {
        console.log('client error ->', err);
        // also handle requests somehow
        socket.end('HTTP/1.1 200 stuffrnrn');
      });

I’m also getting an error (I’m using HTTPS):

client error -> Error: 140737126245312:error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request:../deps/openssl/openssl/ssl/s23_srvr.c:394:

    at Error (native)

@mcollina

@amejiarosario what you get is not a request. You cannot get it. The only thing you can do in 'clientError'  is sending a specialized 5xx code, or customize the error.

@abusada

for fellows still getting this error, i used longjohn which would print long stack traces, it helped me find the root cause of the error.

When a socket hang up is thrown, one of two things happens:

When you’re a customer,

When you send a request to a distant server as a client and don’t get a response in a timely manner. This error is caused by the end of your socket. You should catch this error and decide what to do with it, such as retrying the request or queueing it for later.

If you’re a server or proxy,

When you, as a server, possibly a proxy server, get a request from a client and begin acting on it (or relaying the request to the upstream server), the client decides to cancel/abort the request before you have finished preparing the response.

When a customer cancels a request, this stack trace depicts what happened.

Trace: { [Error: socket hang up] code: 'ECONNRESET' }
    at ClientRequest.proxyError (your_server_code_error_handler.js:137:15)
    at ClientRequest.emit (events.js:117:20)
    at Socket.socketCloseListener (http.js:1526:9)
    at Socket.emit (events.js:95:17)
    at TCP.close (net.js:465:12)

Line http.js:1526:9points to the same socketCloseListener mentioned by @Blender, particularly:

// This socket error fired before we started to
// receive a response. The error needs to
// fire on the request.
req.emit('error', createHangUpError());

...

function createHangUpError() {
  var error = new Error('socket hang up');
  error.code = 'ECONNRESET';
  return error;
}

If the client is a browser user, this is a common scenario. When a request for a resource/page takes a long time to load, visitors simply refresh the page. As a result of this action, the previous request is cancelled, resulting in this error on your server.

Because this issue is the result of a client’s request, they should not expect to receive an error notice. As a result, there’s no reason to consider this error to be important. Simply disregard it. This is bolstered by the fact that on such an error, the res socket that your client was listening to is destroyed, despite the fact that it is still editable.

console.log(res.socket.destroyed); //true

So, no point to send anything, except explicitly closing the response object:

res.end();

However, if you are a proxy server that has already transmitted the request to the upstream, you should abort your internal request to the upstream, signalling that you are uninterested in the response, which will alert the upstream server to maybe halt an expensive activity.

There are two cases when socket hang up gets thrown:

When you are a client

When you, as a client, send a request to a remote server, and receive no timely response. Your socket is ended which throws this error. You should catch this error and decide how to handle it: whether retry the request, queue it for later, etc.

When you are a server/proxy

When you, as a server, perhaps a proxy server, receive a request from a client, then start acting upon it (or relay the request to the upstream server), and before you have prepared the response, the client decides to cancel/abort the request.

This stack trace shows what happens when a client cancels the request.

Trace: { [Error: socket hang up] code: 'ECONNRESET' }
    at ClientRequest.proxyError (your_server_code_error_handler.js:137:15)
    at ClientRequest.emit (events.js:117:20)
    at Socket.socketCloseListener (http.js:1526:9)
    at Socket.emit (events.js:95:17)
    at TCP.close (net.js:465:12)

Line http.js:1526:9points to the same socketCloseListener mentioned by @Blender, particularly:

// This socket error fired before we started to
// receive a response. The error needs to
// fire on the request.
req.emit('error', createHangUpError());

...

function createHangUpError() {
  var error = new Error('socket hang up');
  error.code = 'ECONNRESET';
  return error;
}

This is a typical case if the client is a user in the browser. The request to load some resource/page takes long, and users simply refresh the page. Such action causes the previous request to get aborted which on your server side throws this error.

Since this error is caused by the wish of a client, they don’t expect to receive any error message. So, no need to consider this error as critical. Just ignore it. This is encouraged by the fact that on such error the res socket that your client listened to is, though still writable, destroyed.

console.log(res.socket.destroyed); //true

So, no point to send anything, except explicitly closing the response object:

res.end();

However, what you should do for sure if you are a proxy server which has already relayed the request to the upstream, is to abort your internal request to the upstream, indicating your lack of interest in the response, which in turn will tell the upstream server to, perhaps, stop an expensive operation.

~ Answered on 2015-01-08 07:37:30

There are two cases when socket hang up gets thrown:

When you are a client

When you, as a client, send a request to a remote server, and receive no timely response. Your socket is ended which throws this error. You should catch this error and decide how to handle it: whether retry the request, queue it for later, etc.

When you are a server/proxy

When you, as a server, perhaps a proxy server, receive a request from a client, then start acting upon it (or relay the request to the upstream server), and before you have prepared the response, the client decides to cancel/abort the request.

This stack trace shows what happens when a client cancels the request.

Trace: { [Error: socket hang up] code: 'ECONNRESET' }
    at ClientRequest.proxyError (your_server_code_error_handler.js:137:15)
    at ClientRequest.emit (events.js:117:20)
    at Socket.socketCloseListener (http.js:1526:9)
    at Socket.emit (events.js:95:17)
    at TCP.close (net.js:465:12)

Line http.js:1526:9points to the same socketCloseListener mentioned by @Blender, particularly:

// This socket error fired before we started to
// receive a response. The error needs to
// fire on the request.
req.emit('error', createHangUpError());

...

function createHangUpError() {
  var error = new Error('socket hang up');
  error.code = 'ECONNRESET';
  return error;
}

This is a typical case if the client is a user in the browser. The request to load some resource/page takes long, and users simply refresh the page. Such action causes the previous request to get aborted which on your server side throws this error.

Since this error is caused by the wish of a client, they don’t expect to receive any error message. So, no need to consider this error as critical. Just ignore it. This is encouraged by the fact that on such error the res socket that your client listened to is, though still writable, destroyed.

console.log(res.socket.destroyed); //true

So, no point to send anything, except explicitly closing the response object:

res.end();

However, what you should do for sure if you are a proxy server which has already relayed the request to the upstream, is to abort your internal request to the upstream, indicating your lack of interest in the response, which in turn will tell the upstream server to, perhaps, stop an expensive operation.

Понравилась статья? Поделить с друзьями:
  • Node js error cannot find module express
  • Node js error 404 not found
  • Node js create error
  • Node js cors error
  • Nisa error sims4