Sending failed error cannot get url failure when receiving data from the peer

[Question] Debugging «Error: Failure when receiving data from the peer» #809

tl;dr: Can I see the exact request Insomnia is sending so I can reproduce this via cURL and figure out what our internal tooling doesn’t like (maybe some header Insomnia adds)?

Details

I work on a team of

20 developers, and my boss and I both use Insomnia to tinker with the API/platform we build. This is deployed via a Kubernetes-esque orchestration framework, and about a month ago, both of us witnessed a total breakage of our Insomnia. Our auth endpoints still work, but the rest all fail with «Error: Failure when receiving data from the peer». Meanwhile, this extremely basic request (GET + auth header) succeeds using cURL and Postman.

Given that no other issues include the words «peer», I’m guessing this was triggered by some change in our common libraries or the tooling that connects our VMs to one another. Interestingly, the successful cURL and failed Insomnia requests both show as a 200 response in Datadog and within the VM’s logs.

The Question: Since I refuse to go back to Postman quietly, do you have any tips for debugging this on the client side? Maybe there’s a debug mode, or a way I can see the full headers being sent (perhaps reproducing in cURL and tinkering with individual headers)? Or whether any data is received before the error occurs? Obviously I’ll have some server-side debugging to turn on as well, but I’d appreciate anything that can help me see more on the client side.

The text was updated successfully, but these errors were encountered:

Источник

Schannel: Failure when receiving data from the peer #4427

A user of the R bindings has reported a bug that happens in the schannel backend for this particular server:

Which does succceed in retrieving data, but in the end always errors like this:

This is with libcurl 7.64.1, haven’t been able to test yet with 7.66. The problem does not appear when using the openssl backend.

The text was updated successfully, but these errors were encountered:

I tested using master. There has to be a known termination point otherwise an error will occur. (to be clear, close_notify is an acceptable termination point)

I walked through the code from the point of * schannel: server closed the connection and I don’t see anything amiss. I can see in Wireshark an encrypted alert is sent, which could be close notify. Unfortunately I don’t have a way to decrypt schannel traffic, but apparently it’s possible. I also tried with OpenSSL (which I can decrypt) and I see in that case the encrypted alert is a close notify, so it’s possible there’s a bug somewhere in the schannel code. It’s hard for me to say for sure.

I also tried with OpenSSL (which I can decrypt) and I see in that case the encrypted alert is a close notify, so it’s possible there’s a bug somewhere in the schannel code. It’s hard for me to say for sure.

It looks like I confused myself, I have to walk that back there’s no bug that I can see. In Wireshark while running curl w/openssl the close notify I see is sent by the client (libcurl), and curl w/schannel the encrypted alert is sent by the client (libcurl). In either case no alert is received (definitively with OpenSSL no close_notify is received before FIN; in schannel no encrypted alert is received before FIN therefore no close_notify is received). Therefore there is no known termination point.

I have confirmed close_notify alerts work properly as a solo termination point when I test on other websites. I tested by doing this:

By ignoring content length here and telling the server to close the connection on a server that both sends content length and honors client close request we can simulate * no chunk, no close, no size. Assume close to signal end and close_notify as the termination point. It works fine on the sites that return close_notify in that case (not all do).

I’ve since tried your URL in Windows 7 8 10 and the result is the same. I’ve walked through the code again and watched all calls to DecryptMessage and it never returns SEC_I_CONTEXT_EXPIRED (close_notify received).

Based on this I think it’s a server issue, sorry for the confusion.

Thanks for the debugging! Is there anything that can be done to make libcurl robust against such server behavior?

libcurl is robust in handling missing HTTP response length. It will keep reading until TCP close if there is no found or allowed HTTP termination point. If the response is over SSL then libcurl must ensure the integrity which includes length. TCP close is not an acceptable indicator of length for the SSL layer because it may be manipulated by an attacker. As a last resort libcurl may treat the SSL layer termination point, aka close_notify alert, as an indicator of length.

However, as you’ve seen on the contrary, other backends used by libcurl may treat TCP close as an indicator of length when the HTTP response is over SSL and no other termination point was given. That is for legacy reasons (like your server is 15 years old or misconfigured), is not robust, not healthy and I’m wholly against it. I hope one day it will be possible to fix it (starting in openssl.c) without breaking anything for anyone.

Conceivably we could add a flag like CURLSSLOPT_ALLOW_TRUNCATION, CURLSSLOPT_ALLOW_ABRUPT_CLOSURE etc to allow the possibility of the attack similar to what we did for BEAST. At least with schannel I don’t think it would see very much use since I can only recall few reports of this issue, despite Microsoft adding curl w/ schannel to Windows 10 at least a year? ago.

Источник

[Question] Debugging «Error: Failure when receiving data from the peer» #809

tl;dr: Can I see the exact request Insomnia is sending so I can reproduce this via cURL and figure out what our internal tooling doesn’t like (maybe some header Insomnia adds)?

Details

I work on a team of

20 developers, and my boss and I both use Insomnia to tinker with the API/platform we build. This is deployed via a Kubernetes-esque orchestration framework, and about a month ago, both of us witnessed a total breakage of our Insomnia. Our auth endpoints still work, but the rest all fail with «Error: Failure when receiving data from the peer». Meanwhile, this extremely basic request (GET + auth header) succeeds using cURL and Postman.

Given that no other issues include the words «peer», I’m guessing this was triggered by some change in our common libraries or the tooling that connects our VMs to one another. Interestingly, the successful cURL and failed Insomnia requests both show as a 200 response in Datadog and within the VM’s logs.

The Question: Since I refuse to go back to Postman quietly, do you have any tips for debugging this on the client side? Maybe there’s a debug mode, or a way I can see the full headers being sent (perhaps reproducing in cURL and tinkering with individual headers)? Or whether any data is received before the error occurs? Obviously I’ll have some server-side debugging to turn on as well, but I’d appreciate anything that can help me see more on the client side.

The text was updated successfully, but these errors were encountered:

Источник

System.Net.Http.CurlException: Failure when receiving data from the peer when trying to download a file from S3 #613

I don’t know if this is a .Net Core related bug or a AWS SDK one but I found useful to post it here as well.

We get this error randomly when downloading a lot of files from Amazon S3 using AWSSDK.Core (3.3.8.2) and AWSSDK.S3 (3.3.5.7).

System.Net.Http.CurlException: Failure when receiving data from the peer
at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error)
at System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference`1 easyWrapper, CURLcode messageResult)

Here is the code that seems to produce the error:

The text was updated successfully, but these errors were encountered:

Can you provide some more information about how you are able to encounter this error? How many files are you downloading at any given time? How large are they? How consistently are you seeing this issue? What environment are you running this in? Can you post the contents of your project.json as well?

Does the code that you posted reproduce this exception consistently?

The code is a snippet so I can’t tell if you’re disposing of the GetObjectResponse object or its ResponseStream . If you’re not already doing this, you should be.

1/ This function is used by a single thread so files are downloaded one by one, there is no parallel download.

2/ How large files are: It ranges from a few kilobytes to 2-3 MB maximum.

3/ This happens on a completely random basis for the moment and very rarely. On thousands of file downloads it seems to appear once or twice for the moment. I couldn’t establish a pattern and unable to reproduce this exception constantly.

4/ Environment:
Console app with .Net Core 1.1 (netcoreapp1.1) in a standalone application running on Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-70-generic x86_64)
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3

5/ This is a .Net Core 1.1 project, no project.json

6/ Yes I confirm that I dispose everything. Here is what I do at the end of the function:

I am facing the same error with dotnet core 1.0
When I try to create a bucket using AWS S3 in dotnet core
In Debug Mode — Creates the bucket every time successfully
In Run Mode — Fails every time with the error
Failure when receiving data from the peer
at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error)
at System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference`1 easyWrapper, CURLcode messageResult)
System.Net.Http

My Code:
PutBucketRequest newBucket = new PutBucketRequest();
newBucket.BucketName = bucketName;
await client.PutBucketAsync(newBucket);

@epicberry, it looks like your case is different. Please open a new issue here and provide information about the environment where your code works and where it does not. A complete stack trace would be helpful as well.

@flo8, from your code snippet and latest comment it’s not clear what fs is. Can you post the complete code sample? Are you also disposing of response.Result or response.Result.ResponseStream ?

@PavelSafronov please find the complete code below

You’re not disposing of the GetObjectResponse (and are not safely disposing of the FileStream, though that’s unrelated). Here’s a slight rewrite of your code that disposes of the required resources:

That being said, it’s still not clear from the provided stack trace, code, and the linked corefx issue if this is being caused by an issue with .NET Core or the SDK.

Источник

HTTP Client error: Failure when receiving data from from the peer.

We are connecting to Azure BLOB storage through SAP BODS.
We transfer the files to BLOB basically after the data is retrieved and transformation done.
Intermittently we get HTTP Client error: Failure when receiving data from from the peer.
after we get this, we run an adhoc job to transfer the files and then we do not get any issues.
There is no pattern identified for this issue.

There are around 300 jobs that run each day and we may this issue may be 2 times a day or do not face this issue for 3-4 days at all.

Can anyone help me understand what is the cause of this issue and how do I resolve this ?

5 answers

@Shubhadip Barman May I know what kind files( ex: images, doc,, pdf or tar.gz) are you transferring?
Unluckily, I cannot reproduce it in my environment. Can you set a lower data transfer and try to see if it avoid the network errors? If the issue persist in the lower data transfer also, let me know the status and also share the screen shot of the error message

I assume this could happen due to connection reset, usually means network broken.

Workaround: You can try Azcopy

Hope this helps!

Kindly let us know if the above helps or you need further assistance on this issue.

Please don’t forget to «Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

I have attached the screenshot of the error.
We are uploading the .tsv files.

The file size is really not a matter because files with 1.3GB gets transferred easily. The HTTP Client error happens for any random size file, it may contain only 30 records or 400,000 records.

I am not sure when we connect BODS to Azure, is there a gateway URL that we mention or does it happen based on the Portal configuration where we say Azure BLOB ?

@Shubhadip Barman We would like to work closer on this issue, This may require a deeper investigation, so If you have a support plan, I request you file a support ticket, else please do let us know, we will try and help you get a one-time free technical support. In this case, could you send an email to AzCommunity[at]Microsoft[dot]com referencing this thread. Please mention «ATTN subm» in the subject field and subscription ID. Thank you for your cooperation on this matter and look forward to your reply.

Did you get any solution on the issue.

We have same error , loading with SAP BODS Azure blob, it is systematicaly happening on big volume of data ( 100 million rows).
On smaller size we dont hav ethe issue. have you find in between some workarounds?
Thanks,
Joseph

Did you fix your issue with BODS?

We had the same issue in SAP BODS SP12, but it had impact only on 1 or 2 percent of jobs and was not critical for dataflow. After SAP BODS upgrade to SP14 80% of jobs are impacted.

Have you found a RCA of this issue: HTTP client error: : ?

Did you discuss with SAP before upgrade? we are also facing same issue and SAP suggested to upgrade but we never done since no concrete answer from SAP.

We’re using sps8 and jobs will fail randomly 1 or 2 weekly or monthly once more over this is nothing to do with upgrade,
try to check your parameters related to connectivity, network and file size, etc.

DS can open the same file in write mode within a process only once each time. A File location, like Azure Blob, used in a file reader will open a file in write mode in the local folder specified in the File Location.
It will download the file from Azure Blob. Then, it will reopen the local file in read mode and start processing the file.
If all DF using the reader with the same File Location are running sequentially, then the same File Location in a reader can be used multiple time in the same Job.
If WF containing the DF using the reader with same File Location are running in parallel. They will try to open the local file at the same time leading to the error.
2737777 — Error: «HTTP client error: : » happens randomly in DS jobs with Azure data source.

Источник