An error occurred while sending the request system io ioexception the response ended prematurely

For some reason, I’m getting a HttpRequestException with the message «The response ended prematurely. I’m creating about 500 tasks that use my RateLimitedHttpClient to make a request to a website so it can scrape it.

The exception is being thrown from the line return await response.Content.ReadAsStringAsync();.

Is it possible that with 500 tasks, each with ~20 pages to be downloaded and parsed (~11000 total), that I’m exceeding the capability of .Net’s HttpClient?

public class SECScraper
{
    public event EventHandler<ProgressChangedEventArgs> ProgressChangedEvent;

    public SECScraper(EPSDownloader downloader, FinanceContext financeContext)
    {
        _downloader = downloader;
        _financeContext = financeContext;
    }

    public void Download()
    {
        _numDownloaded = 0;

        var companies = _financeContext.Companies.OrderBy(c => c.Name);
        _interval = companies.Count() / 100;

        var tasks = companies.Select(c => ScrapeSEC(c.CIK) ).ToList();
        Task.WhenAll(tasks);
    }
}

public class RateLimitedHttpClient : IHttpClient
{
    public RateLimitedHttpClient(System.Net.Http.HttpClient client)
    {
        _client = client;
        _client.Timeout = TimeSpan.FromMinutes(30);
        ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
    }
    public async Task<string> ReadAsync(string url)
    {
        if (!_sw.IsRunning)
            _sw.Start();

        await Delay();

        using var response = await _client.GetAsync(url);

        return await response.Content.ReadAsStringAsync();
    }

    private async Task Delay()
    {
        var totalElapsed = GetTimeElapsedSinceLastRequest();

        while (totalElapsed < MinTimeBetweenRequests)
        {
            await Task.Delay(MinTimeBetweenRequests - totalElapsed);
            totalElapsed = GetTimeElapsedSinceLastRequest();
        };

        _timeElapsedOfLastHttpRequest = (int)_sw.Elapsed.TotalMilliseconds;
    }

    private int GetTimeElapsedSinceLastRequest()
    {
        return (int)_sw.Elapsed.TotalMilliseconds - _timeElapsedOfLastHttpRequest;
    }

    private readonly System.Net.Http.HttpClient _client;
    private readonly Stopwatch _sw = new Stopwatch();
    private int _timeElapsedOfLastHttpRequest;
    private const int MinTimeBetweenRequests = 100;
}

It appears that I am getting a few HttpRequestExceptions here.

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The response ended prematurely.
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibHttpRateLimitedHttpClient.cs:line 23
   at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 65
   at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 19
   at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperSECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibHttpRateLimitedHttpClient.cs:line 23
   at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 65
   at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 19
   at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperSECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)
   at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibHttpRateLimitedHttpClient.cs:line 23
   at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 65
   at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 19
   at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperSECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The response ended prematurely.
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibHttpRateLimitedHttpClient.cs:line 23
   at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 65
   at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 19
   at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperSECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The response ended prematurely.
>    at System.Net.Http.HttpConnection.FillAsync()
>    at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
>    at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
>    --- End of inner exception stack trace ---
>    at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
>    at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
>    at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
>    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
>    at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
>    at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibHttpRateLimitedHttpClient.cs:line 23
>    at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 65
>    at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 19
>    at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperSECScraper.cs:line 40
> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
>    --- End of inner exception stack trace ---
>    at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
>    at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
>    at System.Net.Http.HttpConnection.FillAsync()
>    at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
>    at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
>    --- End of inner exception stack trace ---
>    at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
>    at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
>    at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
>    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
>    at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
>    at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibHttpRateLimitedHttpClient.cs:line 23
>    at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 65
>    at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 19
>    at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperSECScraper.cs:line 40
> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
>    --- End of inner exception stack trace ---
>    at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)
>    at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
   at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
   at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
   at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibHttpRateLimitedHttpClient.cs:line 23
   at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 65
   at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 19
   at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperSECScraper.cs:line 40
>    at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
   at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
   at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
   at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibHttpRateLimitedHttpClient.cs:line 23
   at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 65
   at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 19
   at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperSECScraper.cs:line 40
>    --- End of inner exception stack trace ---
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
   at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
   at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
   at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibHttpRateLimitedHttpClient.cs:line 23
   at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 65
   at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperEPSEPSDownloader.cs:line 19
   at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:UsersJoshuasourcereposPortfolioOptimizerPOLibSECScraperSECScraper.cs:line 40

What I Broke – Programming and Web Development

Let’s take a look at what I’ve broken today

Http gRPC with .NET Core and Docker – Error starting gRPC call. System.Net.Http.HttpRequestException: An error occurred while sending the request. —> System.IO.IOException: The response ended prematurely.

I’ve been mucking around with gRPC today while using .NET Core and Docker. I have two Microservices in the setup, one server and one client. Because these services will only be communicating internally I intended to use HTTP instead of HTTPS. Unfortunately, I hit the following error while attempting to set this up:

According Microsoft’s eshoponcontainers documentation there are a few extra steps for getting this to work:

Using gRPC without TLS
gRPC works with HTTP/2 only. Usually when a client connects to a server, the connection is done using HTTP1.1 and promoted to HTTP/2 only if both, server and client, support HTTP/2. This promotion is performed using a protocol negotiation, usually implemented using ALPN protocol which requires TLS.

In order to get it to work you need to add the following to your server :

.ConfigureKestrel(options =>
<
options.Listen(IPAddress.Any, 5001, listenOptions =>
<
listenOptions.Protocols = HttpProtocols.Http2;
>);
>);
>);

You then need to explicitly allow HTTP/2 without TLS when creating the client:

Источник

C# HttpClient.SendAsync throw «An error occurred while sending the request» exception when testing some URLs

I am developing an C# console application for testing whether a URL is valid or works. It works well for most of URLs and can get response with HTTP Status Code from target website. But when testing some other URLs, the application throw an «An error occurred while sending the request» exception when running HttpClient.SendAsync method. So I can’t get any response or HTTP Status Code even this URL actually works in the browser. I am desperate to find out how to handle this case. If the URL doesn’t work or the server reject my request, it should at least give me corresponding HTTP Status code.

Here are the simplified code of my test application:

2 Answers 2

If you look at the InnerException you will see that:

«The remote name could not be resolved: ‘www.fc.edu’»

This URL does not work on my browser either.

In order to get an HTTP response you need the client to be able to communicate with the server (even in order to get error 404) and in your case the error occurred at the DNS level.

Some browsers have auto-completion for this kind of cases where if a specific URL is not found, the browser retries with a different suffix/prefix, for example:

But note that you can change your code from:

And you will be able to see what causes your error.

Also, You should never want to catch the generic Exception unless the thrower has thrown the generic Exception , and even than, never catch and do nothing with the exception, at least log it.

Notice than since you wait only for that one task you can use:

Источник

An error occurred while sending the request to the server

This forum has migrated to Microsoft Q&A. Visit Microsoft Q&A to post new questions.

Answered by:

Question

I’ve installed the Azure MP and it seemed to be successful but when the discoveries run they fail with «Message: An error occurred while sending the request.»

I can’t see any other error or anything to help diagnose what the issue is.

I do use a proxy so that did complicate things during the setup but I managed to get the wizard to complete and I could select the resources and metrics etc. so I know it was able to make the connection at that point.

Does anyone have a suggestion?

Answers

• Marked as answer by Stoyan Chalakov MVP Tuesday, May 26, 2020 9:10 AM

have you waited enough?

It is also very important to have the newest version of the managemnt pack, because there were some issues with discoveries in the past versions. Which version are you running?

As always I would strat by examining the «Operations Manager» even log on your management server. Look for related events and check their details. Anything interesting?

Here a known issue, mentioned in the MP guide (a must read):

Issue: “Full scan” in Azure Wizard has been selected and no objects are discovered, warning with id 33333 and error with id 10801 appear in Event Log.

Resolution: Some hidden services such as networkinterfaces/ipConfigurations, networkSecurityGroups/defaultSecurityRules, networkSecurityGroups/securityRules, virtualNetworks/subnets that obtained with Full Scan might cause discovery failures. Uncheck additional services and their parents in Service types tab, uncheck “Full scan” checkbox in Subscription tab and press [Refresh Data].

Источник

selenoid/chrome: An error occurred while sending the request. Couldn’t connect to server #42

We’re just trying to run selenium auto-case in Selenoid container, but failed with request timeout.
_

Starting ChromeDriver 2.39.562737 (dba483cee6a5f15e2e2d73df16968ab10b38a2bf) on port 35182
Only local connections are allowed.
An error occurred while sending the request. Couldn’t connect to server

The HTTP request to the remote WebDriver server for URL http://localhost:35182/session timed out after 60 seconds.

The text was updated successfully, but these errors were encountered:

• What is your environment?
• How are you starting container?
• What is your client?
• What is your test Selenium URL?

Here are the configurations we’ve done:

Create Jenkins pipeline to get source code from GitHub and call the Jenkinsfile which stored under the project root folder.
_pipeline <
agent <
dockerfile <
args ‘-u root:root’
>
>

>_
2. Jenkinsfile will call dockerfile to install solenoid image.
FROM selenoid/chrome
3. Run auto-case.

1. What’s the solenoid/chrome image doing? Install Chrome, ChromeDriver? and the browser.json? (where it’s stored)
2. How to specify the port?
3. In my dotnetcore code, we just initial chrome driver as driver = new ChromeDriver();

Here url should be something like: http://localhost:4444/ (for Chromedriver path is / instead of /wd/hub ). Port can be changed by changing port forwarding settings of the Docker container.
2) Image build files are here: http://github.com/aerokube/selenoid-images/

I’ve refined my code to initialize chrome driver as below:
_var url = «http://localhost:4444/»;
driver = new RemoteWebDriver(new Uri(url), D

esiredCapabilities.Chrome());_
, but still get similar issues:
A exception with a null response was thrown sending an HTTP request to the remote WebDriver server for URL http://localhost:4444/session. The status of the exception was UnknownError, and the message was: An error occurred while sending the request. Couldn’t connect to server

Attach my jenkinsfile and dockerfile for reference:

Do I need to do much more configurations against Selenoid container? Thanks.

@Benny86 just make sure Docker container is listening on port 4444. According to your error — it does not.

Actually, we install Selenoid image in dockerfile (which called by Jenkins pipeline: Jenkinsfile), and we also EXPOSE 4444 after install Selenoid image in dockerfile.

In my option, they are running in same container, but execute: RUN curl http://[::]:4444, we got below errors:
_Step 13/13 : RUN curl http://[::]:4444
—> Running in 170afcdd529c
�[91mc�[0m�[91mu�[0m�[91mr�[0m�[91ml�[0m�[91m:�[0m�[91m �[0m�[91m(�[0m�[91m7�[0m�[91m)�[0m�[91m �[0m�[91mC�[0m�[91mo�[0m�[91mu�[0m�[91ml�[0m�[91md�[0m�[91mn�[0m�[91m’�[0m�[91mt connect to server

�[0mThe command ‘/bin/sh -c curl http://[::]:4444’ returned a non-zero code: 7_

Based on the Get_started document, if we run solenoid.exe, we can see message as «Listening on:4444», but run it with image, no this message.

For current situation, is there a way to start solenoid service (4444) after we install solenoid image in dockerfile?
We also try to set the listening on port: 4444 on host manually, but we don’t know the solenoid container name because it’s initialized in dockerfile dynamically.

Even through run below cmd in host:
docker run -d —name selenoid -p 4444:4444 -v /var/run/docker.sock:/var/run/docker.sock -v pwd /config/:/etc/selenoid/:ro aerokube/selenoid:latest-release

, and then we see Selenoid container created, but still cannot request to http://localhost:4444 on host
curl http://loalhost:4444

Do I need to do much more configurations?

@Benny86 if Selenoid container is in running state, then port 4444 should be open. If it is not open — then check Docker network connectivity. For example CentOS is known to have issues with Docker and SELinux.

Double check the selenoid container logs:
docker logs b9
2018/06/20 08:39:11 [-] [INIT] [Loading configuration files. ]
2018/06/20 08:39:11 [-] [INIT] [/usr/bin/selenoid: browsers config: read error: open /etc/selenoid/browsers.json: no such file or directory]

seems we need to run Prepare configuration before start selenoid container on host
1.Create config/browsers.json configuration file with content
2.Pull browser Docker image:
$ docker pull selenoid/vnc:firefox_57.0.

please confirm. Thanks.

@Benny86 yes, certainly — you have to create JSON config first. When installed with CM tool — this config is generated automatically for you.

Thanks. I will do this one on host.

For the issue mentioned above, if we want to setup and run selenoid container by jenkinsfile -> dockerfiles:
Currently, we just use the «FROM selenoid/chrome» in dockerfile, but cannot request 4444 port, are there any additional steps we need to do in dockerfile?

Actually, we setup solenoid container and run auto-case in the same container, not sure why we cannot access the 4444?
Attach the Jenkinsfile and dockerfile here:
Jenkinsfile_Dockerfile.zip

Additional:
We’ve setup the Selenoid container on host follow below steps in «Get Started» document, and 4444 is up.
1.Create config/browsers.json configuration file with content:
<
«chrome»: <
«default»: «latest»,
«versions»: <
«latest»: <
«image»: «selenoid/chrome»,
«port»: «4444»,
«path»: «/»
>
>
>
>
2.Pull browser Docker image:
$ docker pull selenoid/chrome
3. start Selenoid container
docker run -d —name selenoid -p 4444:4444 -v /var/run/docker.sock:/var/run/docker.sock -v pwd/config/:/etc/selenoid/:ro aerokube/selenoid:latest-release
4. check status
_docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bc4444a35c88 aerokube/selenoid:latest-release «/usr/bin/selenoid -…» 9 minutes ago Up 9 minutes 0.0.0.0:4444->4444/tcp selenoid

but if we setup solenoid container by dockerfile, the 4444 port cannot be accessed. Seems this 2 way are run under different behaviors? Please double check.
FROM selenoid/chrome

Again, if we call the «FROM selenoid/chrome» in dockerfile to setup solenoid container, what’s kinds of additional settings need to be considered in dockerfile? Thanks.

Источник

An error occurred while sending the request. #711

If I refresh a page particularly if it is still loading I get a 502 error. Is there any way I can capture exceptions and prevent the 502 from being seen in the browser? The errors can be any of these:

The text was updated successfully, but these errors were encountered:

One way I have managed to supress it is by changing the EndPointDefaults in the Kestrel config on the destination to Http1

Issue created from discussion #710

I’m not able to reproduce this. The client reports an aborted request, it does not receive the 502 response. A 502 is what we’d return if the proxy request failed before the response started, but a client browser refresh aborts the client request so the 502 response is never received.

Let’s get some baseline data.
What version of YARP are you using?
What version of .NET Core is this?
What kind of request is in progress, and what is it doing that’s taking long enough for you to refresh the browser before it’s done loading?
Can you simplify your config to only the relevant settings?
Please share the proxy logs, turned up to Debug for everything.

Using YARP 1.0.0-preview.8.21065.1

Framework version net5.0

I found out it’s not actually caused by a browser refresh (though doing this does cause a 502 aproximately half of the time though I did not press refresh at all this time round).

Behind YARP is a web application built using React with Apollo GraphQL client talking through YARP to consume an authenticated GraphQL Api.

Both the application and api are hosted in Kestrel. Apollo client is set to poll graphQL for changes every 2 seconds and this works fine when I remove YARP and go directly to the destination endpoint and also works via YARP if I configure Kestrel to explicitly use Http1.

I simplified the YARP config by removing header stuff from the Route transforms.

This is the Kestrel config I have on the GraphQL Api.

Here’s the full debug log from YARP up until the exception is caught.

Источник

Chaminda’s DevOps Journey with MSFT

DevOps with Azure DevOps

Thursday, 15 October 2020

Resolving “System.IO.IOException: The response ended prematurely.” in HttpClient Request

Some times a silly mistake can waste lot of time of a developer. The exception “System.IO.IOException: The response ended prematurely.” while making an http client request to call an API from another web app has wasted considerable amount of my time diagnosing the issue. Therefore, thought worth sharing the experience.

The error reported was below when an API request via http client was made from one of the web applications. Strangely the other web application handled the same request with no problems and returned the expected data.

Sending HTTP request GET http://localhost:5001/api/v1/businessTypes/1/ServiceCategories

An unhandled exception has occurred while executing the request.

System.Net.Http.HttpRequestException: An error occurred while sending the request.

—> System.IO.IOException: The response ended prematurely.

at System.Net.Http.HttpConnection.FillAsync(Boolean async)

at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean async, Boolean foldedHeadersAllowed)

at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)

— End of inner exception stack trace —

This caused me to look at each implementation compared closely and I could not see any difference. Exactly same code patterns were implemented so I was pulling my hair for some time with this one, as it seems something somewhere wrong but was 100% certain it is not the code as there was no difference at all. After spending couple of hours searching for this “The response ended prematurely.” in web there was no proper reasoning could be found.

Источник

Why do I get «The response ended prematurely» when using the HttpClient in C#?

My first code of HTTP Client in .Net Core. The server is running on my PC as well, so I want to send it to localhost.

I’ve added printing of the inner exception after someone recommended it here but it’s not enough for me to understand the problem.

The error I received from the loop inside the catch is:

How can I figure out why can’t I send a message to the server? Does the content of the message can disrupt the connection generation?

I checked the server with another client and it’s working so the problem has to be in my code.

EDIT: The server is using https so I changed my URL to https as well and this is the error I received:

Do I need to add some SSL commands in order to make a secure connection?

Edit2: The new error after adding a command in callURL()

2 Answers 2

System.IO.IOException: The response ended prematurely means the server to which you’re making an HTTP request closed the connection without having sent back a complete response.

It has nothing to do with your client code, as it looks to compile & send content correctly.

Yes, you should be using await client.PostAsync but that isn’t related at all to your question.

Double-check that http://127.0.0.1:5001 is 100% correct — maybe it’s https:// or not on port 5001 .

If it is an endpoint using HTTPS, you may get System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure. as it looks like you’re calling a local endpoint.

.NET does SSL certificate validation by default & will throw an exception if you are trying to call an HTTPS endpoint for which you have no/invalid SSL certificate. You probably have not set up the localhost certificate correctly on your machine.

Initialise your HttpClient like below:

However, please do note allowing all SSL certificates is a security risk if used on anything other than your local dev environment.

Источник

An error occurred while sending the request system io ioexception the response ended prematurely

#c# #http #.net-core #httpclient

Вопрос:

Мой первый код HTTP-клиента в .Net Core. Сервер также работает на моем компьютере, поэтому я хочу отправить его на локальный хост.

Я добавил печать внутреннего исключения после того, как кто-то рекомендовал его здесь, но этого недостаточно, чтобы я понял проблему.

Ошибка, которую я получил из цикла внутри ловушки, состоит в том, что:

Как я могу понять, почему я не могу отправить сообщение на сервер?
Может ли содержимое сообщения нарушить создание соединения?

Я проверил сервер у другого клиента, и он работает, поэтому проблема должна быть в моем коде.

ИЗМЕНИТЬ: Сервер использует https, поэтому я также изменил свой URL на https, и это ошибка, которую я получил:

Нужно ли мне добавлять некоторые команды SSL, чтобы установить безопасное соединение?

Edit2: Новая ошибка после добавления команды в callURL()

Комментарии:

1. Сделайте catch (Exception err) , а затем Console.WriteLine(err.ToString()); — каковы полные сведения об исключении?

2. HttpClient использует асинхронный ввод — поэтому вы должны использовать async методы с await . Вы не должны использовать .Result , потому что это может привести к взаимоблокировкам.

3. @ErmiyaEskandary Значение ошибки: msg: Произошла одна или несколько ошибок. (При отправке запроса произошла ошибка.) источник: Система. Частное. Трассировка ядра: в системе. Нарезание резьбы. Задачи.Задачи. ThrowIfExceptional(логические исключения includeTaskCanceledExceptions) в системе. Нарезание резьбы. Задачи. Задача 1.GetResultCore(Boolean waitCompletionNotification) at System.Threading.Tasks.Task 1.get_Result() в HttpClient. Program.callURL(URL-адрес строки) в C:sourceHTTPClientHTTPClientProgram.cs:line 84

4. Правильно ли 127.0.0.1:5001 на 100%? Работает ли сервер? Ваш код работает так, что бы это ни было, это серверная сторона

5. @ErmiyaEskandary Я только что увидел, что сервер использует https, и я использовал HTTP, поэтому я изменил HTTP на https, и я получил новую ошибку (В сообщении)

Ответ №1:

System.IO.IOException: The response ended prematurely означает, что сервер, на который вы отправляете HTTP-запрос, закрыл соединение, не отправив полный ответ.

Это не имеет никакого отношения к вашему клиентскому коду, так как он правильно компилирует и отправляет контент.

Да, вы должны использовать await client.PostAsync , но это никак не связано с вашим вопросом.

Дважды проверьте, что http://127.0.0.1:5001 это на 100% правильно — возможно, это https:// или нет на порту 5001 .

Если это конечная точка, использующая HTTPS, вы можете получить System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure. , как это выглядит, если вы вызываете локальную конечную точку.

.NET по умолчанию выполняет проверку SSL-сертификата и создаст исключение, если вы пытаетесь вызвать конечную точку HTTPS, для которой у вас нет/недействительного SSL-сертификата. Возможно, вы неправильно настроили сертификат localhost на своем компьютере.

Инициализируйте ваше HttpClient подобное ниже:

Однако, пожалуйста, обратите внимание, что разрешение всех SSL-сертификатов представляет угрозу безопасности, если они используются в чем-либо, кроме вашей локальной среды разработки.

Комментарии:

1. Теперь это работает. Очень странно.. Большое спасибо!

2. @RoyAncri Не пропускайте проверку сертификации в производстве, однако для разработки это будет нормально. Установка правильного SSL — сертификата на производственном сервере позволит вам удалить приведенный выше код, который представляет угрозу безопасности, если оставить его в таком виде на чем-либо, кроме сборок разработчиков-надеюсь, это поможет!

3. @ErmiyaEskandary Да, я прочитаю, как это сделать для производственного проекта. Еще раз спасибо

Источник

Exception while calling an external API from .NET Core 3.1 — HttpClient #51077

Comments

joshymonmc commented Apr 11, 2021 •

System.Net.Http.HttpRequestException: An error occurred while sending the request. System.IO.IOException: The response ended prematurely.

We are trying to access an external REST API (runs on Java 8, hosted on Solaris linux, Tomcat) from an application on a Windows machine,.NET Core 3.1, using HTTPClient. We try with a PostAsync request.

What we observe is when we make the call first time, if fails with an exception. And the next time, if we make the request within 15 second, the request becomes successful.

And the same API request works fine with Postman, irrespective of whether its first request or second request.

Part of Code

Note : The external apiURL is https and the API server is behind a load balancer.

Part of Exception details:

The exception occurs after 30 seconds, from the request is made.

What we tried so far:

1. Changed the Async Post call to sync, no change in the result.
2. Suggested several options mentioned in the internet like
   a. setting TLS verions to 1, 1.1, 1.2 for the request
   b. made DefaultRequestHeaders.ConnectionClose = true
   c. tried with UseCookies = false
   d. Used .NET Framework 4.8 instead of .NET Core

but no luck so far.

1. We are not able to make any changes on the API hosted on Solaris server, since its within our client’s corporate network.

Looking forward to the support to resolve the issue at the earliest. Appreciate an early response!

To Reproduce

Exceptions (if any)

Further technical details

• ASP.NET Core version 3.1
• The IDE VS Code on Windows

The text was updated successfully, but these errors were encountered:

msftbot bot commented Apr 11, 2021

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

System.Net.Http.HttpRequestException: An error occurred while sending the request. System.IO.IOException: The response ended prematurely.

We are trying to access an external REST API (runs on Java 8, hosted on Solaris linux, Tomcat) from an application on a Windows machine,.NET Core 3.1, using HTTPClient. We try with a PostAsync request.

What we observe is when we make the call first time, if fails with an exception. And the next time, if we make the request within 15 second, the request becomes successful.

And the same API request works fine with Postman, irrespective of whether its first request or second request.

Part of Code

HttpResponseMessage response = client.PostAsync(
apiUrl, new StringContent(postData, Encoding.UTF8,
«application/json»)).Result;
apiResponse = response.Content.ReadAsStringAsync().Result;

Note : The external apiURL is https and the API server is behind a load balancer.

Part of Exception details:

The exception occurs after 30 seconds, from the request is made.

System.Net.Http.HttpRequestException: An error occurred while sending the request.
—> System.IO.IOException: The response ended prematurely.
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
— End of inner exception stack trace —
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

What we tried so far:

1. Changed the Async Post call to sync, no change in the result.
2. Suggested several options mentioned in the internet like
   a. setting TLS verions to 1, 1.1, 1.2 for the request
   b. made DefaultRequestHeaders.ConnectionClose = true
   c. tried with UseCookies = false
   d. Used .NET Framework 4.8 instead of .NET Core

but no luck so far.

1. We are not able to make any changes on the API hosted on Solaris server, since its within our client’s corporate network.

Looking forward to the support to resolve the issue at the earliest. Appreciate an early response!

Источник

What I Broke – Programming and Web Development

Let’s take a look at what I’ve broken today

Http gRPC with .NET Core and Docker – Error starting gRPC call. System.Net.Http.HttpRequestException: An error occurred while sending the request. —> System.IO.IOException: The response ended prematurely.

I’ve been mucking around with gRPC today while using .NET Core and Docker. I have two Microservices in the setup, one server and one client. Because these services will only be communicating internally I intended to use HTTP instead of HTTPS. Unfortunately, I hit the following error while attempting to set this up:

According Microsoft’s eshoponcontainers documentation there are a few extra steps for getting this to work:

Using gRPC without TLS
gRPC works with HTTP/2 only. Usually when a client connects to a server, the connection is done using HTTP1.1 and promoted to HTTP/2 only if both, server and client, support HTTP/2. This promotion is performed using a protocol negotiation, usually implemented using ALPN protocol which requires TLS.

In order to get it to work you need to add the following to your server :

.ConfigureKestrel(options =>
<
options.Listen(IPAddress.Any, 5001, listenOptions =>
<
listenOptions.Protocols = HttpProtocols.Http2;
>);
>);
>);

You then need to explicitly allow HTTP/2 without TLS when creating the client:

Источник

Hi everyone,

I’ve been mucking around with gRPC today while using .NET Core and Docker. I have two Microservices in the setup, one server and one client. Because these services will only be communicating internally I intended to use HTTP instead of HTTPS. Unfortunately, I hit the following error while attempting to set this up:

Error starting gRPC call. System.Net.Http.HttpRequestException: An error occurred while sending the request. —> System.IO.IOException: The response ended prematurely.

According Microsoft’s eshoponcontainers documentation there are a few extra steps for getting this to work:

Using gRPC without TLS
gRPC works with HTTP/2 only. Usually when a client connects to a server, the connection is done using HTTP1.1 and promoted to HTTP/2 only if both, server and client, support HTTP/2. This promotion is performed using a protocol negotiation, usually implemented using ALPN protocol which requires TLS.

In order to get it to work you need to add the following to your server :

You then need to explicitly allow HTTP/2 without TLS when creating the client:

Note that the AppContext.SetSwitch statements need to appear before the client is created to work. There’s also a bit of an overview on the following page: https://github.com/dotnet-architecture/eShopOnContainers/wiki/gRPC

The following stackoverflow post also helped with the ordering issue: https://stackoverflow.com/a/58053460/522859

HttpClient handles redirects automatically. When you send a request, if the response contains a redirect status code (3xx) and redirect location, then it’ll send a new request to the redirect location.

You can turn off this auto-redirect behavior by passing in an HttpClientHandler with AllowAutoRedirect=false. This prevents it from following redirects automatically, and allows you to handle redirects manually if you want:

var handler = new HttpClientHandler()
{
   AllowAutoRedirect = false
};
var client = new HttpClient(handler);

var response = await client.GetAsync("http://api.isevenapi.xyz/api/iseven/7/");

Console.WriteLine($"Status code = {(int)response.StatusCode}");
Console.WriteLine($"Redirect location = {response.Headers.Location}");
Code language: C# (cs)

Reminder: Be sure to reuse a single instance of HttpClient.

This outputs the following:

Status code = 301
Redirect location = https://api.isevenapi.xyz/api/iseven/7/
Code language: plaintext (plaintext)

This is an example of a typical HTTP to HTTPS redirect.

Default redirect behavior

HttpClient uses the RedirectHandler class for dealing with redirects. I’ll explain the default behavior below.

Redirect conditions

It will redirect based on the following conditions:

• Response.StatusCode is 300, 301, 303, 303, 307, or 308.
• Response.Headers.Location (URI) is populated. Note: It can handle absolute and relative URIs.
• It will not do an insecure HTTPS to HTTP redirect.

It will do up to 50 redirects (configurable).

If you get a redirect response code (3xx) back at the end, this means the redirect conditions were not met, it exceeded the max redirect attempts, or there’s something wrong in the redirect data. If this is happening to you, you can troubleshoot it by turning off auto-redirects and examining the response(s).

Original request headers and content

The original request headers (except the auth header) and content are retained and sent in redirect requests. Query string parameters aren’t retained automatically.

If you’re handling redirects manually, you could combine the original query string parameters with the redirect location URI (perhaps using a UriBuilder).

Note: HttpClient internally reuses the same HttpRequestMessage object for redirects. If you try to do this yourself, you’ll get an exception “Cannot send the same request message multiple times.” The framework uses the internal MarkAsRedirected() method to be able to reuse the same request object repeatedly. If you really want to reuse the same request object yourself, you can use reflection to call the internal method. It’s probably simpler to just create a new request object.

Forced GETs

For certain combinations of status codes and HTTP methods, it will send the redirect as a GET instead of using the original HTTP method. For example, if you get a 302 response code when doing a POST request, it’ll do the redirect as a GET.

This is clearly a problem if the redirect location doesn’t allow a GET. In this scenario, you’d get the following misleading exception:

System.Net.Http.HttpRequestException: Response status code does not indicate success: 405 (Method Not Allowed).

It’s misleading because you sent a POST request, which was allowed. It sent a redirect request as a GET, which was disallowed. It’d only be obvious if you knew a redirect happened. It would be much more clear if the error message said something about it failing during a redirect attempt.

I’m not sure about the reasoning behind this Forced GET behavior. This is a pretty good example of why it might be better to disable auto-redirects. That way you avoid unintended behavior like this.

Check if a request was automatically redirected

One simple way to check if your request got automatically redirected is by comparing the original request URI with the response’s request URI. If they’re different, it got redirected.

var client = new HttpClient();

var requestUri = new Uri("http://api.isevenapi.xyz/api/iseven/7/");
var response = await client.GetAsync(requestUri);

if (requestUri != response.RequestMessage.RequestUri)
{
	Console.WriteLine($"Request was redirected to {response.RequestMessage.RequestUri}");
}
Code language: C# (cs)

Note: This is a heuristic.

This outputs the following:

Request was redirected to https://api.isevenapi.xyz/api/iseven/7/
Code language: plaintext (plaintext)

Limit the number of redirects

By default, HttpClient will do up to 50 redirects. You can control this with the HttpClientHandler.MaxAutomaticRedirections setting.

Here’s an example:

var handler = new HttpClientHandler()
{
	MaxAutomaticRedirections = 1
};

var client = new HttpClient(handler);
var response = await client.GetAsync("https://localhost:12345/movies/find");
response.EnsureSuccessStatusCode();
Code language: C# (cs)

When this exceeds the max redirect attempts, it’ll return the response from the last redirect request it attempted. Since this response will have a 3xx status code, EnsureSuccessStatusCode() will throw an exception.

HTTP to HTTPS redirects fails locally

I have an ASP.NET Core web API running locally. It’s configured to do HTTP to HTTPS redirects. To test redirect behavior, I sent the following request:

var client = new HttpClient();
var response = await client.GetAsync("http://localhost:12345/movies/find");
Code language: C# (cs)

It fails with the following exception:

System.Net.Http.HttpRequestException: An error occurred while sending the request.
—> System.IO.IOException: The response ended prematurely.

This error doesn’t have to do with HttpClient’s auto-redirect behavior. The error is happening before that. I also tried using Postman and got the same error. So this problem isn’t specific to HttpClient. I also tried running the web API in a local Windows Sandbox and had the same issue.

This is mostly just a heads up. I didn’t dig deeper into this. If you’re just wanting to test HTTP to HTTPS redirects, be sure the programs (client and web API) are on different machines. And if you have two programs on the same machine that need to talk, use HTTPS (to make HTTP to HTTPS redirects a non-issue).