Resolved - HTTPS with HttpWebRequest results in exception "An existing connection was forcibly closed by the remote host"

So you are all set to call an external URL to grab some data. You setup your code with something like this

private string GetServiceData()
        {
            string serviceResponse = string.Empty;
            HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(YOUR_HTTPS_URL);
            var response = request.GetResponse() as HttpWebResponse;

            using (Stream receiveStream = response.GetResponseStream())
            {
                StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
                serviceResponse = readStream.ReadToEnd();
            }
            return serviceResponse;
        }

And then you run this code expecting everything to work fine. Unfortunately you meet with the last thing you expected i.e. an exception as below

[SocketException (0x2746): An existing connection was forcibly closed by the remote host]
   System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags) +106
   System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) +130

[IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.]
   System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) +291
   System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count) +32
   System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest) +156
   System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) +59
   System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) +49
   System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest) +162
   System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult) +523
   System.Net.TlsStream.CallProcessAuthentication(Object state) +42
   System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) +193
   System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) +21
   System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) +64
   System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result) +795
   System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size) +52
   System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size) +21
   System.Net.ConnectStream.WriteHeaders(Boolean async) +388

At this point if you are looking for one line answer, scroll down, copy the last piece of code and use it! :).. OR continue reading further for some details

What's going on here. Looking at the code, you figure out that you had never seen this even when you used HttpWebRequest earlier, most likely the method is also being reused from somewhere it worked. Quickly you find out, that the only delta here is the requested url being HTTPS and not HTTP. Off course, you validate first that the https end point in question is accessible and it indeed is.

The error in the stack trace indicates that the connection was attempted but was denied. You have figured this out already, that it has something to do with https. And yes you are right, this is most likely because of HTTPS.

So what's exactly happening here. To understand that, we need to understand how exactly client and server process HTTPS request. There are a good number of articles and videos on this, so you can read in details but the one I referred to is here and one of the specifications is here . In a nutshell, it is actually a multi-step process which happens behind the scene, not seen by the consumer, whether you are using browser or a client library. Most of the exchanges between client and server are one/first time exchanges which is sort of negotiated contract between a law firm and it's client, where both parties are laying down basic rules of how they would take forward the interactions.

In this case, first step is called HandShake (again lot of articles on this specific topic), and in the handshake client begins by sending a hello message with variety of parameters such as ProtocolVersion, CipherSuite, SessionID etc. With this message, client is basically saying I am ready to talk to you but these are a few constraints and properties which I am capable of supporting. Now when this reaches to the server, server has to agree on what it's going to support and then accept this request by sending a response back. If, for any reason, server does not accept any of the parameters being provided, then it results into failure from server and ends up closing the connection since the communication can't continue without agreed parameters by both the parties. As you may have noticed, one of such parameters is ProtocolVersion which refers to SSL/TLS protocol version. Server has to agree on what version of SSL/TLS it wants to support. Generally browsers/clients present a range that they can support, and server can decide the highest which it wants to work with.

Back to our problem, if you see the error says that "existing connection was forcibly closed by the remote host" which means client started a connection which was established, and during this handshake procedure server denied to work with this client and hence closed connection from server side. Now we have to find out, if there is a difference. To do that, you would need to identify which TLS version server is looking for and which one client is supporting. Let's start with the client. To find out which SSL/TLS version one client is using, run your code with a break point just before the HttpWebRequest creation. Once you are on the breakpoint, open the immediate window, and look for the value of ServicePointManager.SecurityProtocol, which may show something like this.

ServicePointManager.SecurityProtocol
Ssl3 | Tls

This means the client in question is capable of work with server supporting any of these two versions i.e SSL3, Tls (which happens to be the Default for .Net framework). Now let's find out now what version server is supporting. And if they mismatch, there is our problem. To find out version that server supports is little tricky. You can find through browser (using these various ways) or through OpenSSL command line. I tried opening my website in question in IE and then right click into empty space and look at properties, and there I found "TLS 1.1, AES with 256 bit encryption (High); DH with 1024 bit exchange". (Why I say tricky because, as you might have read by now in specifications that the server will agree on the highest possible TLS version, so the one shown in browser is not necessarily the only one it supports, it is probably the highest one).

Now as you can see, the client is supporting only SSL3 and Tls where but the server seems to expect TLS1.1 and above. There we take to our solution as below. The good part here is that though you don't know the minimum, you can still specify the range and make the server happy

private string GetServiceData()
        {
            string serviceResponse = string.Empty;            
            ServicePointManager.SecurityProtocol = ServicePointManager.SecurityProtocol | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
            HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(YOUR_HTTPS_URL);
            var response = request.GetResponse() as HttpWebResponse;

            using (Stream receiveStream = response.GetResponseStream())
            {
                StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
                serviceResponse = readStream.ReadToEnd();
            }
            return serviceResponse;
        }

There you are, you should now be able to run your code and see that the server is happy to respond with the response.

I hope you liked this article, feel free to leave a comment if you have any suggestions/thoughts or facing the problem and I would be happy to help.