App Service Plan – Outbound Network Connection Limit

Few days back I ran into a problem where our production azure web apps were throwing below error:

[SocketException (0x271d): An attempt was made to access a socket in a way forbidden by its access permissions x.x.x.x:80] System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) +208 System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception) +464

We opened a case with Microsoft and upon investigation they told us that your App Service Plan (running on Standard S1 2 Instances) are hitting the outbound connection limit. What? How the heck we know that? As of when i am writing, below were the connection limits given my MS.

App Service Plan Connection Limit
Free F1 250
Shared D1 250
Basic B1 1 Instance 1920
Basic B2 1 Instance 3968
Basic B3 1 Instance 8064
Standard S1 1 Instance 1920
Standard S1 2 Instances 1920 per instance
Standard S2 1 Instance 3968
Standard S3 1 Instance 8064
Premium P1 1 Instance (Preview)  1920

 

On further request, MS gave us a table of apps under the app service place and their open socket connection count. It clearly indicates that Web App 1 worker process is not reusing the connection pool and creating new connections hitting the overall limit of the app service plan.

WebApp Name Process Name Open Socket Count
App2 <WebJob>.WebJob.exe 4
App2 <WebJob>.WebJob.exe 4
App2 w3wp.exe 2
App1 <WebJob>.WebJob.exe 8
App1 <WebJob>.WebJob.exe 4
App1 <WebJob>.WebJob.exe 6
App1 w3wp.exe 2
App1 w3wp.exe 1870
App1 <WebJob>.WebJob.exe 6
App1 w3wp.exe 2
App1 <WebJob>.WebJob.exe 6
App3 w3wp.exe 4
App3 w3wp.exe 2
Total 1920

 

With the above data from MS at least you would be able to know where the problem lies and can review the app again.

For your web apps, you can at least review the code (ensuring it doesn't happen to your azure web apps) where you are handing the connection with external entities. Some of the common external dependencies in modern cloud world are:

  1. SQL - https://azure.microsoft.com/en-us/documentation/articles/sql-database-develop-dotnet-simple/
  2. Redis - https://azure.microsoft.com/en-us/documentation/articles/cache-dotnet-how-to-use-azure-redis-cache/
  3. Service Bus - https://azure.microsoft.com/en-us/documentation/articles/service-bus-performance-improvements/

Thanks to the blog http://www.freekpaans.nl/2015/08/starving-outgoing-connections-on-windows-azure-web-sites/ which explains about the same problem.

However the fact is with no monitoring tool available which monitors the open socket count, you will never be able to know the number of open socket connections for your app service plan unless requested from Microsoft.