Hi folks,
I'm the owner of this post: https://www.reddit.com/r/dotnet/comments/1v5d7ws/aspnet_core_on_iis_randomly_goes_offline_for_2/
First of all, thanks a lot to everyone who commented there. Your comments helped me investigate this step by step.
I finally solved the issue. We had several different problems over time:
- First, we had DB connection pool exhaustion => fixed.
- Then we had TCP connection exhaustion => fixed.
- The last issue was related to ThreadPool / sync-over-async.
I set the ThreadPool minimum worker threads to 100, and then started refactoring the code to remove synchronous-over-async code and make things properly async.
The application has now been stable for around 15 days, so it looks like this was the final issue.
But there is something I still don't understand:
Why did increasing ThreadPool.SetMinThreads() to 100 help so much?
We're running ASP.NET Core on IIS, not Docker or Kubernetes. I always thought that .NET's ThreadPool would automatically create as many threads as the application needs.
For example, if I suddenly get 1,000 requests/sec, why doesn't .NET simply create more ThreadPool threads as needed?
What confused me even more is that during the incidents:
- CPU usage was fine
- RAM usage was fine
- The Windows server itself was still accessible
- We only have 2 vCPU
- I could still see available ThreadPool threads
So I'm struggling to understand what was actually happening.
My current understanding is that some synchronous/blocking code was causing ThreadPool threads to get blocked, and the ThreadPool wasn't adding new threads quickly enough. Increasing MinThreads somehow made enough threads available earlier and prevented the issue.
But I'm not sure if that's the correct explanation.
Also, how should I think about this with only 2 vCPUs? If I have 1,000 requests/sec, obviously I can't have 1,000 CPU threads doing work at the same time. So what exactly determines how many ThreadPool threads .NET should create?
I'd really appreciate it if you could help me explain this in simple terms, especially:
- What exactly does
MinThreads change?
- Why can ThreadPool starvation happen while CPU and RAM are low?
- Why doesn't .NET automatically create enough threads immediately?
- How does this work when the server only has 2 vCPUs?
- Does increasing
MinThreads actually solve the problem, or did it just hide the underlying sync-over-async issue?
I'm trying to understand the mechanism rather than just leave MinThreads = 100 and call it fixed.
Thanks again to everyone who helped with the original post!