HttpWebRequest class and Threads in .NET

So I was messing around with something this weekend and found something rather annoying about a particular bit of functionality in the .NET framework. This is incredibly rare, actually, so I thought I would blog about it. I was doing two things: multi-threaded coding using the System.Threading.ThreadPool and the System.Net.HttpWebRequest class.

Using the ThreadPool is really easy to do. Using the HttpWebRequest class is easy to do. Getting them to play nicely together takes some work. Take this syntax for example:

ThreadPool.QueueUserWorkItem( new WaitCallback(someMethod) );

That’s all it takes to add some work item to the thread pool. Easy enough. I never thought multi-threaded programming could be so easy! This method, however, used an object which used the HttpWebRequest class. In my testing in the beginning I was queuing maybe 10 items at a time. All the while everything worked very well. I was happy. The program was happy. No exceptions for thread issues were being thrown. Woohoo!

Then I started queuing 100 items at a time. CRASH! It started throwing expections, saying that there were not enough free threads in the thread pool. Of course, I’m thinking "But the ThreadPool class is supposed to handle this for me!" After some looking I found the problem, and I’m not the first to experience this.

The HttpWebRequest class also spawns its own threads, and it doesn’t really communicate with the ThreadPool. So if the ThreadPool gets to its max limit of threads, and then the HttpWebRequest class instances start spawning their own threads, the runtime is going to start throwing exceptions. You can only have so many threads per processor, you know! Not cool! You see the problem?

So, how do you handle this? I heard of one way and thought of another, and ended up going with the latter. I saw it recommended somewhere to handle my own thread pooling. That makes sense, I suppose. But that’s not what I did. So the problem, we see, is related to how many items we put in the ThreadPool at one time, right? Yes. So, you can also solve this problem by limiting that number. In my case, I did this by having a class that tracked the state of my work items. I had a timer that looked at my various collections and checked how many items were in the threadpool at the moment.

First, it would take the amount set by me as the max amount and put those in the ThreadPool.
Second, every second my timer would check back and see how many items had been completed. If, say, 3 had been completed, that means that I had three free threads, so I would add three more. Etc.
Third, when the number of completed items equaled the number of total items, I knew I was finished. At that point I knew to update my UI to show that I was done.

Make sense? If anyone else found a different way to handle this, I’m interested in hearing about it.

I found a knowledge base article thingy on msdn about the topic (can’t remember the url right off the top of my head). It says that this interaction between the ThreadPool and the HttpWebRequest class was actually by design. Why, I haven’t figured out. It is incredibly rare for me to find something about the behavior of the .NET framework that I don’t really like. This is one of those times. But, I survived.