Today we encountered a very strange issue. One of our clients' ASP.NET based website was running really slow, at least on one of the two servers it is running on. And by slow I mean it was almost unreachable.
Some inspection on the issue showed us that the CPU load on the affected server was somewhere between 95% and 100%. The uptime of the site on this server was some days. And before that the site was already running for some weeks without any problems. So why should something like that happen out of a sudden? Well, as we later found out, it had something to do with thread safety of Dictionaries.
But first things first: ASP.NET websites running under IIS are multithreaded by default. What this means is that the execution of the code is asynchronous and simultaneous.
What happened in our case is the following: We have a class (let's call it SomeContainer) that has a static property that is of type Dictionary. Each time this property is accessed it is referencing the exact same instance - that's just what we want to achieve with this approach. But the way dictionaries are designed they were not meant for this scenario of reading and writing concurrently.
When the entries in a Dictionary change while they are being iterated over, an InvalidOperationException will be thrown saying that the Collection was modified; enumeration operation may not execute. and the iteration steps on to the next entry - where the Exception is thrown again - and so on. You get the point.
This happens endlessly causing a deadlock. The high CPU load only seems to be a symptom, not the problem itself.
To illustrate what exactly happened let's take a look at this code snippet:
public static class SomeContainer {
public static IDictionary<string, int> Static = new Dictionary<string, int>() {
{ "test0", 0 }
};
}This is our container. As you can see it's got one initial entry so there's something to loop over.
The easiest way to provoke this error is to add a new entry while iterating over the dictionary.
foreach (var entry in SomeContainer.Static) {
SomeContainer.Static.Add("test" + entry.Value + 1, entry.Value + 1);
}This will immediately throw an InvalidOperationException. And another one. And even another one. Have a look at the CPU load of this program.
So much for theory. Next we'll construct the rare case that two threads read and write to the dictionary at the same time. For this I'm simply spawning two threads, one that iterates over the dictionary and one that adds new entries to it.
As this case is really rare, I've added some concurrency so that the chance for the threads to access at the same time increases.
For this test case we don't need an initial entry in the dictionary, so let's remove it:
public static class SomeContainer {
public static IDictionary<string, int> Static = new Dictionary<string, int>();
}And finally:
public static void Main(string[] args) {
const int concurrencyCount = 20000;
var writingThread = new Thread(() => {
// add some concurrency to increase our chances for the error to occur
for (var i = 0; i < concurrencyCount; i++) {
SomeContainer.Static.Add("test" + i, i);
}
});
var readingThread = new Thread(() => {
// iterate over the dictionary entries
foreach (var entry in SomeContainer.Static) {
// do something
}
});
// start the threads
writingThread.Start();
readingThread.Start();
Console.ReadLine();
}You may need to run this snippet multiple times for the error to show up... As I said before, it's somewhat shy and only shows up quite seldom :)
So what did we do to prevent this happening again? Luckily, there's a thread save implementation for dictionaries called ConcurrentDictionary that can be used as a drop in replacement.