Sometimes I get a bug that I just can't figure out. If the problem is reproducible with a good test case it's usually fairly easy to narrow the problem down pretty quickly. But what do you do if your product crashed once on a customer's server, and hasn't failed again?
Well, you start with the logs and diagnostic files. Usually we can figure it out from tracepoints and a core file. But sometimes this doesn't work.
In cases like this I don't like to throw in the towel without doing something. It feels like defeat (probably because it is). Instead, I always try to figure out what additional information could have helped me solve the problem.
How did we get to the point of failure? If I can identify two or three paths to the failure point and can't infer which one was taken I'll add some tracepoints to those paths. Or maybe I can add assertions on those paths to detect the error a bit earlier.
Since the problem isn't reproducible they won't help me now, but they might help me in the future. If the problem does occur again (and 'not reproducible' really just means 'very rare') hopefully these diagnostics will get me one step closer to the actual problem. And if it didn't hit any of my new tracepoints or assertions when it reoccurs, that's useful (and potentially maddening) too.
Of course I still might not be able to figure out what's happening. Then I add another round of tracepoints and assertions. Each failure gets me one step closer to the solution.
Showing posts with label debugging. Show all posts
Showing posts with label debugging. Show all posts
Friday, December 24, 2010
Saturday, August 28, 2010
Debugging tip: timing holes
Parallel programming is hard. Really hard. And, as we're now firmly in the era of multi-core computing, it's an increasingly important skillset.
A couple of days ago we hit an assertion in some code. The failing test case reproduced the problem 2 out of 300 times. So it was reproducible, but highly intermittent.
The developer who was debugging the problem speculated that there might be a timing hole. There was a section of code which set two variables kind of like this:
He hypothesized that if one thread ran through this code while another thread read the same fields, the second thread might see that the section has red objects, but, briefly, might think they start at NULL instead of the correct address somewhere in the middle of the section.
We used a really simple technique to prove this: make the timing hole bigger.
Adding a 10ms sleep between the two assignments caused the problem to occur nearly every time, confirming his hypothesis.
(We fixed the problem by getting rid of the bool hasRedObjects field. We defined a new function, bool hasRedObjects() { return NULL != baseOfRedObjects; }. With only one field, the updates are atomic and the timing hole is eliminated.)
A couple of days ago we hit an assertion in some code. The failing test case reproduced the problem 2 out of 300 times. So it was reproducible, but highly intermittent.
The developer who was debugging the problem speculated that there might be a timing hole. There was a section of code which set two variables kind of like this:
section->hasRedObjects = true;
section->baseOfRedObjects = basePointer;
He hypothesized that if one thread ran through this code while another thread read the same fields, the second thread might see that the section has red objects, but, briefly, might think they start at NULL instead of the correct address somewhere in the middle of the section.
We used a really simple technique to prove this: make the timing hole bigger.
section->hasRedObjects = true;
usleep(10000); // sleep for 10 ms
section->baseOfRedObjects = basePointer;
(We fixed the problem by getting rid of the bool hasRedObjects field. We defined a new function, bool hasRedObjects() { return NULL != baseOfRedObjects; }. With only one field, the updates are atomic and the timing hole is eliminated.)
Subscribe to:
Posts (Atom)