Post-silicon validation: It's the unique fails that hurt you

We have found that logic, design, and architectural bugs do not control the difficulty of bringing up a new microprocessor. Anything that can be reproduced in simulation can be fixed rapidly. The bugs that are hard to reproduce, which occur sporadically, and which don't fail consistently with voltage or temperature are the ones we remember. We describe one such bug, called SSEL for the system error message it caused, which one test engineer said was the strangest bug seen in his long career. It was limited to only one output, and did not occur in other similar outputs. It never failed on a consistent schedule. Failure rates showed a strong correlation with wafer location. Finally, one of the best system level tests for the failure was letting the system sit at the command line prompt, since the failure was not related to system activity. We will describe the characteristics of the bug, the results of experiments with it, our mitigation strategy, our fix, and the root cause. Reliability and availability features built into our servers allowed us to protect customers from the impact of the problem. We will show a large amount of real data from the effort to find the cause of this problem.