I recently observed an interesting conversation regarding code coverage standards. It boiled down to one simple question:
Should we enforce mandatory code coverage standards by failing any build with low code coverage?
Well, I think that the answer to this question is a resounding NO, and here is why:
Reason 1: Coverage does not mean Quality
Low code coverage tells you that you don’t have enough tests. High code coverage tells you that you have a lot of tests. Neither tells you anything particularly useful about quality of your production code or, for that matter, quality of your tests. I readily concede that low code coverage is a smell of bad quality, but it’s not a proof. Absence of unit tests is alarming, and should be corrected, but what if your production code is actually pretty good? On the other hand, high code coverage may be achieved by bad test code – imagine battery of unit tests with no asserts in them because what your code returns in unpredictable. Yes, your code coverage is high, but your production code stinks. At best, code coverage is an indirect indicator of overall quality and therefore by no means should be considered a build blocker (something that randomizes your entire team).
Reason 2: You get exactly what you measure
If you evaluate your team success based on the code coverage trends, you will get the positive trends. You might, however, eventually land in a situation when you have good code coverage but bad quality, because when pushed to meet the code coverage bar (thing that you measure), people will cut quality corners. Compare this situation with measuring high-severity bugs that escaped your system all the way to the customer. That is a direct indicator of quality and something that is being done by an independent party (your customer), which is even better!
Reason 3: More process won’t solve people problems
So, we have a fact: Your engineers (notice that I bundle both Developers and Testers in one group) do not write enough unit tests to exercise most of your system. But why? What could be the real story behind this behavior?
There could be multiple scenarios at play here, but all of them point to People problems, not to lack of Process. Breaking low coverage builds constitutes nothing but more process and will not correct one of these possible people problems long-term:
- Our Engineers don’t understand the value of writing good tests.
Well, educate them! Explain that good test suites are not just burden, they are your safety net, your anti-regression armor, your documentation, you design driver, you early design verification and your sample code all in one!
- Our Engineers don’t have time to write tests – they are busy coding the features.
Your estimates don’t include enough time to outfit your features with proper test automation, and your management is apprehensive about spending 10 to 30% more time on each feature. I suggest a simple experiment: pick two features roughly similar in complexity, build one without tests and one with tests. Follow up by calculating number of bugs per feature that will follow in the next release and keep track of total amount of time, including shipping service packs and deploying quick fixes, that was needed in both cases. The results will be self-explanatory. Now you have hard data to take to your management.
- Our Developers don’t think that writing tests is in their job description.
Get another group of developers!
Measure your code coverage religiously and use it as one of the indicators of your overall quality. But don’t rely on it as your only indicator – don’t break your build over it, don’t yell at your engineers when code coverage drops 1%. Instead, make everybody aware why you think code coverage is important and review trends in your retrospective meetings. Encourage or even require your engineers to write tests to cover bugs and interesting usage scenarios, not your coverage targets. Enforce this rule using peer reviews. Correlate high-value bug escapes with components and use cases and adjust your efforts accordingly, so that if you have to improve things quickly, you can focus on unstable and buggy code first.