I recently observed an interesting conversation regarding code coverage standards. It boiled down to one simple question:
Should we enforce mandatory code coverage standards by failing any build with low code coverage?
Well, I think that the answer to this question is a resounding NO, and here is why:
Reason 1: Coverage does not mean Quality
Low code coverage tells you that you don’t have enough tests. High code coverage tells you that you have a lot of tests. Neither tells you anything particularly useful about quality of your production code or, for that matter, quality of your tests. I readily concede that low code coverage is a smell of bad quality, but it’s not a proof. Absence of unit tests is alarming, and should be corrected, but what if your production code is actually pretty good? On the other hand, high code coverage may be achieved by bad test code – imagine battery of unit tests with no asserts in them because what your code returns in unpredictable. Yes, your code coverage is high, but your production code stinks. At best, code coverage is an indirect indicator of overall quality and therefore by no means should be considered a build blocker (something that randomizes your entire team).
Reason 2: You get exactly what you measure
If you evaluate your team success based on the code coverage trends, you will get the positive trends. You might, however, eventually land in a situation when you have good code coverage but bad quality, because when pushed to meet the code coverage bar (thing that you measure), people will cut quality corners. Compare this situation with measuring high-severity bugs that escaped your system all the way to the customer. That is a direct indicator of quality and something that is being done by an independent party (your customer), which is even better!
Reason 3: More process won’t solve people problems
So, we have a fact: Your engineers (notice that I bundle both Developers and Testers in one group) do not write enough unit tests to exercise most of your system. But why? What could be the real story behind this behavior?
There could be multiple scenarios at play here, but all of them point to People problems, not to lack of Process. Breaking low coverage builds constitutes nothing but more process and will not correct one of these possible people problems long-term:
- Our Engineers don’t understand the value of writing good tests.
Well, educate them! Explain that good test suites are not just burden, they are your safety net, your anti-regression armor, your documentation, you design driver, you early design verification and your sample code all in one!
- Our Engineers don’t have time to write tests – they are busy coding the features.
Your estimates don’t include enough time to outfit your features with proper test automation, and your management is apprehensive about spending 10 to 30% more time on each feature. I suggest a simple experiment: pick two features roughly similar in complexity, build one without tests and one with tests. Follow up by calculating number of bugs per feature that will follow in the next release and keep track of total amount of time, including shipping service packs and deploying quick fixes, that was needed in both cases. The results will be self-explanatory. Now you have hard data to take to your management.
- Our Developers don’t think that writing tests is in their job description.
Get another group of developers!
Measure your code coverage religiously and use it as one of the indicators of your overall quality. But don’t rely on it as your only indicator – don’t break your build over it, don’t yell at your engineers when code coverage drops 1%. Instead, make everybody aware why you think code coverage is important and review trends in your retrospective meetings. Encourage or even require your engineers to write tests to cover bugs and interesting usage scenarios, not your coverage targets. Enforce this rule using peer reviews. Correlate high-value bug escapes with components and use cases and adjust your efforts accordingly, so that if you have to improve things quickly, you can focus on unstable and buggy code first.
As a software professional, I’ve been dealing with different Source Control and Build systems for a while now. Just a couple of years ago, Continuous Integration used to be a novelty reserved for the technology thinkers such as Martin Fowler. As any new tool that helped software development process to deviate from the waterfall model and become more adaptive, it was treated with some hostility. Now, of course, the software world is mostly Lean and Agile, and that’s a good thing. At least I think it is.
Anyway, what does it all have to do with your Source Control and Build system? Actually, a lot. As an integral part of the agile software development, the discipline of software configuration management is evolving as well. I submit to you that by just looking at your source control repository and your build processes one can make an educated guess about your Product, your Process and your Team. Let me lay out some “smells” that I consider important and worrysome. For the purposes of this discussion, I am assuming that your organization can answer “Yes” to questions 1, 2 and 3 on The Joel Test.
1. Source Control: You do not Branch or Label every time you Release.
Either you are incredibly good or incredibly lucky because you never had to roll back a bad release or deploy a hotfix to a live system at a point when your developers already moved forward to the new set of features. Don’t worry – it will happen sooner or later. Ask yourself: can you quickly get a copy of your code that matches EXACTLY with what is currently running in production? If not, you should really consider branching or tagging/labeling your code every time you release. Most of the Source Control systems allow this and the process is really easy to implement and automate. I suggest that you take a look at some practical examples from Microsoft’s TFS Branching Guide (it is applicable to most source control systems out there) and pick a strategy that works for you.
2. Source Control: You have multiple branches that are not integrated for a long time
This is a problem opposite to the one above – you have many branches, some private and some public, and you don’t have a definitive version of your product because you don’t integrate your branches often enough. This probably indicates that in your planning you allow too many features to span multiple releases. This, in turn, indicates that you have too many unknowns and you are mitigating the risk by creating branches of the code that you can just kill if things don’t work out. For a developer, nothing can be more frustrating than management deciding to not ship their work. Having many branches increases this risk because it typically (not always) indicates that your product does not have a clear technology roadmap or that you are starting development too early, before you can complete a prototype. Personally, I think that two branches (not counting release branches) are optimal: Main and vNext. Main is your current release, vNext is a branch where you start work on the next biggest most important thing on your roadmap and smaller related things. I don’t really like branching for feature teams because it smells to me like lack of architecture and design.
3. Source Control: You have a dedicated person responsible for merging code
Simply put, your Dev team is not strong enough or your process lacks enough checks to quickly flag bad code. When developers are afraid to merge or build is so weak that it can’t detect when a merge causes a problem, teams will resort to having a person serve a gateway for all merges. This is far from agile and I would suggest that your resources are much better spent fixing the underlying cause rather than dealing with the symptom – instead of creating a bottleneck on your team, review your branching strategy, educate your developers, improve your build instead. Making everybody responsible for merging their code will improve communication on your team when merge conflicts occur and it will make your Developers know more about the overall system.
4. Build: You don’t run unit tests on every check-in
Why not? You don’t have unit tests? They fail? They are too slow? Either way, it is a serious problem indicating lack of good testable product architecture or lack of discipline on the Development team.
5. Build: Your daily builds don’t include system deployment and end-to-end scenario tests
There could be multiple reasons for this. Most likely, it indicates that your product does not have an automated deployment or it is very difficult to configure an internal non-production environment to deploy to. It means that your deployments are painfully slow, manual, and prone to errors, and that your operational costs are sky-high. It also means that your QA team spends more time than they should on certifying your release because of the constant nagging deployment issues they have to investigate. I would consider investing in deployment infrastructure that can be utilized/invoked on every build – thankfully, there are plenty of scripting languages and deployment tools to chose from. When designing this infrastructure, I would keep in mind that your test data is consistent and can be loaded automatically. This is not trivial, but very important thing to do that will allow you to enable automated end-to-end testing of your system during daily builds.
6. Build: You can’t build locally
Unless you are a huge project the size of Windows, there is no reason why your Developers should not be able to execute a limited version of the full build (including local deployment and unit tests) locally. If your build only runs in a dedicated lab, then it is too complicated, which also means slow and prone to errors. You are probably having to keep a dedicated team of build engineers on staff that do nothing by fix build problems all day long. It’s OK to have somebody take responsibility for your build hardware, but it is my heartfelt belief that any developer should be able to run a build locally and not be afraid to examine the build logs, figure out what is wrong and fix your build scripts. If this is not happening, ask yourself if your Development team is cohesive enough or if there is too much specialization and not enough cross-training going on.
7. Build: You don’t measure code coverage
You are flying blind when it comes to testing your product. While high code coverage is not a guarantee of a high quality, low code coverage is a bad sign that your are not testing enough, testing wrong things or not automating your tests. In any case, you can’t measure your overall product quality without knowing what percentage of your system is being tested. There are multiple code coverage tools available on the market, and virtually every tool I know has decent integration with build systems. I would really recommend that you invest a modest amount of your Development time into measuring this on every build.
Hopefully, this was helpful for you. If you disagree or want to talk further about it, feel free to leave a comment.