cbachellam
August 22, 2015
Dear Readers,
Today is a brand new fresh day and we need to sharpen our brain for another worthwhile reading about the misleading and fallacious reasoning behind code coverage. This blog post debunks the myth that every manager to quality assurance tester and in between, proclaims boldly with seemingly faintest doubt that a 100% coverage means that above par quality standards are achieved. Its not true folks!. Lets dig in and seek the thrill in burning our software dendrites and neurons.
What is code coverage?
We are in the business or rather precisely, engineering of writing code and propping with unit test case suite(UTC) that vet/verify/validate the intention of code, there are tools out there which takes the UTC and maps or matches against the code in reassuring that the UTC in context has executed the real intention of the code and run/tested through all scenarios. Ultimately all code coverage tool after cranking, throws a percentage number that translates exactly to “How much of code has been tested?”. Like if i say the code coverage is 82%, means 82% of code has been tested and 18% of code roughly are untested.
Meaty part of blog: Why specious?
Why can this figure be misleading? We typically depend on coverage analysis tool that come with our IDEs. These IDEs hide the process and how the coverage percentage is measured. Now there are numerous measurement points or parameters , just to mention few of them,
- Line coverage.
- Statement coverage.
- Condition Coverage / Multiple Condition Coverage.
- Loop Coverage.
- Path Coverage.
- Branch Coverage.
- Function Coverage.
I would disclaim here that this blog post is by no means an exhaustive body of literature on metrics and measurement and takes no attempt to describe every engineering key words. Hence the juicy part is:
“ Lets take if we go by the line coverage of a function with 10 lines and UTC has tested all 10 lines- the downside would be: there are languages which has more than one statement per line.In that case 100% line coverage can be totally misleading if the UTC fails to execute and skips statements that are ensconced or nested inside one line[ for e.g.:10 lines = 20 statements]. OR lets take an instance if we go by condition coverage: has each Boolean sub-expression evaluated both to true and false? -for e.g. pseudo logic : if a and b then, here a=true/b=false and a=false/b=true would yield me 100% coverage; again misleading”
I don’t want to stress more on this as it has become explicit now that, what kind of metrics and measurement point/parameters we choose accentuates validity or factuality of the coverage percentage. It would be enlightening to mention here that, one particular third party code coverage tool takes this measurement a notch up by measuring the sequence point which are like drilling down further into statements and going to fine grained intermediate instructions set level and there by measuring code coverage percentages.
Conclusion
Next time when anybody claims 100% coverage report, question the validity and the measurement points that was employed in the process. Because you know it can be fallacious reasoning.
Lets Build a Better World through Better Software!.
Design Nutcracker 