The Downside of Coverage Testing
| Posted on Wednesday, October 01, 2008 |
0
commentsAdd comments |
This is part two of a two part discussion on Coverage Testing. The complement to this discussion is The Upside of Coverage Testing. These discussions are follow-ups from my blog Stable Builds: Using Ant and QA Tools.
Coverage tools are good, but they aren’t perfect. Just like no one tool can be used for every job, so too it can be said of coverage testing tools. Coverage testing ensures that every class, method, block, and line of code has been reached, but that does not guarantee that each one of these has been tested appropriately or adequately.
Sometimes You Have to Break Code to Fix Code
To prove the point, I decided to introduce a bug into the complete coverage tested
The bug I introduced was the addition of a
The one I added to the
Not only does the new implementation make the mistake of not clearing all the objects from the stack, it compounds the problem by accessing the first object placed on the stack. The latter violation is the worse of the two, because stack by definition is a first in/last out abstract data type. Whimsically removing the first item placed onto the stack changes the definition of what a stack is.
Resetting Splints
Okay, so at this point you should be asking yourself, but won’t your coverage tool just indicate that there is an error with this erroneous implementation. Well, that answer depends on how well your test cases are written.
For example, I can maintain 100% coverage testing of this code by adding just three new lines to my test suite method
Yet, the addition of a new test
Aftermath
Probably the worst aspect of encountering an unexpected or difficult to resolve bug in your code, is believing you have tested for just an event to later find out that just such a bug exists in your project. As I stated in my earlier blog, any testing is better than none. But, testing, especially coverage testing has its weaknesses. Coverage testing assists a developer in reaching every line of code in an attempt to assure him that all of his code has been tested. However, there are many problems with believing that just reaching a line is an adequate test of the line. In my example, it obviously fails, but worse it redefines what a stack is, which is most certainly not what a developer would want to do. Other places where coverage testing fails are reaching lines that contain multiple statements (e.g.,
My
Coverage tools are good, but they aren’t perfect. Just like no one tool can be used for every job, so too it can be said of coverage testing tools. Coverage testing ensures that every class, method, block, and line of code has been reached, but that does not guarantee that each one of these has been tested appropriately or adequately.
Sometimes You Have to Break Code to Fix Code
To prove the point, I decided to introduce a bug into the complete coverage tested
Stack implementation. The bug I introduce isn’t one that would typically go unnoticed in the process of a normal development cycle, but none-the-less, it demonstrates that coverage testing has some serious weaknesses to it.The bug I introduced was the addition of a
popAll() method to clear an existing stack of all objects that had been placed on to it. Since this implementation of a stack uses a List as its underlying implementation, a properly functioning popAll() method would look like:1: public void popAll() {
2: this.elements.clear();
3: }
The one I added to the
Stack implementation looked like:1: public void popAll() {
2: this.elements.remove(0);
3: }
Not only does the new implementation make the mistake of not clearing all the objects from the stack, it compounds the problem by accessing the first object placed on the stack. The latter violation is the worse of the two, because stack by definition is a first in/last out abstract data type. Whimsically removing the first item placed onto the stack changes the definition of what a stack is.
Resetting Splints
Okay, so at this point you should be asking yourself, but won’t your coverage tool just indicate that there is an error with this erroneous implementation. Well, that answer depends on how well your test cases are written.
For example, I can maintain 100% coverage testing of this code by adding just three new lines to my test suite method
testNormalOperation():1: stack.push(one);
2: stack.popAll();
3: assertEquals("Testing stack popAll is clear", 0, stack.toArray().length);
Yet, the addition of a new test
testPopAll() to the test suite clearly demonstrates that the JUnit build process will fail:1: public void testPopAll() throws EmptyStackException {
2: Stack stack = new Stack();
3: stack.push(one);
4: stack.push(two);
5: stack.push(three);
6: stack.popAll();
7: assertEquals("Testing stack popAll is clear", 0, stack.toArray().length);
8: }
Aftermath
Probably the worst aspect of encountering an unexpected or difficult to resolve bug in your code, is believing you have tested for just an event to later find out that just such a bug exists in your project. As I stated in my earlier blog, any testing is better than none. But, testing, especially coverage testing has its weaknesses. Coverage testing assists a developer in reaching every line of code in an attempt to assure him that all of his code has been tested. However, there are many problems with believing that just reaching a line is an adequate test of the line. In my example, it obviously fails, but worse it redefines what a stack is, which is most certainly not what a developer would want to do. Other places where coverage testing fails are reaching lines that contain multiple statements (e.g.,
x++; y = x;), or branching statements where not all conditions are tested (e.g., if ((x == 1) || (y != x))). A coverage tool is part of a developer’s first aid kit in code triage, but the developer has to maintain that it’s not the entire kit.My
Stack implementation with full coverage testing, plus introduced bug: Stack-JUnit-Lab-Reeves-6.0.630.zip