The Benefits of Manual and Automated Quality Assurance Methodologies
| Posted on Wednesday, September 24, 2008 |
0
commentsAdd comments |
When I first start thinking of this topic, I perceived the relationship between manual and automated quality assurance methodologies as an either/or relationship. I believed the relationship to be that of one being a better suited methodology over the other. Of course, that tendency would be to favor automation tools over a manual process. However, this is a fundamentally flawed belief.
The longer I thought on the topic, the more I came to the realization that each has its strengths complementing the weaknesses the other methodology exhibits. This becomes evident in analyzing the overlaps and disparity between peer reviews and the results of running quality assurance tools (QATs) against the CodeRuler implementation from my blog entry “Lessons Learned From CodeRuler”. The peer reviews associated with this analysis are provided by
• John Ly, CodeRuler Review• Creighton Okada, CodeRuler Review
Analysis
John and Creighton succeeded in discovering six types of uniquely occurring errors, while the QATs located nine uniquely occurring error types. Of these errors only three errors were identified by both peers and the QATs—lines exceeding 100 characters, usage of a wildcard import statement, and use of tabulation versus spaces. These errors are attributed to violations against programming styling specifications.
The remaining peer errors relate to poor choice of variable naming and the capacity to clarify documentation of the implementation. The QATs identified four additional deviations from the prescribed programming styling specifications (Checkstyle) and two from good programming practices (PMD). The stylistic issues ranged from missing spacing around braces and operands to missing documentation blocks. The latter problems were due to loose coupling (usage of ListArray) [4 occurrences] and confusing ternary (using not equal as the leading condition statement) [9 occurrences]. There were no issues in the CodeRuler implementation recognized through “bug pattern” analysis (FindBugs).
Conclusion
QATs have the advantage over peers in this analysis in respect to identifying the types of coding errors they specialize in, as well as, identifying all occurrences of these errors. QATs are developed to expressly and comprehensively locate all deviations from standard programming practices (stylistic and pattern analysis). QATs have the capability to do this quickly, repetitively, and reliably. Whereas, leaving reviews of code to humans is likely to be slow and very unreliable, especially in subsequent reviews of code already reviewed which may cause the human reader to read over errors.
However, manual reviews by humans can accomplish goals that QATs are not capable of resolving. These goals include writing concise, clear and appropriate levels of documentation. Also, a manual process can ensure that the conceptual model is reflected appropriately and accurately within the implementation with proper naming of variables and member fields, that the implementation and the documentation are in sync with one another, and that the code strives to be readable and self documenting. All tasks that QATs are not capable of performing. Thus, defining the complementary relationship manual and automated quality assurance methodologies have.
“Manual and automated quality assurance methodologies are integral components
with each complementing the other in the creation of high quality, stable solutions.”
with each complementing the other in the creation of high quality, stable solutions.”
The longer I thought on the topic, the more I came to the realization that each has its strengths complementing the weaknesses the other methodology exhibits. This becomes evident in analyzing the overlaps and disparity between peer reviews and the results of running quality assurance tools (QATs) against the CodeRuler implementation from my blog entry “Lessons Learned From CodeRuler”. The peer reviews associated with this analysis are provided by
• John Ly, CodeRuler Review• Creighton Okada, CodeRuler Review
Analysis
John and Creighton succeeded in discovering six types of uniquely occurring errors, while the QATs located nine uniquely occurring error types. Of these errors only three errors were identified by both peers and the QATs—lines exceeding 100 characters, usage of a wildcard import statement, and use of tabulation versus spaces. These errors are attributed to violations against programming styling specifications.
The remaining peer errors relate to poor choice of variable naming and the capacity to clarify documentation of the implementation. The QATs identified four additional deviations from the prescribed programming styling specifications (Checkstyle) and two from good programming practices (PMD). The stylistic issues ranged from missing spacing around braces and operands to missing documentation blocks. The latter problems were due to loose coupling (usage of ListArray) [4 occurrences] and confusing ternary (using not equal as the leading condition statement) [9 occurrences]. There were no issues in the CodeRuler implementation recognized through “bug pattern” analysis (FindBugs).
Conclusion
QATs have the advantage over peers in this analysis in respect to identifying the types of coding errors they specialize in, as well as, identifying all occurrences of these errors. QATs are developed to expressly and comprehensively locate all deviations from standard programming practices (stylistic and pattern analysis). QATs have the capability to do this quickly, repetitively, and reliably. Whereas, leaving reviews of code to humans is likely to be slow and very unreliable, especially in subsequent reviews of code already reviewed which may cause the human reader to read over errors.
However, manual reviews by humans can accomplish goals that QATs are not capable of resolving. These goals include writing concise, clear and appropriate levels of documentation. Also, a manual process can ensure that the conceptual model is reflected appropriately and accurately within the implementation with proper naming of variables and member fields, that the implementation and the documentation are in sync with one another, and that the code strives to be readable and self documenting. All tasks that QATs are not capable of performing. Thus, defining the complementary relationship manual and automated quality assurance methodologies have.
