1. Reviews
Terms
Audit, IEEE 1028, informal
review, inspection, inspection leader, management review, moderator, review,
reviewer, technical review, walkthrough.
A successful review process requires
planning, participation and follow-up. Training providers will need to ensure
that test managers understand the responsibilities they have for the planning
and follow-up activities. Testers must be active participants in the review
process, providing their unique views. They should have formal review training to
better understand their respective roles in any technical review process. All
review participants must be committed to the benefits of a well-conducted
technical review. When done properly, inspections are the single biggest, and
most cost-effective, contributor to overall delivered quality. An international
standard on reviews is IEEE 1028.
1.2 The
Principles of Reviews
A review is a type of static testing.
Reviews frequently have as a major objective the detection of defects.
Reviewers find defects by directly examining documents.
The fundamental types of reviews are
described in section 3.2 of the Foundation Level Syllabus (version 2005) and
are listed below in chapter 6.3.
All types of review are best executed as
soon as the relevant source documents (documents which describe the project
requirements) and standards (to which the project must adhere) are available.
If one of the documents or standards is missing, then faults and
inconsistencies across all documentation cannot be discovered, only those
within one document can be discovered. Reviewers must be provided with the
document to be reviewed in adequate time to allow them to become familiar with
the contents of the document.
All types of documents can be subjected
to a review, e.g. source code, requirements specifications, concepts, test
plans, test documents, etc. Dynamic testing normally follows a source code
review; it is designed to find any defects that cannot be found by static
examination.
A review can lead to three possible
results:
·
The
document can be used unchanged or with minor changes
·
The
document must be changed but a further review is not necessary
·
The
document must be extensively changed and a further review is necessary
The roles and responsibilities of those
involved in a typical formal review are covered in the Foundation Syllabus,
i.e. manager, moderator or leader, author, reviewers and scribe. Others who may
be involved in reviews include decision makers or stakeholders, and customer or
user representatives. An additional optional role sometimes used in inspections
is that of a reader, who is intended to paraphrase sections of the work product
in the meeting. In addition to review roles, individual reviewers may each be
assigned a defect-based role to look for particular types of defect.
More than one of the review types may be
employed on a single product. For example, a team may hold a technical review
to decide which functionalities to implement in the next iteration. An
inspection might then be performed on the specifications for the included
functionalities.
The Foundation Syllabus introduced the
following types of review:
·
Informal
review
·
Walkthrough
·
Technical
review
·
Inspection
Hybrids of these types of reviews may
also occur in practice, such as a technical review using rule sets.
1.3.1 Management
review and audit
In addition to the types mentioned in the
Foundation Syllabus, IEEE 1028 also describes the following types of review:
·
Management
review
·
Audit
The key characteristics of a management
review are:
·
Main
purposes: to monitor progress, assess status, and make decisions about future
actions
·
Carried
out by or for managers having direct responsibility for the project or system
·
Carried
out by or for a stakeholder or decision maker, e.g. a higher level manager or
director
·
Checks
consistency with and deviations from plans, or adequacy of management
procedures
·
Includes
assessment of project risks
·
Outcome
includes action items and issues to be resolved
·
Participants
expected to prepare, decisions are documented
Note that test managers should
participate in and may instigate management reviews of testing progress.
Audits are extremely formal, and are
usually performed to demonstrate conformance to some set of expectations, most
likely an applicable standard or a contractual obligation. As such, audits are
the least effective at revealing defects.
The key characteristics of an audit are:
·
Main
purpose: provide independent evaluation of compliance to processes,
regulations, standards etc.
·
A
lead auditor is responsible for the audit and acts as the moderator
·
Auditors
collect evidence of compliance through interviews, witnessing and examining
documents
·
Outcome
includes observations, recommendations, corrective actions and a pass/fail
assessment
1.3.2 Reviews of
particular work products
Reviews may be described in terms of the
work products or activities that are subject to reviews, such as:
·
Contractual
review
·
Requirements
review
·
Design
review
o
preliminary
design review
o
critical
design review
·
Acceptance
review / qualification review
·
Operational
readiness review
A contractual review may be associated
with a contract milestone, and would typically be a management review for a
safety-critical or safety-related system. It would involve managers, customers
and technical staff.
A requirement review may be a
walkthrough, technical review or inspection, and may consider safety and
dependability requirements as well as functional and non-functional
requirements. A requirement review may include acceptance criteria and test
conditions.
Design reviews are typically technical reviews
or inspections, and involve technical staff and customers or stakeholders. The
Preliminary Design Review proposes the initial approach to some technical
designs and tests; the Critical Design Review covers all of the proposed design
solutions, including test cases and procedures.
Acceptance reviews are to obtain
management approval for a system. This is also referred to as a Qualification
Review, and is normally a management review or audit.
1.3.3 Performing
a formal review
The Foundation Syllabus describes six
phases of a formal review: planning, kick-off, individual preparation, review
meeting, rework and follow-up. The work product to be reviewed should be
appropriate for the qualification or the reviewer, e.g. a Test Plan for a Test
Manager, a business requirements or test design for a Test Analyst, or
functional specification, test cases or test scripts for Technical Test
Analyst.
In order for reviews to be successfully
introduced into an organization, the following steps should occur (not
necessarily in this order):
·
Securing
management support
·
Educating
managers about the costs, benefits and implementation issues
·
Selecting
and documenting review procedures, forms and infrastructure (e.g. reviews
metrics database)
·
Training
in review techniques and procedures
·
Obtaining
support from those who will be doing reviews and having their work reviewed
·
Executing
pilot reviews
·
Demonstrating
the benefit of reviews through cost savings
·
Applying
reviews to the most important documents, e.g. requirements, contracts, plans
etc.
Metrics such as reducing or avoiding cost
of fixing defects and/or their consequences may be used to evaluate the success
of the introduction of reviews. Savings may also be measured in elapsed time
saved by finding and fixing defects early.
Review processes should be continually
monitored and improved over time. Managers should be aware that learning a new
review technique is an investment – the benefits are not instant but will grow
significantly over time.
1.5 Success
Factors for Reviews
There are a number of factors that
contribute to successful reviews. Reviews need not be difficult to perform, but
they can go wrong in various ways if factors such as these are not considered.
Technical factors
·
Ensure
the defined process for the review type is followed correctly, particularly for
more formal types of review such as inspection
·
Record
the costs of reviews (including time spent) and benefits achieved
·
Review
early drafts or partial documents to identify patterns of defects before they
are built into the whole document
·
Ensure
that the documents or partial documents are review-ready before starting a
review process (i.e. apply entry criteria)
·
Use
organization-specific checklists of common defects
·
Use
more than one type of review, depending on objectives, such as document
cleanup, technical improvement, transfer of information, or progress management
·
Review
or inspect documents on which important decisions will be made, for example,
inspect a proposal, contract or high level requirement before a management
review authorizing major expenditure on the project
·
Sample
a limited subset of a document for assessment not clean-up
·
Encourage
finding the most important defects: focus on content not format
·
Continuously
improve the review process
Organizational factors
·
Ensure
managers allow adequate time to be spent on review activities, even under
deadline pressure
·
Remember,
time and budget spent are not in proportion to the errors found.
·
Allow
adequate time for rework of defects identified by reviews
·
Never
use the metrics from reviews for individual performance appraisal
·
Ensure
that the right people are involved in the different types of review
·
Provide
training in reviews, particularly the more formal review types
·
Support
a review leader forum to share experience and ideas
·
Ensure
everyone participates in reviews and everyone has their own documents reviewed
·
Apply
the strongest review techniques to the most important documents
·
Ensure
a well-balanced review team of people with different skills and backgrounds
·
Support
process improvement actions must be supported to address systemic problems
·
Recognize
improvements gained through the review process
People issues
·
Educate
stakeholders to expect that defects will be found and to allow time for rework
and rereview
·
Ensure
the review is a positive experience for the author
·
Welcome
the identification of defects in a blame-free atmosphere
·
Ensure
comments are constructive, helpful and objective, not subjective
·
Do
not review if the author does not agree or is not willing
·
Encourage
everyone to think deeply about the most important aspects of the documents
being reviewed