Planning review process in test management

1. Reviews

Terms

Audit, IEEE 1028, informal review, inspection, inspection leader, management review, moderator, review, reviewer, technical review, walkthrough.

1.1 Introduction

A successful review process requires planning, participation and follow-up. Training providers will need to ensure that test managers understand the responsibilities they have for the planning and follow-up activities. Testers must be active participants in the review process, providing their unique views. They should have formal review training to better understand their respective roles in any technical review process. All review participants must be committed to the benefits of a well-conducted technical review. When done properly, inspections are the single biggest, and most cost-effective, contributor to overall delivered quality. An international standard on reviews is IEEE 1028.

1.2 The Principles of Reviews

A review is a type of static testing. Reviews frequently have as a major objective the detection of defects. Reviewers find defects by directly examining documents.

The fundamental types of reviews are described in section 3.2 of the Foundation Level Syllabus (version 2005) and are listed below in chapter 6.3.

All types of review are best executed as soon as the relevant source documents (documents which describe the project requirements) and standards (to which the project must adhere) are available. If one of the documents or standards is missing, then faults and inconsistencies across all documentation cannot be discovered, only those within one document can be discovered. Reviewers must be provided with the document to be reviewed in adequate time to allow them to become familiar with the contents of the document.

All types of documents can be subjected to a review, e.g. source code, requirements specifications, concepts, test plans, test documents, etc. Dynamic testing normally follows a source code review; it is designed to find any defects that cannot be found by static examination.

A review can lead to three possible results:

·                                 The document can be used unchanged or with minor changes

·                                 The document must be changed but a further review is not necessary

·                                 The document must be extensively changed and a further review is necessary

The roles and responsibilities of those involved in a typical formal review are covered in the Foundation Syllabus, i.e. manager, moderator or leader, author, reviewers and scribe. Others who may be involved in reviews include decision makers or stakeholders, and customer or user representatives. An additional optional role sometimes used in inspections is that of a reader, who is intended to paraphrase sections of the work product in the meeting. In addition to review roles, individual reviewers may each be assigned a defect-based role to look for particular types of defect.

More than one of the review types may be employed on a single product. For example, a team may hold a technical review to decide which functionalities to implement in the next iteration. An inspection might then be performed on the specifications for the included functionalities.

1.3 Types of Reviews

The Foundation Syllabus introduced the following types of review:

·                                 Informal review

·                                 Walkthrough

·                                 Technical review

·                                 Inspection

Hybrids of these types of reviews may also occur in practice, such as a technical review using rule sets.

1.3.1 Management review and audit

In addition to the types mentioned in the Foundation Syllabus, IEEE 1028 also describes the following types of review:

·                                 Management review

·                                 Audit

The key characteristics of a management review are:

·                                 Main purposes: to monitor progress, assess status, and make decisions about future actions

·                                 Carried out by or for managers having direct responsibility for the project or system

·                                 Carried out by or for a stakeholder or decision maker, e.g. a higher level manager or director

·                                 Checks consistency with and deviations from plans, or adequacy of management procedures

·                                 Includes assessment of project risks

·                                 Outcome includes action items and issues to be resolved

·                                 Participants expected to prepare, decisions are documented

Note that test managers should participate in and may instigate management reviews of testing progress.

Audits are extremely formal, and are usually performed to demonstrate conformance to some set of expectations, most likely an applicable standard or a contractual obligation. As such, audits are the least effective at revealing defects.

The key characteristics of an audit are:

·                                 Main purpose: provide independent evaluation of compliance to processes, regulations, standards etc.

·                                 A lead auditor is responsible for the audit and acts as the moderator

·                                 Auditors collect evidence of compliance through interviews, witnessing and examining documents

·                                 Outcome includes observations, recommendations, corrective actions and a pass/fail assessment

1.3.2 Reviews of particular work products

Reviews may be described in terms of the work products or activities that are subject to reviews, such as:

·                                 Contractual review

·                                 Requirements review

·                                 Design review

o                                                        preliminary design review

o                                                        critical design review

·                                 Acceptance review / qualification review

·                                 Operational readiness review

A contractual review may be associated with a contract milestone, and would typically be a management review for a safety-critical or safety-related system. It would involve managers, customers and technical staff.

A requirement review may be a walkthrough, technical review or inspection, and may consider safety and dependability requirements as well as functional and non-functional requirements. A requirement review may include acceptance criteria and test conditions.

Design reviews are typically technical reviews or inspections, and involve technical staff and customers or stakeholders. The Preliminary Design Review proposes the initial approach to some technical designs and tests; the Critical Design Review covers all of the proposed design solutions, including test cases and procedures.

Acceptance reviews are to obtain management approval for a system. This is also referred to as a Qualification Review, and is normally a management review or audit.

1.3.3 Performing a formal review

The Foundation Syllabus describes six phases of a formal review: planning, kick-off, individual preparation, review meeting, rework and follow-up. The work product to be reviewed should be appropriate for the qualification or the reviewer, e.g. a Test Plan for a Test Manager, a business requirements or test design for a Test Analyst, or functional specification, test cases or test scripts for Technical Test Analyst.

1.4 Introducing Reviews

In order for reviews to be successfully introduced into an organization, the following steps should occur (not necessarily in this order):

·                                 Securing management support

·                                 Educating managers about the costs, benefits and implementation issues

·                                 Selecting and documenting review procedures, forms and infrastructure (e.g. reviews metrics database)

·                                 Training in review techniques and procedures

·                                 Obtaining support from those who will be doing reviews and having their work reviewed

·                                 Executing pilot reviews

·                                 Demonstrating the benefit of reviews through cost savings

·                                 Applying reviews to the most important documents, e.g. requirements, contracts, plans etc.

Metrics such as reducing or avoiding cost of fixing defects and/or their consequences may be used to evaluate the success of the introduction of reviews. Savings may also be measured in elapsed time saved by finding and fixing defects early.

Review processes should be continually monitored and improved over time. Managers should be aware that learning a new review technique is an investment – the benefits are not instant but will grow significantly over time.

1.5 Success Factors for Reviews

There are a number of factors that contribute to successful reviews. Reviews need not be difficult to perform, but they can go wrong in various ways if factors such as these are not considered.

Technical factors

·                                 Ensure the defined process for the review type is followed correctly, particularly for more formal types of review such as inspection

·                                 Record the costs of reviews (including time spent) and benefits achieved

·                                 Review early drafts or partial documents to identify patterns of defects before they are built into the whole document

·                                 Ensure that the documents or partial documents are review-ready before starting a review process (i.e. apply entry criteria)

·                                 Use organization-specific checklists of common defects

·                                 Use more than one type of review, depending on objectives, such as document cleanup, technical improvement, transfer of information, or progress management

·                                 Review or inspect documents on which important decisions will be made, for example, inspect a proposal, contract or high level requirement before a management review authorizing major expenditure on the project

·                                 Sample a limited subset of a document for assessment not clean-up

·                                 Encourage finding the most important defects: focus on content not format

·                                 Continuously improve the review process

Organizational factors

·                                 Ensure managers allow adequate time to be spent on review activities, even under deadline pressure

·                                 Remember, time and budget spent are not in proportion to the errors found.

·                                 Allow adequate time for rework of defects identified by reviews

·                                 Never use the metrics from reviews for individual performance appraisal

·                                 Ensure that the right people are involved in the different types of review

·                                 Provide training in reviews, particularly the more formal review types

·                                 Support a review leader forum to share experience and ideas

·                                 Ensure everyone participates in reviews and everyone has their own documents reviewed

·                                 Apply the strongest review techniques to the most important documents

·                                 Ensure a well-balanced review team of people with different skills and backgrounds

·                                 Support process improvement actions must be supported to address systemic problems

·                                 Recognize improvements gained through the review process

People issues

·                                 Educate stakeholders to expect that defects will be found and to allow time for rework and rereview

·                                 Ensure the review is a positive experience for the author

·                                 Welcome the identification of defects in a blame-free atmosphere

·                                 Ensure comments are constructive, helpful and objective, not subjective

·                                 Do not review if the author does not agree or is not willing

·                                 Encourage everyone to think deeply about the most important aspects of the documents being reviewed