Metrics Used In Testing

In this blog page we will learn about metrics used in testing eg

The Product Quality Measures –

1. Customer satisfaction index,

2. Delivered defect quantities,

3. Responsiveness (turnaround time) to users,

4. Product volatility,

5. Defect ratios,

6. Defect removal efficiency,

7. Complexity of delivered product,

8. Test coverage,

9. Cost of defects,

10. Costs of quality activities,

11. Re-work,

12. Reliability and Metrics for Evaluating Application System Testing.

Test Efficiency Metrics

•Rejected Defects

% rejected defects = (Number of defects rejected/ Total number of defects logged) X 100

Test Effectiveness Metrics

•Defect Removal Efficiency (DRE)

% DRE (for a testing phase)

# valid defects detected in the current testing phase

(# valid defects detected in the current phase + # valid defects detected in the next testing phase)

% DRE (for all testing phases)

# valid defects detected pre-production

(# valid defects detected pre-production + # defects detected post-production

Product volatility

Ratio of maintenance fixes (to repair the system & bring it into compliance with specifications), vs. enhancement requests (requests by users to enhance or change functionality)

.Defect ratios

§                     Defects found after product delivery per function point.

§                     Defects found after product delivery per LOC

§                     Pre-delivery defects: annual post-delivery defects

§                     Defects per function point of the system modifications

Defect removal efficiency

§                     Number of post-release defects (found by clients in field operation), categorized by level of severity

§                     Ratio of defects found internally prior to release (via inspections and testing), as a percentage of all defects

§                     All defects include defects found internally plus externally (by customers) in the first year after product delivery

Complexity of delivered product

§                     McCabe's cyclomatic complexity counts across the system

§                     Halstead’s measure

§                     Card's design complexity measures

§                     Predicted defects and maintenance costs, based on complexity measures

Test coverage

§                     Breadth of functional coverage

§                     Percentage of paths, branches or conditions that were actually tested

§                     Percentage by criticality level: perceived level of risk of paths

§                     The ratio of the number of detected faults to the number of predicted faults.

Cost of defects

§                     Business losses per defect that occurs during operation

§                     Business interruption costs; costs of work-arounds

§                     Lost sales and lost goodwill

§                     Litigation costs resulting from defects

§                     Annual maintenance cost (per function point)

§                     Annual operating cost (per function point)

§                     Measurable damage to your boss's career

Costs of quality activities

§                     Costs of reviews, inspections and preventive measures

§                     Costs of test planning and preparation

§                     Costs of test execution, defect tracking, version and change control

§                     Costs of diagnostics, debugging and fixing

§                     Costs of tools and tool support

§                     Costs of test case library maintenance

§                     Costs of testing & QA education associated with the product

§                     Costs of monitoring and oversight by the QA organization (if separate from the development and test organizations)

 Re-work

§                     Re-work effort (hours, as a percentage of the original coding hours)

§                     Re-worked LOC (source lines of code, as a percentage of the total delivered LOC)

§                     Re-worked software components (as a percentage of the total delivered components)

 Reliability

§                     Availability (percentage of time a system is available, versus the time the system is needed to be available)

§                     Mean time between failure (MTBF).

§                     Man time to repair (MTTR)

§                     Reliability ratio (MTBF / MTTR)

§                     Number of product recalls or fix releases

§                     Number of production re-runs as a ratio of production runs

Metrics for Evaluating Application System Testing:

Metric = Formula

Test Coverage = Number of units (KLOC/FP) tested / total size of the system. (LOC represents Lines of Code)

Number of tests per unit size = Number of test cases per KLOC/FP (LOC represents Lines of Code).

Acceptance criteria tested = Acceptance criteria tested / total acceptance criteria

Defects per size = Defects detected / system size

Test cost (in %) = Cost of testing / total cost *100

Cost to locate defect = Cost of testing / the number of defects located

Achieving Budget = Actual cost of testing / Budgeted cost of testing

Defects detected in testing = Defects detected in testing / total system defects

Defects detected in production = Defects detected in production/system size

Quality of Testing = No of defects found during Testing/(No of defects found during testing + No of acceptance defects found after delivery) *100

Effectiveness of testing to business = Loss due to problems / total resources processed by the system.

System complaints = Number of third party complaints / number of transactions processed

Scale of Ten = Assessment of testing by giving rating in scale of 1 to 10

Source Code Analysis = Number of source code statements changed / total number of tests.

Effort Productivity = Test Planning Productivity = No of Test cases designed / Actual Effort for Design and Documentation

Test Execution Productivity = No of Test cycles executed / Actual Effort for testing

What is eCATT and Working with eCATT (Extended Computer Aided Test Tool)

  eCATT (Extended Computer Aided Test Tool) for SAP Automation Testing

Description:

In this article first I will introduce some of the basic concepts regarding eCATT and then we will see how to develop a test script to upload a test data file with a practical example.

So in precise in this article you will see what eCATT is, how we can use it; how we can load a test data from a file in 4.7X and it end with an example.

After reading this article you will have a clear idea about eCATT tool. And you will be able to write a test script in which test data can be loaded from a file.

This article is divided into following sections,

1. What is eCATT?
2. How to load test data from a file in 4.7X with an example?

 What is eCATT?

eCATT stands for extended Computer Aided Test Tool (eCATT) which is built is testing tool to test SAP system. By using testing tool we can test the entire business process, and we can also use this tool with a third party testing tool (I am not covering this topic). Execution of every test script ends with a log, which explains the results of the test script.

By using eCATT we can do following operations,

�        Test SAP transactions, reports, and scenarios

�        Call BAPIs and function modules

�        Test remote systems

�        Check authorizations (user profiles)

�        Test updates (database, applications, GUI)

�        Test the effect of changes to customizing settings

�        Check system messages

For more information go to: eCATT::Extended Computer Aided Test Tool (sdn.com)

Here I am discussing fundamentals about eCATT in detail. You can find very good documentation in sdn.com.

To develop a test script in eCATT we need to follow the following steps,

1. Creating Test Scripts.
2. Creating Test Data Containers.
3. Understanding System Data Containers .
4. Executing Test Configurations.

There is a very good web blog on eCATT in sdn.com which explains eCATT with necessary screen shots. To read document click here Blog on eCATT in sdn.com

Loading test data from a file in 4.7X with example:

We can load test data from file using the ABAP�ENDABAP statements in eCATT. In this article I have given an example on transaction code MM01 with sample code. To upload the test data file follow the steps given,

1. Record the transaction

Open eCATT tool. This can be done using �SECATT� tcode.

 

 Give the script name and version number example �ZTS_MM01�. Version number can be used to maintain different program constructs under same program name. And choose create  button.

Give description and component name as �BC-TWB-TST-ECA�.

Select editor tab panel. Then click on �pattern�    button. Or go to Edit->Pattern or Press Ctrl+F6. This opens a �insert statement� dialog box.

In that command dropdown box choose TDC (Record) option. Then press enter.

Enter the transaction name as �mm01� then MM01_1 interface automatically created. Then press enter.

Then �Create Material: Initial screen� will appear. Enter the necessary fields.

And be careful while recording, if not recording may fail. Note that while recording no error messages should pop up. If they happen restart the recording process. Here in this example I have considered a simple recording process. You can do any complex recording. If you have any doubts do feel free to mail me. I will reply to you.

After completing the recording process recording ended dialog will appear.

Choose �Yes�.

Then a TCD statement will appear in the editor area.

With this we have finished recording

Now let us see the variable declaration, assignment and programming part.

After developing as number of scripts I found one simple method to develop these test scripts. If you feel comfortable with this method you can also use it.

First note down the all screen fields in which you are entering values during recording process. Then create local variables in eCATT with the same name as the screen field technical name. (This method makes assignment easier).

 Example:

In MM01 (material master) I have entered values for material, industry sector and material type. And their respective technical screen field values are,

RMMG1-MATNR

RMMG1-MBRSH

RMMG1-MTART

To find out technical value of the screen field select the field press F1, then clicks on technical information button.  .key button

And now create the local variable as

V_MATNR,

V_MBRSH,

V_MTART,

 To create local variables first click  button, and then to create new variable click  button. And enter the variable name  as (V_MATNR), Visibility of parameter as �V� and finally parameter reference SAPTAO (name of the actual parameter). After declaring all the parameter it will look like this,

After declaring the local variables we need to assign them to screen field values.

 To do that again press  or double click on MM01_1 in TCD (MM01, MM01_1) statement. This will take you to the command interface which look like,

To assign the screen field values double click on the �FIELD�.

Then replace the screen VALIN values with the local variable names. After changing the values the interface look like this,

Repeat the above step with all the screen field values. With this we have finished the process of declaring and assigning variables. Now we will see how to program and run the script successfully. To write the program we need to follow two steps. They are, 

1. Get the number (count) of records in file.
2. Loop through count number of times and load the data from the file pass to TCD through local variables of eCATT.

Let us see how to handle the things,

By using ABAP�ENDABAP statements we can do that.

Before we start writing the program we need to declare some local variables (to run he example program given) which help us to store values between two ABAP blocks.

Because in eCATT each ABAP�ENDABAP block creates different internal function module with different contexts. So we need to declare some intermediate variables they are,

COUNT : Holds the number of records.

FILE_V : Holds file path of the test data file.

INT_LOC : Index to read next from the file.

I am giving sample code to get the number of records from file in eCATT. Use this code and try for MM01 for Basic view. It will work fine.
This is very simple ABAP code. For understanding purpose necessary comments are provided.

Step 1:

First ABAP�ENDABAP block, to get the number of records from the file and store the value in the COUNT local variable.

* ABAP BLOCK TO GET THE NUMBER OF TEST CASES

ABAP.

 * TOT : holds total number of records

* FILE: holds file path of test file

DATA : TOT TYPE I VALUE 0,

       FILE TYPE STRING.

* ITAB TO HOLD TEST DATA FROM FILE

DATA : BEGIN OF I_MARA OCCURS 0,

         V_MATNR LIKE RMMG1-MATNR, " Material Number

         V_MBRSH LIKE RMMG1-MBRSH, " Industry Sector

         V_MTART LIKE RMMG1-MTART, " Material Type

*        Basic View

         V_MAKTX LIKE MAKT-MAKTX, " Material Description

         V_MEINS LIKE MARA-MEINS, " Basic Unit of Measure

       END OF I_MARA.

* TO OPEN FILE DIALOG FOR TEST DATA FILE

CALL FUNCTION 'F4_FILENAME'

  EXPORTING

    PROGRAM_NAME  = SYST-CPROG

    DYNPRO_NUMBER = SYST-DYNNR

    FIELD_NAME    = 'FILE'

  IMPORTING

    FILE_NAME     = FILE_V-PATH.

FILE = FILE_V-PATH.

* LOADING DATA FROM THE FILE

CALL FUNCTION 'GUI_UPLOAD

  EXPORTING

    FILENAME            = FILE

    HAS_FIELD_SEPARATOR = 'X'

  TABLES

    DATA_TAB            = I_MARA.

* GETTING NUMBER OF RECORDS IN THE TABLE

DESCRIBE TABLE  I_MARA LINES TOT.

* STORING NUMBER OF RECORDS IN LOCAL VARIABLE

COUNT = TOT. 

* CLEARING INTERNAL TABLE

CLEAR I_MARA.

ENDABAP.

Step 2:

Looping through the records count number of times and reading from the internal table and passing them to the screen field values.

This sample code explains how to read, and pass values to the screen.

* LOOPING THROUGH (COUNT) NUMBER OF RECORDS

DO (COUNT ).

 ABAP.

 * V_READINDX : holds index number to read the internal table

* FILE: holds file path of test file

DATA : V_READINDX TYPE I,

       FILE TYPE STRING,

       INDX TYPE I VALUE 0.

* ITAB TO HOLD TEST DATA FROM FILE

DATA : BEGIN OF I_MARA OCCURS 0,

         V_MATNR LIKE RMMG1-MATNR, " Material Number

         V_MBRSH LIKE RMMG1-MBRSH, " Industry Sector

         V_MTART LIKE RMMG1-MTART, " Material Type

         V_MAKTX LIKE MAKT-MAKTX, " Material Description

         V_MEINS LIKE MARA-MEINS, " Basic Unit of Measure

       END OF I_MARA.

* WORKAREA TO HOLD THE I_MARA DATA

DATA : WA LIKE I_MARA.

FILE = FILE_V-PATH.

* LOADING MASTER DATA FROM THE FILE

CALL FUNCTION 'GUI_UPLOAD'

  EXPORTING

    FILENAME            = FILE

    HAS_FIELD_SEPARATOR = 'X'

  TABLES

    DATA_TAB            = I_MARA.

* INT_LOC : is a local variable hold the current index to read I_MARA

  V_READINDX = INT_LOC.

*  READING I_MARA UGING ITS INDEX

  READ TABLE I_MARA INDEX V_READINDX INTO WA.

* assigning work area values to the screen field values

         V_MATNR = WA-V_MATNR. " Material Number

         V_MBRSH = WA-V_MBRSH. " Industry Sector

         V_MTART = WA-V_MTART. " Material Type

         V_MAKTX = WA-V_MAKTX. " Material Description

         V_MEINS = WA-V_MEINS. " Basic Unit of Measure

ENDABAP.

* TCD Transaction /////////////////////

TCD ( MM01 , MM01_1 ).

* move index position by one

  INT_LOC = INT_LOC + 1.

ENDDO.

With this we have finished programming. Finally we need to prepare the test data file and execute the program either in Foreground or Background mode.

Please note that data in test file should resemble the order of the elements in the internal table. Other wise it won�t work.

 To execute the given test script, follow the steps, copy the code given and declare the necessary variables.

Disclaimer:

These programs are tested and running well on many, but not all SAP versions. The author is not responsible for any data loss or other kind of damage originating from the unintentional or intentional misuse of these programs. The person, who implements these programs, have to be sure that such damage does not happen.

My email id is saptaouk@gmail.com you can mail me at any time regarding this procedure. Please feel free to mail any comments and suggestion on this document.