What is a good set of metrics that can measure the test progress of your project?
This is a question that is asked quite frequently from test managers.
Metrics are a helpful vehicle to determine the success of a project. In this article, we are outlining some useful and measurable list of test metrics that is used frequently in testing environments.
For your own projects, you can pick and chose from this list, and may be come up with your own that suits your particular organization.
Metrics Collection
How do we identify and document the metrics to be used from potentially long list?
It is important to have the goal of your project and management perspective in mind when you sort through in search of good set of metrics.
Ideally, they should be easily collected and understood while measuring the important milestones of your project. As such, you have control over the metrics and it may be the case as you go along you would want to change or modify the set of metrics.
While not mandatory, it may be helpful to get buy-in from stakeholders and get constant feedback as well.
If it is possible, automating the metrics collection is a time saving as well as a convenient endeavor.
Metrics are a helpful vehicle to determine the success of a project. In this article, we are outlining some useful and measurable list of test metrics that is used frequently in testing environments.
Metrics Reporting
What is the best way to display the collected metrics?
Collected metrics should display a snapshot of the project status at a certain time.
Using graphical representations to display metrics will help the reader understand the information immediately.
If it is feasible, you can make the charts interactive so the users will get more out of them.
Also consider a separate or a sub report which details the Test Progress – this is an item Project and Test Managers find very useful, and an item frequently asked or referred to in status meetings.
Metrics List
Some of the metrics that have been used frequently and that we have found to be of value are:
Test Progress:
- 1. Number of passed / failed / no run / in progress / not applicable tests
- 2. Requirements coverage: Number of requirements tested / Number of requirements passed / failed / not tested / out of testing scope
- 3. Number of defects found during testing
- 4. Planned vs Actual test execution time
- 5. Planned vs actual number of tests executed
Test Automation:
- 1. Effort (time in hours) to create automated tests
- 2. Effort (time in hours) to maintain automated tests
- 3. Execution time of automated tests
- 4. Number of automated tests
- 5. Code coverage
- 6. Number of passed / failed / no run automated tests
Defects:
- 1. Priority / Severity of defects found
- 2. Source of defects found (example: Requirements, Regression Tests, Environment, etc)
- 3. Number of defects fixed and rejected
- 4. Root causes of defects
- 5. Number of defect fixes that introduced new defects
Product Quality:
- 1. Number of defects found during testing
- 2. Number of defects found in production / after release
- 3. Severity (Critical, Major, Minor, Cosmetic) of defects found in production / after release
References:
1. http://www.istqb.org/downloads/send/10-advanced-level-syllabus-2012/54-advanced-level-syllabus-2012-test-manager.html
2. http://www.istqb.org/downloads/send/48-advanced-level-test-automation-engineer-documents/201-advanced-test-automation-engineer-syllabus-ga-2016.html
Author: Niluka Bamunuarachchige, CTAL, PMP, CSM, SSGB, ITIL, PCWA