Effect of test completeness and redundancy measurement on post release failures - An industrial experience report

Tamás Gergely, A. Beszédes, T. Gyimóthy, Milán Imre Gyalai

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

In risk-based testing, compromises are often made to release a system in spite of knowing that it has outstanding defects. In an industrial setting, time and cost are often the "exit criteria" and - unfortunately - not the technical aspects like coverage or defect ratio. In such situations, the stakeholders accept that the remaining defects will be found after release, so sufficient resources are allocated to the "stabilization" phases following the release. It is hard for many organizations to see that such an approach is significantly costlier than trying to locate the defects earlier. We performed an empirical investigation of this for one of our industrial partners (a financial company). In this project, significant perfective maintenance was performed on the large information system. Based on changes made to the system, we carried out procedure level code coverage measurements with code level change impact analysis, and a similarity-based comparison of test cases in order to quantitatively check the completeness and redundancy of the tests performed. In addition, we logged and compared the number of defects found during testing and live operation. The data obtained were surprising for both the developers and the customer as well, leading to a major reorganization of their development, testing, and operation processes. After the reorganization, a significant improvement in these indicators for testing efficiency was observed.

Original languageEnglish
Title of host publicationIEEE International Conference on Software Maintenance, ICSM
DOIs
Publication statusPublished - 2010
Event2010 IEEE International Conference on Software Maintenance, ICSM 2010 - Timisoara
Duration: Sep 12 2010Sep 18 2010

Other

Other2010 IEEE International Conference on Software Maintenance, ICSM 2010
CityTimisoara
Period9/12/109/18/10

Fingerprint

Redundancy
Defects
Testing
Information systems
Stabilization
Costs
Industry

Keywords

  • Code coverage
  • Impact analysis
  • Risk-based testing
  • Test efficiency
  • Test redundancy
  • Test similarity
  • White-box testing

ASJC Scopus subject areas

  • Software

Cite this

Effect of test completeness and redundancy measurement on post release failures - An industrial experience report. / Gergely, Tamás; Beszédes, A.; Gyimóthy, T.; Gyalai, Milán Imre.

IEEE International Conference on Software Maintenance, ICSM. 2010. 5609692.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Gergely, T, Beszédes, A, Gyimóthy, T & Gyalai, MI 2010, Effect of test completeness and redundancy measurement on post release failures - An industrial experience report. in IEEE International Conference on Software Maintenance, ICSM., 5609692, 2010 IEEE International Conference on Software Maintenance, ICSM 2010, Timisoara, 9/12/10. https://doi.org/10.1109/ICSM.2010.5609692
Gergely, Tamás ; Beszédes, A. ; Gyimóthy, T. ; Gyalai, Milán Imre. / Effect of test completeness and redundancy measurement on post release failures - An industrial experience report. IEEE International Conference on Software Maintenance, ICSM. 2010.
@inproceedings{c8dbce825f2345b1b0b00c4d2d7d3aba,
title = "Effect of test completeness and redundancy measurement on post release failures - An industrial experience report",
abstract = "In risk-based testing, compromises are often made to release a system in spite of knowing that it has outstanding defects. In an industrial setting, time and cost are often the {"}exit criteria{"} and - unfortunately - not the technical aspects like coverage or defect ratio. In such situations, the stakeholders accept that the remaining defects will be found after release, so sufficient resources are allocated to the {"}stabilization{"} phases following the release. It is hard for many organizations to see that such an approach is significantly costlier than trying to locate the defects earlier. We performed an empirical investigation of this for one of our industrial partners (a financial company). In this project, significant perfective maintenance was performed on the large information system. Based on changes made to the system, we carried out procedure level code coverage measurements with code level change impact analysis, and a similarity-based comparison of test cases in order to quantitatively check the completeness and redundancy of the tests performed. In addition, we logged and compared the number of defects found during testing and live operation. The data obtained were surprising for both the developers and the customer as well, leading to a major reorganization of their development, testing, and operation processes. After the reorganization, a significant improvement in these indicators for testing efficiency was observed.",
keywords = "Code coverage, Impact analysis, Risk-based testing, Test efficiency, Test redundancy, Test similarity, White-box testing",
author = "Tam{\'a}s Gergely and A. Besz{\'e}des and T. Gyim{\'o}thy and Gyalai, {Mil{\'a}n Imre}",
year = "2010",
doi = "10.1109/ICSM.2010.5609692",
language = "English",
isbn = "9781424486298",
booktitle = "IEEE International Conference on Software Maintenance, ICSM",

}

TY - GEN

T1 - Effect of test completeness and redundancy measurement on post release failures - An industrial experience report

AU - Gergely, Tamás

AU - Beszédes, A.

AU - Gyimóthy, T.

AU - Gyalai, Milán Imre

PY - 2010

Y1 - 2010

N2 - In risk-based testing, compromises are often made to release a system in spite of knowing that it has outstanding defects. In an industrial setting, time and cost are often the "exit criteria" and - unfortunately - not the technical aspects like coverage or defect ratio. In such situations, the stakeholders accept that the remaining defects will be found after release, so sufficient resources are allocated to the "stabilization" phases following the release. It is hard for many organizations to see that such an approach is significantly costlier than trying to locate the defects earlier. We performed an empirical investigation of this for one of our industrial partners (a financial company). In this project, significant perfective maintenance was performed on the large information system. Based on changes made to the system, we carried out procedure level code coverage measurements with code level change impact analysis, and a similarity-based comparison of test cases in order to quantitatively check the completeness and redundancy of the tests performed. In addition, we logged and compared the number of defects found during testing and live operation. The data obtained were surprising for both the developers and the customer as well, leading to a major reorganization of their development, testing, and operation processes. After the reorganization, a significant improvement in these indicators for testing efficiency was observed.

AB - In risk-based testing, compromises are often made to release a system in spite of knowing that it has outstanding defects. In an industrial setting, time and cost are often the "exit criteria" and - unfortunately - not the technical aspects like coverage or defect ratio. In such situations, the stakeholders accept that the remaining defects will be found after release, so sufficient resources are allocated to the "stabilization" phases following the release. It is hard for many organizations to see that such an approach is significantly costlier than trying to locate the defects earlier. We performed an empirical investigation of this for one of our industrial partners (a financial company). In this project, significant perfective maintenance was performed on the large information system. Based on changes made to the system, we carried out procedure level code coverage measurements with code level change impact analysis, and a similarity-based comparison of test cases in order to quantitatively check the completeness and redundancy of the tests performed. In addition, we logged and compared the number of defects found during testing and live operation. The data obtained were surprising for both the developers and the customer as well, leading to a major reorganization of their development, testing, and operation processes. After the reorganization, a significant improvement in these indicators for testing efficiency was observed.

KW - Code coverage

KW - Impact analysis

KW - Risk-based testing

KW - Test efficiency

KW - Test redundancy

KW - Test similarity

KW - White-box testing

UR - http://www.scopus.com/inward/record.url?scp=78650078716&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=78650078716&partnerID=8YFLogxK

U2 - 10.1109/ICSM.2010.5609692

DO - 10.1109/ICSM.2010.5609692

M3 - Conference contribution

AN - SCOPUS:78650078716

SN - 9781424486298

BT - IEEE International Conference on Software Maintenance, ICSM

ER -