Effect of test completeness and redundancy measurement on post release failures - An industrial experience report

Tamás Gergely, Árpád Beszédes, Tibor Gyimóthy, Milán Imre Gyalai

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

In risk-based testing, compromises are often made to release a system in spite of knowing that it has outstanding defects. In an industrial setting, time and cost are often the "exit criteria" and - unfortunately - not the technical aspects like coverage or defect ratio. In such situations, the stakeholders accept that the remaining defects will be found after release, so sufficient resources are allocated to the "stabilization" phases following the release. It is hard for many organizations to see that such an approach is significantly costlier than trying to locate the defects earlier. We performed an empirical investigation of this for one of our industrial partners (a financial company). In this project, significant perfective maintenance was performed on the large information system. Based on changes made to the system, we carried out procedure level code coverage measurements with code level change impact analysis, and a similarity-based comparison of test cases in order to quantitatively check the completeness and redundancy of the tests performed. In addition, we logged and compared the number of defects found during testing and live operation. The data obtained were surprising for both the developers and the customer as well, leading to a major reorganization of their development, testing, and operation processes. After the reorganization, a significant improvement in these indicators for testing efficiency was observed.

Original languageEnglish
Title of host publicationProceedings - 2010 IEEE International Conference on Software Maintenance, ICSM 2010
DOIs
Publication statusPublished - Dec 20 2010
Event2010 IEEE International Conference on Software Maintenance, ICSM 2010 - Timisoara, Romania
Duration: Sep 12 2010Sep 18 2010

Publication series

NameIEEE International Conference on Software Maintenance, ICSM

Conference

Conference2010 IEEE International Conference on Software Maintenance, ICSM 2010
CountryRomania
CityTimisoara
Period9/12/109/18/10

Keywords

  • Code coverage
  • Impact analysis
  • Risk-based testing
  • Test efficiency
  • Test redundancy
  • Test similarity
  • White-box testing

ASJC Scopus subject areas

  • Software

Fingerprint Dive into the research topics of 'Effect of test completeness and redundancy measurement on post release failures - An industrial experience report'. Together they form a unique fingerprint.

  • Cite this

    Gergely, T., Beszédes, Á., Gyimóthy, T., & Gyalai, M. I. (2010). Effect of test completeness and redundancy measurement on post release failures - An industrial experience report. In Proceedings - 2010 IEEE International Conference on Software Maintenance, ICSM 2010 [5609692] (IEEE International Conference on Software Maintenance, ICSM). https://doi.org/10.1109/ICSM.2010.5609692