Code coverage differences of Java bytecode and source code instrumentation tools

F. Horváth, Tamás Gergely, A. Beszédes, Dávid Tengeri, Gergő Balogh, T. Gyimóthy

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Many software testing fields, like white-box testing, test case generation, test prioritization, and fault localization, depend on code coverage measurement. If used as an overall completeness measure, the minor inaccuracies of coverage data reported by a tool do not matter that much; however, in certain situations, they can lead to serious confusion. For example, a code element that is falsely reported as covered can introduce false confidence in the test. This work investigates code coverage measurement issues for the Java programming language. For Java, the prevalent approach to code coverage measurement is using bytecode instrumentation due to its various benefits over source code instrumentation. As we have experienced, bytecode instrumentation-based code coverage tools produce different results than source code instrumentation-based ones in terms of the reported items as covered. We report on an empirical study to compare the code coverage results provided by tools using the different instrumentation types for Java coverage measurement on the method level. In particular, we want to find out how much a bytecode instrumentation approach is inaccurate compared to a source code instrumentation method. The differences are systematically investigated both in quantitative (how much the outputs differ) and in qualitative terms (what the causes for the differences are). In addition, the impact on test prioritization and test suite reduction—a possible application of coverage measurement—is investigated in more detail as well.

Original languageEnglish
Pages (from-to)1-45
Number of pages45
JournalSoftware Quality Journal
DOIs
Publication statusAccepted/In press - Dec 4 2017

Fingerprint

Java programming language
Software testing
Testing

Keywords

  • Code coverage
  • Coverage tools
  • Empirical study
  • Java bytecode instrumentation
  • Source code instrumentation
  • White-box testing

ASJC Scopus subject areas

  • Software
  • Safety, Risk, Reliability and Quality

Cite this

Code coverage differences of Java bytecode and source code instrumentation tools. / Horváth, F.; Gergely, Tamás; Beszédes, A.; Tengeri, Dávid; Balogh, Gergő; Gyimóthy, T.

In: Software Quality Journal, 04.12.2017, p. 1-45.

Research output: Contribution to journalArticle

@article{834c533875c54754a1ab1504e1827148,
title = "Code coverage differences of Java bytecode and source code instrumentation tools",
abstract = "Many software testing fields, like white-box testing, test case generation, test prioritization, and fault localization, depend on code coverage measurement. If used as an overall completeness measure, the minor inaccuracies of coverage data reported by a tool do not matter that much; however, in certain situations, they can lead to serious confusion. For example, a code element that is falsely reported as covered can introduce false confidence in the test. This work investigates code coverage measurement issues for the Java programming language. For Java, the prevalent approach to code coverage measurement is using bytecode instrumentation due to its various benefits over source code instrumentation. As we have experienced, bytecode instrumentation-based code coverage tools produce different results than source code instrumentation-based ones in terms of the reported items as covered. We report on an empirical study to compare the code coverage results provided by tools using the different instrumentation types for Java coverage measurement on the method level. In particular, we want to find out how much a bytecode instrumentation approach is inaccurate compared to a source code instrumentation method. The differences are systematically investigated both in quantitative (how much the outputs differ) and in qualitative terms (what the causes for the differences are). In addition, the impact on test prioritization and test suite reduction—a possible application of coverage measurement—is investigated in more detail as well.",
keywords = "Code coverage, Coverage tools, Empirical study, Java bytecode instrumentation, Source code instrumentation, White-box testing",
author = "F. Horv{\'a}th and Tam{\'a}s Gergely and A. Besz{\'e}des and D{\'a}vid Tengeri and Gergő Balogh and T. Gyim{\'o}thy",
year = "2017",
month = "12",
day = "4",
doi = "10.1007/s11219-017-9389-z",
language = "English",
pages = "1--45",
journal = "Software Quality Journal",
issn = "0963-9314",
publisher = "Springer New York",

}

TY - JOUR

T1 - Code coverage differences of Java bytecode and source code instrumentation tools

AU - Horváth, F.

AU - Gergely, Tamás

AU - Beszédes, A.

AU - Tengeri, Dávid

AU - Balogh, Gergő

AU - Gyimóthy, T.

PY - 2017/12/4

Y1 - 2017/12/4

N2 - Many software testing fields, like white-box testing, test case generation, test prioritization, and fault localization, depend on code coverage measurement. If used as an overall completeness measure, the minor inaccuracies of coverage data reported by a tool do not matter that much; however, in certain situations, they can lead to serious confusion. For example, a code element that is falsely reported as covered can introduce false confidence in the test. This work investigates code coverage measurement issues for the Java programming language. For Java, the prevalent approach to code coverage measurement is using bytecode instrumentation due to its various benefits over source code instrumentation. As we have experienced, bytecode instrumentation-based code coverage tools produce different results than source code instrumentation-based ones in terms of the reported items as covered. We report on an empirical study to compare the code coverage results provided by tools using the different instrumentation types for Java coverage measurement on the method level. In particular, we want to find out how much a bytecode instrumentation approach is inaccurate compared to a source code instrumentation method. The differences are systematically investigated both in quantitative (how much the outputs differ) and in qualitative terms (what the causes for the differences are). In addition, the impact on test prioritization and test suite reduction—a possible application of coverage measurement—is investigated in more detail as well.

AB - Many software testing fields, like white-box testing, test case generation, test prioritization, and fault localization, depend on code coverage measurement. If used as an overall completeness measure, the minor inaccuracies of coverage data reported by a tool do not matter that much; however, in certain situations, they can lead to serious confusion. For example, a code element that is falsely reported as covered can introduce false confidence in the test. This work investigates code coverage measurement issues for the Java programming language. For Java, the prevalent approach to code coverage measurement is using bytecode instrumentation due to its various benefits over source code instrumentation. As we have experienced, bytecode instrumentation-based code coverage tools produce different results than source code instrumentation-based ones in terms of the reported items as covered. We report on an empirical study to compare the code coverage results provided by tools using the different instrumentation types for Java coverage measurement on the method level. In particular, we want to find out how much a bytecode instrumentation approach is inaccurate compared to a source code instrumentation method. The differences are systematically investigated both in quantitative (how much the outputs differ) and in qualitative terms (what the causes for the differences are). In addition, the impact on test prioritization and test suite reduction—a possible application of coverage measurement—is investigated in more detail as well.

KW - Code coverage

KW - Coverage tools

KW - Empirical study

KW - Java bytecode instrumentation

KW - Source code instrumentation

KW - White-box testing

UR - http://www.scopus.com/inward/record.url?scp=85036539615&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85036539615&partnerID=8YFLogxK

U2 - 10.1007/s11219-017-9389-z

DO - 10.1007/s11219-017-9389-z

M3 - Article

AN - SCOPUS:85036539615

SP - 1

EP - 45

JO - Software Quality Journal

JF - Software Quality Journal

SN - 0963-9314

ER -