Which performance parameters are best suited to assess the predictive ability of models?

K. Heberger, Anita Rácz, Dávid Bajusz

Research output: Chapter in Book/Report/Conference proceedingChapter

Abstract

We have revisited the vivid discussion in the QSAR-related literature concerning the use of external versus cross-validation, and have presented a thorough statistical comparison of model performance parameters with the recently published SRD (sum of (absolute) ranking differences) method and analysis of variance (ANOVA). Two case studies were investigated, one of which has exclusively used external performance merits. The SRD methodology coupled with ANOVA shows unambiguously for both case studies that the performance merits are significantly different, independently from data preprocessing. While external merits are generally less consistent (farther from the reference) than training and cross-validation based merits, a clear ordering and a grouping pattern of them could be acquired. The results presented here corroborate our earlier, recently published findings (SAR QSAR Environ. Res., 2015, 26, 683–700) that external validation is not necessarily a wise choice, and is frequently comparable to a random evaluation of the models.

Original languageEnglish
Title of host publicationChallenges and Advances in Computational Chemistry and Physics
PublisherSpringer
Pages89-104
Number of pages16
DOIs
Publication statusPublished - Jan 1 2017

Publication series

NameChallenges and Advances in Computational Chemistry and Physics
Volume24
ISSN (Print)2542-4491
ISSN (Electronic)2542-4483

Fingerprint

ranking
Analysis of variance (ANOVA)
analysis of variance
preprocessing
education
methodology
evaluation
Environ

Keywords

  • Cross-validation
  • External validation
  • Performance parameters (merits)
  • QSAR modeling
  • Ranking

ASJC Scopus subject areas

  • Computer Science Applications
  • Chemistry (miscellaneous)
  • Physics and Astronomy (miscellaneous)

Cite this

Heberger, K., Rácz, A., & Bajusz, D. (2017). Which performance parameters are best suited to assess the predictive ability of models? In Challenges and Advances in Computational Chemistry and Physics (pp. 89-104). (Challenges and Advances in Computational Chemistry and Physics; Vol. 24). Springer. https://doi.org/10.1007/978-3-319-56850-8_3

Which performance parameters are best suited to assess the predictive ability of models? / Heberger, K.; Rácz, Anita; Bajusz, Dávid.

Challenges and Advances in Computational Chemistry and Physics. Springer, 2017. p. 89-104 (Challenges and Advances in Computational Chemistry and Physics; Vol. 24).

Research output: Chapter in Book/Report/Conference proceedingChapter

Heberger, K, Rácz, A & Bajusz, D 2017, Which performance parameters are best suited to assess the predictive ability of models? in Challenges and Advances in Computational Chemistry and Physics. Challenges and Advances in Computational Chemistry and Physics, vol. 24, Springer, pp. 89-104. https://doi.org/10.1007/978-3-319-56850-8_3
Heberger K, Rácz A, Bajusz D. Which performance parameters are best suited to assess the predictive ability of models? In Challenges and Advances in Computational Chemistry and Physics. Springer. 2017. p. 89-104. (Challenges and Advances in Computational Chemistry and Physics). https://doi.org/10.1007/978-3-319-56850-8_3
Heberger, K. ; Rácz, Anita ; Bajusz, Dávid. / Which performance parameters are best suited to assess the predictive ability of models?. Challenges and Advances in Computational Chemistry and Physics. Springer, 2017. pp. 89-104 (Challenges and Advances in Computational Chemistry and Physics).
@inbook{22f54450ad50466b8285f23334de3924,
title = "Which performance parameters are best suited to assess the predictive ability of models?",
abstract = "We have revisited the vivid discussion in the QSAR-related literature concerning the use of external versus cross-validation, and have presented a thorough statistical comparison of model performance parameters with the recently published SRD (sum of (absolute) ranking differences) method and analysis of variance (ANOVA). Two case studies were investigated, one of which has exclusively used external performance merits. The SRD methodology coupled with ANOVA shows unambiguously for both case studies that the performance merits are significantly different, independently from data preprocessing. While external merits are generally less consistent (farther from the reference) than training and cross-validation based merits, a clear ordering and a grouping pattern of them could be acquired. The results presented here corroborate our earlier, recently published findings (SAR QSAR Environ. Res., 2015, 26, 683–700) that external validation is not necessarily a wise choice, and is frequently comparable to a random evaluation of the models.",
keywords = "Cross-validation, External validation, Performance parameters (merits), QSAR modeling, Ranking",
author = "K. Heberger and Anita R{\'a}cz and D{\'a}vid Bajusz",
year = "2017",
month = "1",
day = "1",
doi = "10.1007/978-3-319-56850-8_3",
language = "English",
series = "Challenges and Advances in Computational Chemistry and Physics",
publisher = "Springer",
pages = "89--104",
booktitle = "Challenges and Advances in Computational Chemistry and Physics",

}

TY - CHAP

T1 - Which performance parameters are best suited to assess the predictive ability of models?

AU - Heberger, K.

AU - Rácz, Anita

AU - Bajusz, Dávid

PY - 2017/1/1

Y1 - 2017/1/1

N2 - We have revisited the vivid discussion in the QSAR-related literature concerning the use of external versus cross-validation, and have presented a thorough statistical comparison of model performance parameters with the recently published SRD (sum of (absolute) ranking differences) method and analysis of variance (ANOVA). Two case studies were investigated, one of which has exclusively used external performance merits. The SRD methodology coupled with ANOVA shows unambiguously for both case studies that the performance merits are significantly different, independently from data preprocessing. While external merits are generally less consistent (farther from the reference) than training and cross-validation based merits, a clear ordering and a grouping pattern of them could be acquired. The results presented here corroborate our earlier, recently published findings (SAR QSAR Environ. Res., 2015, 26, 683–700) that external validation is not necessarily a wise choice, and is frequently comparable to a random evaluation of the models.

AB - We have revisited the vivid discussion in the QSAR-related literature concerning the use of external versus cross-validation, and have presented a thorough statistical comparison of model performance parameters with the recently published SRD (sum of (absolute) ranking differences) method and analysis of variance (ANOVA). Two case studies were investigated, one of which has exclusively used external performance merits. The SRD methodology coupled with ANOVA shows unambiguously for both case studies that the performance merits are significantly different, independently from data preprocessing. While external merits are generally less consistent (farther from the reference) than training and cross-validation based merits, a clear ordering and a grouping pattern of them could be acquired. The results presented here corroborate our earlier, recently published findings (SAR QSAR Environ. Res., 2015, 26, 683–700) that external validation is not necessarily a wise choice, and is frequently comparable to a random evaluation of the models.

KW - Cross-validation

KW - External validation

KW - Performance parameters (merits)

KW - QSAR modeling

KW - Ranking

UR - http://www.scopus.com/inward/record.url?scp=85053574001&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85053574001&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-56850-8_3

DO - 10.1007/978-3-319-56850-8_3

M3 - Chapter

T3 - Challenges and Advances in Computational Chemistry and Physics

SP - 89

EP - 104

BT - Challenges and Advances in Computational Chemistry and Physics

PB - Springer

ER -