Comparison of validation variants by sum of ranking differences and ANOVA

K. Heberger, Klára Kollár-Hunek

Research output: Contribution to journalArticle

Abstract

The old debate is revived: Definite differences can be observed in suggestions of estimation for prediction performances of models and for validation variants according to the various scientific disciplines. However, the best and/or recommended practice for the same data set cannot be dependent on the field of usage. Fortunately, there is a method comparison algorithm, which can rank and group the validation variants; its combination with variance analysis will reveal whether the differences are significant or merely the play of random errors. Therefore, three case studies have been selected carefully to reveal similarities and differences in validation variants. The case studies illustrate the different significance of these variants well. In special circumstances, any of the influential factors for validation variants can exert significant influence on evaluation by sums of (absolute) ranking differences (SRDs): stratified (contiguous block) or repeated Monte Carlo resampling and how many times the data set is split (5-7-10). The optimal validation variant should be determined individually again and again. A random resampling with sevenfold cross-validations seems to be a good compromise to diminish the bias and variance alike. If the data structure is unknown, a randomization of rows is suggested before SRD analysis. On the other hand, the differences in classifiers, validation schemes, and models proved to be always significant, and even subtle differences can be detected reliably using SRD and analysis of variance (ANOVA).

Original languageEnglish
Article numbere3104
JournalJournal of Chemometrics
DOIs
Publication statusAccepted/In press - Jan 1 2019

Fingerprint

Analysis of variance
Analysis of variance (ANOVA)
Ranking
Random errors
Data structures
Classifiers
Resampling
Method Comparison
Alike
Random Error
Performance Prediction
Randomisation
Cross-validation
Data Structures
Classifier
Unknown
Dependent
Evaluation
Model

Keywords

  • cross-validation
  • method comparison
  • model validation
  • ranking
  • resampling

ASJC Scopus subject areas

  • Analytical Chemistry
  • Applied Mathematics

Cite this

Comparison of validation variants by sum of ranking differences and ANOVA. / Heberger, K.; Kollár-Hunek, Klára.

In: Journal of Chemometrics, 01.01.2019.

Research output: Contribution to journalArticle

@article{f7796dca414b4d20b84e360a6076434f,
title = "Comparison of validation variants by sum of ranking differences and ANOVA",
abstract = "The old debate is revived: Definite differences can be observed in suggestions of estimation for prediction performances of models and for validation variants according to the various scientific disciplines. However, the best and/or recommended practice for the same data set cannot be dependent on the field of usage. Fortunately, there is a method comparison algorithm, which can rank and group the validation variants; its combination with variance analysis will reveal whether the differences are significant or merely the play of random errors. Therefore, three case studies have been selected carefully to reveal similarities and differences in validation variants. The case studies illustrate the different significance of these variants well. In special circumstances, any of the influential factors for validation variants can exert significant influence on evaluation by sums of (absolute) ranking differences (SRDs): stratified (contiguous block) or repeated Monte Carlo resampling and how many times the data set is split (5-7-10). The optimal validation variant should be determined individually again and again. A random resampling with sevenfold cross-validations seems to be a good compromise to diminish the bias and variance alike. If the data structure is unknown, a randomization of rows is suggested before SRD analysis. On the other hand, the differences in classifiers, validation schemes, and models proved to be always significant, and even subtle differences can be detected reliably using SRD and analysis of variance (ANOVA).",
keywords = "cross-validation, method comparison, model validation, ranking, resampling",
author = "K. Heberger and Kl{\'a}ra Koll{\'a}r-Hunek",
year = "2019",
month = "1",
day = "1",
doi = "10.1002/cem.3104",
language = "English",
journal = "Journal of Chemometrics",
issn = "0886-9383",
publisher = "John Wiley and Sons Ltd",

}

TY - JOUR

T1 - Comparison of validation variants by sum of ranking differences and ANOVA

AU - Heberger, K.

AU - Kollár-Hunek, Klára

PY - 2019/1/1

Y1 - 2019/1/1

N2 - The old debate is revived: Definite differences can be observed in suggestions of estimation for prediction performances of models and for validation variants according to the various scientific disciplines. However, the best and/or recommended practice for the same data set cannot be dependent on the field of usage. Fortunately, there is a method comparison algorithm, which can rank and group the validation variants; its combination with variance analysis will reveal whether the differences are significant or merely the play of random errors. Therefore, three case studies have been selected carefully to reveal similarities and differences in validation variants. The case studies illustrate the different significance of these variants well. In special circumstances, any of the influential factors for validation variants can exert significant influence on evaluation by sums of (absolute) ranking differences (SRDs): stratified (contiguous block) or repeated Monte Carlo resampling and how many times the data set is split (5-7-10). The optimal validation variant should be determined individually again and again. A random resampling with sevenfold cross-validations seems to be a good compromise to diminish the bias and variance alike. If the data structure is unknown, a randomization of rows is suggested before SRD analysis. On the other hand, the differences in classifiers, validation schemes, and models proved to be always significant, and even subtle differences can be detected reliably using SRD and analysis of variance (ANOVA).

AB - The old debate is revived: Definite differences can be observed in suggestions of estimation for prediction performances of models and for validation variants according to the various scientific disciplines. However, the best and/or recommended practice for the same data set cannot be dependent on the field of usage. Fortunately, there is a method comparison algorithm, which can rank and group the validation variants; its combination with variance analysis will reveal whether the differences are significant or merely the play of random errors. Therefore, three case studies have been selected carefully to reveal similarities and differences in validation variants. The case studies illustrate the different significance of these variants well. In special circumstances, any of the influential factors for validation variants can exert significant influence on evaluation by sums of (absolute) ranking differences (SRDs): stratified (contiguous block) or repeated Monte Carlo resampling and how many times the data set is split (5-7-10). The optimal validation variant should be determined individually again and again. A random resampling with sevenfold cross-validations seems to be a good compromise to diminish the bias and variance alike. If the data structure is unknown, a randomization of rows is suggested before SRD analysis. On the other hand, the differences in classifiers, validation schemes, and models proved to be always significant, and even subtle differences can be detected reliably using SRD and analysis of variance (ANOVA).

KW - cross-validation

KW - method comparison

KW - model validation

KW - ranking

KW - resampling

UR - http://www.scopus.com/inward/record.url?scp=85060203012&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85060203012&partnerID=8YFLogxK

U2 - 10.1002/cem.3104

DO - 10.1002/cem.3104

M3 - Article

JO - Journal of Chemometrics

JF - Journal of Chemometrics

SN - 0886-9383

M1 - e3104

ER -