Cautionary note on calculating standardized effect size (SES) in randomization test

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

In community ecology, randomization tests with problem specific test statistics (e.g., nestedness, functional diversity, etc.) are often applied. Researchers in such studies may want not only to detect the significant departure from randomness, but also to measure the effect size (i.e., the magnitude of this departure). Measuring the effect size is necessary, for instance, when the roles of different assembly forces (e.g., environmental filtering, competition) are compared among sites. The standard method is to calculate standardized effect size (SES), i.e., to compute the departure from the mean of random communities divided by their standard deviations. Standardized effect size is a useful measure if the test statistic (e.g., nestedness index, phylogenetic or functional diversity) in the random communities follows a symmetric distribution. In this paper, I would like to call attention to the fact that SES may give us misleading information if the distribution is asymmetric (skewed). For symmetric distribution median and mean values are equal (i.e., SES = 0 indicates p = 0.5). However, this condition does not hold for skewed distributions. For symmetric distributions departure from the mean shows the extremity of the value, regardless of the sign of departure, while in asymmetric distributions the same deviation can be highly probable and extremely improbable, depending on its sign. To avoid these problems, I recommend checking symmetry of null-distribution before calculating the SES value. If the distribution is skewed, I recommend either log-transformation of the test statistic, or using probit-transformed p-value as effect size measure.

Original languageEnglish
Pages (from-to)77-83
Number of pages7
JournalCommunity Ecology
Volume19
Issue number1
DOIs
Publication statusPublished - Jun 1 2018

Fingerprint

nestedness
statistics
functional diversity
testing
community ecology
researchers
distribution
effect
test
phylogeny
symmetry
phylogenetics
methodology

Keywords

  • Functional diversity
  • Randomization test
  • Standardized effect size
  • Statistics

ASJC Scopus subject areas

  • Ecology, Evolution, Behavior and Systematics
  • Ecology

Cite this

Cautionary note on calculating standardized effect size (SES) in randomization test. / Botta-Dukát, Z.

In: Community Ecology, Vol. 19, No. 1, 01.06.2018, p. 77-83.

Research output: Contribution to journalArticle

@article{a4b48484f6d94b9e97d00dd5b6df252f,
title = "Cautionary note on calculating standardized effect size (SES) in randomization test",
abstract = "In community ecology, randomization tests with problem specific test statistics (e.g., nestedness, functional diversity, etc.) are often applied. Researchers in such studies may want not only to detect the significant departure from randomness, but also to measure the effect size (i.e., the magnitude of this departure). Measuring the effect size is necessary, for instance, when the roles of different assembly forces (e.g., environmental filtering, competition) are compared among sites. The standard method is to calculate standardized effect size (SES), i.e., to compute the departure from the mean of random communities divided by their standard deviations. Standardized effect size is a useful measure if the test statistic (e.g., nestedness index, phylogenetic or functional diversity) in the random communities follows a symmetric distribution. In this paper, I would like to call attention to the fact that SES may give us misleading information if the distribution is asymmetric (skewed). For symmetric distribution median and mean values are equal (i.e., SES = 0 indicates p = 0.5). However, this condition does not hold for skewed distributions. For symmetric distributions departure from the mean shows the extremity of the value, regardless of the sign of departure, while in asymmetric distributions the same deviation can be highly probable and extremely improbable, depending on its sign. To avoid these problems, I recommend checking symmetry of null-distribution before calculating the SES value. If the distribution is skewed, I recommend either log-transformation of the test statistic, or using probit-transformed p-value as effect size measure.",
keywords = "Functional diversity, Randomization test, Standardized effect size, Statistics",
author = "Z. Botta-Duk{\'a}t",
year = "2018",
month = "6",
day = "1",
doi = "10.1556/168.2018.19.1.8",
language = "English",
volume = "19",
pages = "77--83",
journal = "Community Ecology",
issn = "1585-8553",
publisher = "Akademiai Kiado",
number = "1",

}

TY - JOUR

T1 - Cautionary note on calculating standardized effect size (SES) in randomization test

AU - Botta-Dukát, Z.

PY - 2018/6/1

Y1 - 2018/6/1

N2 - In community ecology, randomization tests with problem specific test statistics (e.g., nestedness, functional diversity, etc.) are often applied. Researchers in such studies may want not only to detect the significant departure from randomness, but also to measure the effect size (i.e., the magnitude of this departure). Measuring the effect size is necessary, for instance, when the roles of different assembly forces (e.g., environmental filtering, competition) are compared among sites. The standard method is to calculate standardized effect size (SES), i.e., to compute the departure from the mean of random communities divided by their standard deviations. Standardized effect size is a useful measure if the test statistic (e.g., nestedness index, phylogenetic or functional diversity) in the random communities follows a symmetric distribution. In this paper, I would like to call attention to the fact that SES may give us misleading information if the distribution is asymmetric (skewed). For symmetric distribution median and mean values are equal (i.e., SES = 0 indicates p = 0.5). However, this condition does not hold for skewed distributions. For symmetric distributions departure from the mean shows the extremity of the value, regardless of the sign of departure, while in asymmetric distributions the same deviation can be highly probable and extremely improbable, depending on its sign. To avoid these problems, I recommend checking symmetry of null-distribution before calculating the SES value. If the distribution is skewed, I recommend either log-transformation of the test statistic, or using probit-transformed p-value as effect size measure.

AB - In community ecology, randomization tests with problem specific test statistics (e.g., nestedness, functional diversity, etc.) are often applied. Researchers in such studies may want not only to detect the significant departure from randomness, but also to measure the effect size (i.e., the magnitude of this departure). Measuring the effect size is necessary, for instance, when the roles of different assembly forces (e.g., environmental filtering, competition) are compared among sites. The standard method is to calculate standardized effect size (SES), i.e., to compute the departure from the mean of random communities divided by their standard deviations. Standardized effect size is a useful measure if the test statistic (e.g., nestedness index, phylogenetic or functional diversity) in the random communities follows a symmetric distribution. In this paper, I would like to call attention to the fact that SES may give us misleading information if the distribution is asymmetric (skewed). For symmetric distribution median and mean values are equal (i.e., SES = 0 indicates p = 0.5). However, this condition does not hold for skewed distributions. For symmetric distributions departure from the mean shows the extremity of the value, regardless of the sign of departure, while in asymmetric distributions the same deviation can be highly probable and extremely improbable, depending on its sign. To avoid these problems, I recommend checking symmetry of null-distribution before calculating the SES value. If the distribution is skewed, I recommend either log-transformation of the test statistic, or using probit-transformed p-value as effect size measure.

KW - Functional diversity

KW - Randomization test

KW - Standardized effect size

KW - Statistics

UR - http://www.scopus.com/inward/record.url?scp=85048927238&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85048927238&partnerID=8YFLogxK

U2 - 10.1556/168.2018.19.1.8

DO - 10.1556/168.2018.19.1.8

M3 - Article

VL - 19

SP - 77

EP - 83

JO - Community Ecology

JF - Community Ecology

SN - 1585-8553

IS - 1

ER -