Do batch and user evaluations give the same results?

William Hersh, Andrew Turpin, Susan Price, Benjamin Chan, D. Kraemer, Lynetta Sacherek, Daniel Olson

Research output: Contribution to journalArticle

70 Citations (Scopus)

Abstract

Do improvements in system performance demonstrated by batch evaluations confer the same benefit for real users? We carried out experiments designed to investigate this question. After identifying a weighting scheme that gave maximum improvement over the baseline in a non-interactive evaluation, we used it with real users searching on an instance recall task. Our results showed the weighting scheme giving beneficial results in batch studies did not do so with real users. Further analysis did identify other factors predictive of instance recall, including number of documents saved by the user, document recall, and number of documents seen by the user.

Original languageEnglish
Pages (from-to)17-24
Number of pages8
JournalSIGIR Forum (ACM Special Interest Group on Information Retrieval)
Publication statusPublished - 2000

Fingerprint

Experiments
Evaluation
Batch
Weighting
Factors
Experiment

ASJC Scopus subject areas

  • Hardware and Architecture
  • Management Information Systems

Cite this

Hersh, W., Turpin, A., Price, S., Chan, B., Kraemer, D., Sacherek, L., & Olson, D. (2000). Do batch and user evaluations give the same results? SIGIR Forum (ACM Special Interest Group on Information Retrieval), 17-24.

Do batch and user evaluations give the same results? / Hersh, William; Turpin, Andrew; Price, Susan; Chan, Benjamin; Kraemer, D.; Sacherek, Lynetta; Olson, Daniel.

In: SIGIR Forum (ACM Special Interest Group on Information Retrieval), 2000, p. 17-24.

Research output: Contribution to journalArticle

Hersh, William ; Turpin, Andrew ; Price, Susan ; Chan, Benjamin ; Kraemer, D. ; Sacherek, Lynetta ; Olson, Daniel. / Do batch and user evaluations give the same results?. In: SIGIR Forum (ACM Special Interest Group on Information Retrieval). 2000 ; pp. 17-24.
@article{bd7faa5bc7be47f7b0edbcf6d5cb6a36,
title = "Do batch and user evaluations give the same results?",
abstract = "Do improvements in system performance demonstrated by batch evaluations confer the same benefit for real users? We carried out experiments designed to investigate this question. After identifying a weighting scheme that gave maximum improvement over the baseline in a non-interactive evaluation, we used it with real users searching on an instance recall task. Our results showed the weighting scheme giving beneficial results in batch studies did not do so with real users. Further analysis did identify other factors predictive of instance recall, including number of documents saved by the user, document recall, and number of documents seen by the user.",
author = "William Hersh and Andrew Turpin and Susan Price and Benjamin Chan and D. Kraemer and Lynetta Sacherek and Daniel Olson",
year = "2000",
language = "English",
pages = "17--24",
journal = "SIGIR Forum (ACM Special Interest Group on Information Retrieval)",
issn = "0163-5840",
publisher = "Association for Computing Machinery (ACM)",

}

TY - JOUR

T1 - Do batch and user evaluations give the same results?

AU - Hersh, William

AU - Turpin, Andrew

AU - Price, Susan

AU - Chan, Benjamin

AU - Kraemer, D.

AU - Sacherek, Lynetta

AU - Olson, Daniel

PY - 2000

Y1 - 2000

N2 - Do improvements in system performance demonstrated by batch evaluations confer the same benefit for real users? We carried out experiments designed to investigate this question. After identifying a weighting scheme that gave maximum improvement over the baseline in a non-interactive evaluation, we used it with real users searching on an instance recall task. Our results showed the weighting scheme giving beneficial results in batch studies did not do so with real users. Further analysis did identify other factors predictive of instance recall, including number of documents saved by the user, document recall, and number of documents seen by the user.

AB - Do improvements in system performance demonstrated by batch evaluations confer the same benefit for real users? We carried out experiments designed to investigate this question. After identifying a weighting scheme that gave maximum improvement over the baseline in a non-interactive evaluation, we used it with real users searching on an instance recall task. Our results showed the weighting scheme giving beneficial results in batch studies did not do so with real users. Further analysis did identify other factors predictive of instance recall, including number of documents saved by the user, document recall, and number of documents seen by the user.

UR - http://www.scopus.com/inward/record.url?scp=0033661288&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0033661288&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:0033661288

SP - 17

EP - 24

JO - SIGIR Forum (ACM Special Interest Group on Information Retrieval)

JF - SIGIR Forum (ACM Special Interest Group on Information Retrieval)

SN - 0163-5840

ER -