Reconciling Batch and User System Evaluations

Dr Andrew Turpin

School of Computer Science and IT, RMIT University

Date and time: 11.30am - 12.30pm, Friday 4th July, 2008

Venue: 10.08.04 (Building 10, Level 8, Room 4)

Abstract:

Several recent papers have shown that when two information retrieval systems are ranked as superior and inferior using a Cranfield-style batch experiment, real users often perform equally well with either system. This paper reports controlled user experiments that explore three possible reasons why such a lack of correspondence may exist: a mismatch in the demographic of judges in the batch experiment and users in the user experiments; a mismatch in relevance judging criteria between judges and users; and a mismatch in outcome measures employed in the two experiments. We demonstrate that when each of these factors is controlled carefully to avoid a mismatch between the batch and user experiments, then the batch results are reflected in the user experience. We conclude that the Cranfield batch methodology is suitable for information retrieval system evaluation, but that meaningful interpretation of the results from such experiments in real world user environments requires careful consideration of the measures employed, the demographics of the judges, and the relevance criteria used by judges.

About the speaker:

Dr Andrew Turpin is an ARC Queen Elizabeth II Senior Research Fellow in the School of Computer Science and Information Technology at RMIT University, Melbourne. He completed his PhD at The University of Melbourne on data compression in 1999, subsequently spent several years at Devers Eye Institute and Oregan Health and Sciences University in Portland Oregon, then four more years after that teaching computer science at Curtin University of Technology in Perth. After a short stint as Senior Lecturer at The University of Melbourne, he has spent the last 3.5 years at RMIT as part of The Search Engine Group. His research interests include computational problems in ophthalmology, information retrieval, and string algorithms.

---

Seminar Organisation

Seminars are free and open to the general public. No booking is necessary. If you are interested in giving a presentation in this seminar series, or to make suggestions for speakers, please contact Xiaodong Li, the seminar co-ordinator.