SWIRL'12 Pre-workshop Discussion

All SWIRL participants were asked to review these past visionary reports about the IR field:

In addition, SWIRL participants were asked to nominate three papers that, in their opinion, represent important new directions, research areas, or results in the IR field. Each of the papers was annotated with a sentence describing the reason why the paper was chosen. Note that the papers are not intended to represent older, "classic" IR papers, but papers that represent important directions for the future of the IR field.

James Allan
  • Diversifying search results of controversial queries
    M. Kacimi and J. Gamper
    Proc. CIKM 2011, pp. 93-98
    Finding relevant information is clearly important, but there is (allegedly) a tendency for people (other than you, the reader, of course) to only look at information they find "comfortable": texts forwarded or recommended by friends or appearing in familiar and friendly venues. I believe that IR technology can and should also be used to remind people that there are other perspectives that they might want to consider -- both for topics they're already familiar with and for new topics they're considering. This paper is an example of one that attempts to do that by showing multiple perspectives for "controversial" topics.
  • Releasing search queries and clicks privately
    A. Korolova, K. Kenthapadi, N. Mishra, and A. Ntoulas
    Proc. WWW 2009, pp. 171-180
    Though extremely useful, query logs are a source of (1) concern to privacy advocates because of the personal and sensitive information associated with individuals, and (2) frustration for researchers who do not have access to them. Efforts to find ways to provide privacy guarantees -- particularly in ways that make the logs useful by more than a handful of large corporations -- strike me as extremely important for the future of research on interactive IR in particular.
  • Information seeking support systems
    G. Marchionini et al. (eds)
    Final report from NSF workshop on information seeking support systems, 2009
    There are numerous papers pointing to the importance of interactive IR. This report is a collection of papers discussing interactive, personalized, long-term search and exploration of information from a variety of perspectives.
Jay Aslam
  • Repeated Labeling Using Multiple Noisy Labelers
    Panagiotis G. Ipeirotis, Foster Provost, Victor S. Sheng, Jing Wang
    A working paper derived from "Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers", ACM SIGKDD 2008, pp. 614--622.
    Crowd sourcing is playing an increasingly important role for building training and test collections for IR, and there are many interesting papers on the subject (e.g., Kazai et al.). This is a paper from outside the mainstream IR community that discusses how to efficiently and effectively leverage Amazon Mechanical Turk (and similar systems) for crowd sourcing labels, with a focus on training collections but applicable for testing collections. Ipeirotis has built a system to evaluate MTurk results and workers based on these ideas. (http://qmturk.appspot.com/)
  • Adaptively Learning the Crowd Kernel
    Omer Tamuz, Ce Liu, Serge Belongie, Ohad Shamir, Adam Tauman Kalai
    ICML 2011, pp. 673--680
    Another paper on crowd sourcing, again from outside the IR community. Here the focus is on efficiently and effectively learning a crowd sourced "kernel", a human-induced similarity matrix over objects, with applications to search, classification, and clustering.
  • Search needs a shake-up
    Oren Etzioni
    Nature 476, pp. 25--26, August 2011
    Much of IR is focused on document retrieval, but we need to move beyond this to true information retrieval. Oren tackles this issue (in part), discussing the merits and deficits of traditional web search, as well systems such as Wolfram Alpha and IBM's Watson.
Leif Azzopardi
  • Some(what) Grand Challenges for Information Retrieval
    Nicholas J. Belkin
    SIGIR Forum 2008, 41(1):47-54
    I find this paper quite useful and inspiring. Belkin has does an excellent job at surmising the major directions yet to be adequately tackled in IIR. Within each of the 8 directions put forward there is plenty of further work and research that needs to be undertaken if the field is to make significant advancements over the next decade. The most important challenge raised by Belkin, in my opinion, is the development of (in)formal models for interactive IR, where such models will not only be descriptive but also predictive in nature.
  • User Adaptation: Good Results from Poor Systems
    Catherine L. Smith & Paul B. Kantor
    SIGIR 2008 Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pg147--154
    This paper throws up a very interesting finding for how users adapt to systems. One of the main implications is that different systems may require different usage and perhaps even different queries to obtain similar levels of performance. It also suggests that the effort required to undertake the search, and the time to adapt may be just as important as tradition precision/recall measures.
  • Improvements That Don't Add Up: Ad-Hoc Retrieval Results Since 1998
    Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel
    CIKM '09 Proceedings of the 18th ACM conference on Information and knowledge management, pg601-610
    This introspective paper examines dozens of studies performed in IR which reveals a dark hidden inner truth: that little improvements have been made and that most reports of significant improvements are mainly due to weak baselines. While we like to feel that the field has been advancing in terms of traditional performance measures, this paper forces us (as a field) to reconsider how we undertake our science.
Nick Belkin
Pia Borlund
  • Some(what) grand challenges for information retrieval.
    Belkin, N.J.
    ACM SIGIR Forum, 2008, 42(1), 47-54.
    This paper explicitly points out a number of issues we ought to address, not the least with reference to evaluation of IIR systems.
  • Understanding Casual-leisure Information Behaviour.
    Elsweiler, D., Wilson M.L., & Kirkegaard Lunn, B.
    In: Spink, A. and Heinstrom, J. (Eds.) New directions in information behaviour (pp. 211-241). Emerald Group Publishing Limited. 2011.
    This chapter is included as an example of how nowadays information retrieval can take place as causal-leisure IR, as a form of escapism and might not as such be directed by the fulfilling of an explicit information need. Further that chapter indirectly addresses the research area of Everyday life information seeking (ELIS).
  • On the Evaluation of Interactive Information Retrieval Systems
    Belkin, N.J.
    In: Larsen, B., Schneider, J.W., & Åström (Eds.). The Janus Faced Scholar: A Festschrift in Honour of Peter Ingwersen (pp. 13-21). Copenhagen: Royal School of Library and Information Science. 2010.
    This paper contributes with a new framework for IIR evaluation focusing on usefulness.
Peter Bruza
  • A markov random field model for term dependencies
    D. Metzler and W. B. Croft
    In SIGIR'05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval
    This, plus a subsequent paper on latent query expansion provided a sound formal foundation and new direction for modeling term dependencies
  • Discovering Key Concepts in Verbose Queries
    M. Bendersky, W. B. Croft
    In SIGIR'08 Proceedings of 31st annual international ACM SIGIR conference on Research and development in information retrieval
    This paper introduced interest into verbose queries Ð breaking open a new area as previously focus was usually on short queries
  • Beyond search: statistical topic models for text analysis
    C. Zhai
    Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
    Paints a promising and justifiable picture of going beyond search (as it is now) to information analysis by means of topic models
Jamie Callan
  • Quantifying test collection quality based on the consistency of relevance judgements
    Falk Scholer, Andrew Turpin, Mark Sanderson
    SIGIR 2011
    Many new test collections have been created during the last 20 years by TREC and its offspring, however the research community still lacks good tools for determining the quality of a test collection. This is especially problematic now that many are creating their own test collections using crowdsourcing services. This paper identifies and studies the effect of a source of assessor inconsistency that few have considered.
  • Query forwarding in geographically distributed search engines
    B. Barla Cambazoglu, Emre Varol, Enver Kayaaslan, Cevdet Aykanat, Ricardo Baeza-Yates
    SIGIR 2010
    Indexes are getting bigger all the time, and search services are becoming increasingly complex. However, few in the research community think seriously about indexes that are distributed geographically, the risk vs. reward of searching partitioned indexes, or the tradeoffs between sequential and parallel search of partitioned indexes. This paper proposes nice solutions, but I liked best the way it frames the problem.
  • Ad-hoc object retrieval in the web of data
    Jeffrey Pound, Peter Mika, Hugo Zaragoza
    WWW 2010: 771-780
    This paper studies adhoc object retrieval, which provides a nice framework for thinking about structured 'web of data' resources. It includes analysis of a commercial search engine log, providing information about the distribution of query types and domains for the whole log, the subset with semantic resource intent, and the subset with entity intent.
Mark Carman
  • #TwitterSearch: A Comparison of Microblog Search and Web Search
    Jaime Teevan, Daniel Ramage, and Meredith Ringel Morris
    Proceedings of WSDM 2011
    This paper investigates what differences there are (in terms of user motivation, etc.) are between searching the web and searching on Twitter.
  • Predicting consumer behavior with Web search
    Sharad Goel1, Jake M. Hofman1, SŽbastien Lahaie1, David M. Pennock1, and Duncan J. Watts1
    Proceedings of the National Academy of Sciences of the United States of America, PNAS 2010
    This paper investigates what future behaviour of consumers can and cannot be predicted from search volume in a query-log.
  • Hierarchical Pitman-Yor Language Model for Information Retrieval
    Saeedeh Momtazi and Dietrich Klakow
    In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval (SIGIR '10). ACM, New York, NY, USA, 793-794.
    This SIGIR poster demonstrates that "power-law discounting Language Models" (which are justified by the fact that they reproduce Zipfian distributions) outperform "standard" Dirichlet smoothing for unigram language model based retrieval.
Charles Clarke
Nick Craswell
  • Query Logs Alone are not Enough
    Carrie Grimes, Diane Tang and Daniel M. Russell
    Workshop on Query Log Analysis, WWW2007
    This is a Googly perspective on three different ways of observing user behavior, each with its own advantages. Insights into users (in person, via toolbar and via log analysis) can fuel new developments in IR test collections and models, across all IR applications.
  • Generating Query Substitutions
    Rosie Jones, Benjamin Rey, Omid Madani and Wiley Greiner
    WWW 2006
    Rather than pseudo-relevance feedback, industry papers such as this one do `query rewriting', which is related to machine translation. With a growing interest in query and intent modeling in IR, maybe we will apply such language-based techniques more often.
  • Personalizing Web Search using Long Term Browsing History
    Nicolaas Matthijs and Filip Radlinski
    WSDM 2011
    This paper does not use or generate a test collection. Its experiments study real users in situ, with a toolbar-based methodology that anyone could replicate, and the clicks+judgments indicate they beat the baseline (which was Google).
Bruce Croft
J. Shane Culpepper
  • Efficient algorithms for document retrieval problems.
    S. Muthukrishnan
    Proc. SODA, pp 657-666, Jan 2002.
    This paper presents a novel approach to indexing and document retrieval. The algorithms are derivatives of suffix arrays, and provide bounded theoretical performance guarantees. While the paper is largely theoretical, applied algorithmic researchers have recently been applying these ideas to build a new class of search engines with different trade-offs and capabilities than traditional inverted index-based systems.
  • Efficient document retrieval in main memory.
    T. Strohman and W. B. Croft
    Proc SIGIR, pp 175-182, 2007
    This paper investigates several standard information retrieval strategies entirely in main memory. Surprisingly little recent research on retrieval efficiency has accounted for fundamental hardware advances such as multicore processors, solid state drives, and cheap, ubiquitous DRAM.
  • Parameterized concept weighting in verbose queries.
    M. Bendersky, D. Metzler, and W. B. Croft
    Proc. SIGIR, pp 605-614, July 2011
    This paper produces state-of-the-art retrieval effectiveness by mixing supervised learning models (in this case, a CRF) with more traditional ranking algorithms. The paper opens the door to further work on improving retrieval effectiveness by using more than just simple collection statistics.
Maarten de Rijke
  • Large Scale Validation and Analysis of Interleaved Search Evaluation
    O. Chapelle, T. Joachims, F. Radlinski, Y. Yue
    ACM TOIS, to appear. The preprint is available.
    Interleaved comparison methods, which compare rankers using naturally occuring user interactions such as clicks, are interesting as a complement to traditional TREC-style evaluations for IR. Compared to evaluations based on manual relevance assessments, interleaved comparison methods rely only data that can be collected cheaply and unobtrusively. Since this data is based on the behavior of real users, it reflects how well their actual information needs are met. This paper is a solid overview of recent work in the area.
  • Efficient Multiple-Click Models in Web Search
    F. Guo, C. Liu, and Y.M. Wang
    In WSDM '09, pages 124-131, 2009.
    Simulation in IR. Our field has had a very strong emphasis on evaluating systems as opposed to evaluating algorithms or theories about relevance. It also, and increasingly, relies on data that is not widely available. Simulations, of queries, of users, of users' contexts are a natural instrument to help address both issues, an instrument that is being used in many disciplines and one that has seen some (but limited) use in IR since its early. The SimInt workshop (at SIGIR 2010) has a nice list of publications on simulation, going back at least three decades. The explicit model building underlying most simulation methods is an interesting way being more explicit about theories and it may also offer a way of learning from sensitive data without having to export the data from the silos where it sits. This paper is an interesting example of this type of model building.
  • Session based click features for recency ranking.
    Y. Inagaki, N. Sadagopan, G. Dupret, C. Liao, A. Dong, Y. Chang, and Z. Zheng
    In AAAI 2010
    Ensembles of rankers. To satisfy an information need we need to look at more than just topical relevance of results and should also take into account whether results are "fresh", "interesting", "authoritative", "available", … Feature-based models and learning to rank have received a lot of attention over the past couple of years. Evaluating and learning ranking functions becomes quite an interesting challenge when criteria such as those just listed are to be taken into account. We need better ways of discovering multiple aspects of user satisfaction, modeling those aspects, computing them, learning to adjust their relative importance online as we process incoming queries for which these aspects might be applicable. This is a paper that combines temporal aspects of clicks in an interesting way for recency ranking.
Arjen P. de Vries
  • From databases to dataspaces: a new abstraction for information management
    Michael Franklin, Alon Halevy, and David Maier
    SIGMOD Rec. 34, 4 (December 2005), 27-33
    A few gurus in the database community proposed the new notion of `dataspaces', where they aim to remove the schema requirement - schema information is only added in a pay-as-you-go fashion. Dataspace support systems cry for help from the IR community to realize effective access to the data under its control.
  • Personalizing web search results by reading level
    Kevyn Collins-Thompson, Paul N. Bennett, Ryen W. White, Sebastian de la Chica, and David Sontag
    Proceedings of the 20th ACM international conference on Information and knowledge management (CIKM '11)
    This paper considers readability in IR, demonstrating how it is a relevance criterion that matters
  • The Anatomy of a Multi-domain Search Infrastructure
    Stefano Ceri, Alessandro Bozzon and Marco Brambilla
    Web Engineering: 11th International Conference, ICWE 2011, Paphos, Cyprus, June 20-24, 2011
    The Search Computing project (see also the SIGMOD demo on http://www.youtube.com/watch?v=BCvxadh-MNQ) developed one of few search environments that support join queries over web sources
Fernando Diaz
  • Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms
    Li, L., Chu, W., Langford, J., and Wang, X.
    In Proceedings of the fourth ACM international conference on Web search and data mining (New York, NY, USA, 2011), WSDM'11, ACM, pp. 297--306.
    Evaluation of an algorithm in production environments usually requires an expensive live test on some fraction of search traffic. Unfortunately, the data from such a test is not reusable for evaluating a new algorithm, especially if its behavior is very different. This paper describes a method for reusing data from previously logged tests for new algorithms. This thread of research, known as "off-policy policy evaluation" in reinforcement learning, is an important tool in offline system development.
  • No clicks, no problem: using cursor movements to understand and improve search
    Huang, J., White, R.W., and Dumais, S.
    In Proceedings of the 2011 annual conference on Human factors in computing systems (New York, NY, USA, 2011), CHI'11, ACM, pp. 1225--1234.
    User feedback is a core area of information retrieval research. This paper suggests that we think about feedback beyond binary clicks toward for expressive mouse-tracking data.
  • Reducing the risk of query expansion via robust constrained optimization
    Collins-Thompson, K.
    In Proceedings of the Eighteenth International Conference on Information and Knowledge Management (CIKM 2009) (2009).
    This work, and prior work from NIPS 2008, introduces evaluation of retrieval functions using risk-based metrics found in portfolio theory. The general field of risk-based evaluation in information retrieval is relatively new but deserves further attention.
Susan Dumais
  • Entropy of Search Logs: How Hard is Search? With Personalization? With Backoff?
    Qiaozhu Mei and Ken Church
    WSDM 2009, pp. 45-54.
    An entropy-based approach to understanding the difficulty of search with and without context. Although there are many papers looking at how specific contextual information can be incorporated into retrieval (e.g., short- and long-term personalization, location, time, devices notably mobile), this paper presents a general approach based on the reduction in uncertainty given various contexts in order to understand the potential improvements that can be achieved contextual information.
  • Controlled Experiments on the Web: Survey and Practical Guide
    Ron Kohavi, Roger Longbotham, Dan Sommerfield and Randal M. Henne
    Data Mining and Knowledge Discovery, 2009, pp.140-181.
    An overview of controlled experimentation at Web-scale. Much of the research on large-scale log analysis has focused on characterizing behaviors with existing systems. This paper describes how to conduct carefully controlled experiments online, providing both general methodological recommendations as well as several case studies. An overriding theme is to let the data not the "hippo" lead the way.
  • Towards Recency Ranking and Web Search
    Anlei Dong, Yi Chang, Zhaohui Zheng, Gilad Mishne, Jing Bai, Ruiqang Zhang, Karolina Buchner, Ciya Liao, Fernando Diaz
    WSDM 2010, pp. 11-20.
    A recent paper describing an approach to incorporating important temporal features into ranking. Temporal dynamics are an important, but not well-studied, aspect of information retrieval. Change is everywhere in information systems (e.g., new documents appear, existing documents change, query frequency varies, what's relevant to a query changes), and it influences many aspects of retrieval systems (e.g., crawling, representation, ranking, extraction).
Nicola Ferro
  • Overview of the Reliable Information Access Workshop
    Donna Harman and Chris Buckley
    Information Retrieval, Volume 12, Number 6, pages 615-641, 2009
    The paper describes a collaborative attempt to carry out systematic failure analysis. This is an especially challenging and human consuming task which is too often overlooked even if it is crucial to get a deeper understanding of system behaviour. Therefore, the IR community should discuss how to adopt rigorous approaches and proper tools and infrastructures to carry it out.
  • Automated Component-Level Evaluation: Present and Future
    Allan Hanbury and Henning Müller
    in Maristella Agosti, Nicola Ferro, Carol Peters, Maarten de Rijke and Alan Smeaton (eds.) Multilingual and Multimodal Information Access Evaluation. Proceedings of the International Conference of the Cross-Language Evaluation Forum (CLEF 2010), Lecture Notes in Computer Science Volume 6360, pages 124-135, 2010
    The paper describes the current approaches to component-based evaluation and discusses its challenges. Component-based evaluation is is particularly difficult and challenging but would represent a next-step in the evaluation scenario since would allow us to move from systems evaluated as "black-box" to systems evaluated in terms of the contributions and interactions provided by the different components that constitute them.
  • Potential for Personalization
    Jaime Teevan, Susan T. Dumais, And Eric Horvitz
    ACM Transactions on Computer-Human Interaction, Volume 17, Number 1, Article 4, 2010
    This paper propose normalize Discounted Cumulative Gain (nDCG) as an analysis tool to discover the potential for further personalization of explicit and of implicit content-based and behavior features. Besides the specific purpose of the paper, i.e. personalization which represent a very important topics per se, rank analysis via DCG or other metrics should be further investigated. In particular, it would be interesting to develop visual analytics tools that allow us to get deeper insights about a ranked lists. This would provide a better undestanding of system behaviour and could be especially useful in failure analysis.
Shlomo Geva
  • Reflective Random Indexing and indirect inference: A scalable method for discovery of implicit connections
    Trevor Cohen, Roger Schvaneveldt, Dominic Widdows
    Journal of Biomedical Informatics 43 (2010) 240-256
    justification : The paper describes an interesting approach to the efficient random indexing of document collections. It does however ignore a few facts that deserve highlighting. This approach has promise that had so far not been realised. It will be interesting to discuss.
  • Journal peer review as an information retrieval process
    L. Bornmann ,L. Egghe
    This is a preprint of an article which is accepted for publication in the Journal of Documentation
    justification : The paper takes an interesting look at peer review of journal papers as an IR exercise. They draw analogies from Precision/Recall and trade-off in relation to the peer review process. There is an undeniable discontent with the peer review process. It is not new, but it always deserves discussing as the fast advancing technology facilitates different models.
  • What Do People Want from Information Retrieval? (The Top 10 Research Issues for Companies that Use and Sell IR Systems)
    W. Bruce Croft
    D-Lib Magazine, November 1995
    justification : It would be interesting to discuss how things panned out since then. Which issues are still open, which were easily resolved and what would the same paper contain if it was written at SWIRL'12.
Julio Gonzalo
  • Distributional memory: A general framework for corpus-based semantics
    Baroni, Marco and Lenci, Alessandro
    Computational Linguistics, Volume 36 Issue 4, December 2010, MIT Press Cambridge, MA, USA
    15 years ago, Natural Language Processing had a marginal role in Information Retrieval. Nowadays, the frontiers between both disciplines are less and less clear; and semantic processing for text mining is one of the key challenges in both areas. One of the grand challenges for research in semantic interpretation is how to merge distributional semantic representations, i.e. how to bring compositionality into distributional semantics. This is one of the papers in this area with a broadest scope.
  • Adding Semantics to Microblog Posts
    Meij, Edgar and Weerkamp, Wouter and de Rijke, Maarten
    Proceedings WSDM 2012
    This paper lies at the cross roads of two relevant challenges for Information Access: real time mining and retrieval in microblog posts, and use of linked data to enhance semantic interpretation and aggregation of textual sources.
  • Twitter mood predicts the stock marketpaper
    Johan Bollena, Huina Maoa, Xiaojun Zeng
    Journal of Computational Science, Volume 2, Issue 1, March 2011, Pages 1--8.
    Twitter has become the central nervous system of online information; in certain situations, Information Access technologies may help not only retrieving and mining information, but also predicting the future. This paper shows that Twitter data may be used to make more accurate stock market predictions, which is an interesting example of "mining the future", if only because it may help researchers get rich independently of their salary ;-)
David Hawking
  • On the Feasibility of Multi-Site Web Search Engines
    Ricardo Baeza-Yates and Aristides Gionis and Flavio Junqueira and Vassilis Plachouras and Luca Telloli
    CIKM 2009, pp. 425--434
    A very interesting contribution to web search engineering, exposing a fascinating distributed IR problem and providing insight into the economics and environmental impacts of large scale search.
  • Methods for Evaluating Interactive Information Retrieval Systems with Users
    Diane Kelly
    Foundations and Trends in Information Retrieval, 3(1-2) pp. 1-224, 2009
    The future of IR research and practice rely very heavily on faithful modeling of an increasingly diverse range of search scenarios -- consequently, understanding how to study users and to how to interpret user studies is an increasingly important skill.
  • Relevance Weighting for Query Independent Evidence
    Nick Craswell and Stephen Robertson and Hugo Zaragoza and Michael Taylor
    SIGIR 2005, pp. 416-423
    This paper provides useful insights into the perennial problem of how to transform raw features and how to combine evidence in ranking -- that problem becomes more important as the number of sources of evidence increases.
Kal Jarvelin
Gareth Jones
  • Evaluation Challenges and Direction for Information-Seeking Support Systems.
    D. Kelly, S. Dumais, and J. O. Pedersen.
    IEEE Computer, 42(3):60{66, 2009
    New approach to evaluation worthy of further consideration of how it might be applied to other information seeking activities.
  • Assessing the Scenic Route: Measuring the Value of Search Trails in Web Logs
    R. W. White and J. Huang
    Proceedings of the 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2010), pages: 587-594
    Very interesting exploration of an area of search taking the subject beyond ranked lists. This paper is also excellently structured and very clearly written, and one that I would recommend to students in terms of how to write a paper. However, a touch ironically, the paper also highlights the problem that students are increasingly not able to do work in important developing areas such as this since they do not have access to data of sufficent size and quality to carry research deemed to be meaningful or conclusive by the community.
  • The Probabilistic Relevance Framework: BM25 and Beyond
    S. E. Robertson and H. Zaragoza
    Foundations and Trends in Information Retrieval, Vol. 3, No. 4 (2009) 333–389
    The whole methodology underlying the development of BM25 and later extension to BM25F continues to impress. This is a summary article bringing together the development BM25 and beyond, so is not a new direction as such. However, the approach of analysing a problem in detail, careful theoretical consideration, development of practical outcomes, with careful exposition of strengths and shortcomings is itself a valuable contribution, There have been other recent efforts to taking an approach to semi-structured data search which I considered suggeted instead, but I think that this is richer source for discussion. I would like to see more of this detailed and careful thinking in addressing other areas of information retrieval.
Rosie Jones
  • Find It If You Can: A Game for Modeling Different Types of Web Search Success Using Interaction Data
    Mikhail Ageev, Qi Guo, Dmitry Lagun and Eugene Agichtein
    SIGIR 2011 pages 345--354
    Search session success is important to define and predict. This paper considers multiple definitions of search success and evaluates predictive performance of models and features for predicting search success trained and tested using crowd-sourcing and also tested on a large-scale search log.
  • Modeling and Analysis of Cross-Session Search Tasks
    Alexander Kotov, Paul N. Bennett, Ryen W. White, Susan T. Dumais and Jaime Teevan
    SIGIR 2011 pages 5--14
    Real user information needs can exist over long time periods, and multiple search sessions. This paper looks at search logs including information needs that span multiple search sessions over the course of a week. It includes models that identify queries from earlier sessions on the same task, and predict whether a user will continue a task in a later session.
  • Learning about the World through Long-Term Query Logs
    Matthew Richardson
    ACM Transactions on the Web, Vol. 2, No. 4, Article 21, Publication date: October 2008.
    This paper looks at a year's worth of query logs and shows how user information needs evolve over long periods of time in predictable ways, given a query earlier in their history. It also argues for query logs as a potential data source for data mining for other field, such as medicine and sociology.
Jaap Kamps
  • The effect of assessor error on IR system evaluation
    Ben Carterette, Ian Soboroff
    SIGIR 2010: 539-546
    A paper using a refreshingly new methodology of simulation and formal modeling to study assessor effects. Although simulation has been used in IR since the 70s, this paper is a great example of the power of simulation for analysis -- allowing to test what-if scenarios -- and nicely bridges theoretical aspects (models of systemic errors) and practical experiments.
  • A probability ranking principle for interactive information retrieval
    Norbert Fuhr
    Information Retrieval 11(3): 251-265 (2008)
    An inspiring paper that demonstrates the value of a theoretical approach to interactive IR -- an area of the field that is often treated as a blind spot -- and defines a probability ranking principle for interactive IR. This direction opens up many connections with related fields such as decision theory, or rather revives this connection that was prominent in the early years of IR.
  • Characterizing the influence of domain expertise on web search behavior
    Ryen W. White, Susan T. Dumais, Jaime Teevan
    WSDM 2009: 132-141
    Transaction logs have had a revolutionary impact on IR research, and this paper is of particular interest since it beautifully demonstrates how much more can be extracted from logs if particular domains/user groups are characterized: in this case focusing on domain expertise in medicine, finance, law, and computer science. Let me just say: it makes folks in academia without access to such data very envious...
Noriko Kando
  • Stuff I've seen: a system for personal information retrieval and re-use
    Susan Dumais, Edward Cutrell, JJ Cadiz, Gavin Jancke, Raman Sarin, and Daniel C. Robbins
    In Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2003). ACM, New York, NY, USA, pp.72-79, 2003
    Personalized search on heterogenous contents that each user has seen so far. Recording and storing all the materials that each user has seen provides various opportunities to understand the user's context, situation, tasks and (hidden) information needs, which are usable for better search and recommendation. The research on this direction is getting more and more important in the age of mobile and SNS.
  • Personalizing web search results by reading level
    Kevyn Collins-Thompson, Paul N. Bennett, Ryen W. White, Sebastian de la Chica, and David Sontag
    In Proceedings of the 20th ACM international conference on information and knowledge management (CIKM 2011), pp.403--412, 2011.
    Readability is a new to IR, but it is critical feature to provide better service and appropriate within-session support for various types of users with various reading ability including children, users with any sort of disability and/or unconfortableness.
  • Potential for personalization
    Jaime Teevan, Susan, T. Dumais, and Eric Horvitz
    ACM Transactions on Computer Human Interaction, Vol. 17, No.1, pp.
    A good overview how and in which area personalization shall be usable and expected to be effective.
Evangelos Kanoulas
  • Estimating Average Precision with Incomplete and Imperfect Judgments
    Emine Yilmaz and Javed A. Aslam
    In Proceedings of the 15th ACM international Conference on Information and Knowledge Management (CIKM 2006), 2006, p. 102--111
    This is one of the early works that proposed a theoretically founded solution to the problem of retrieval evaluation with incomplete relevance judgments. The problem of incomplete judgments belongs to a broader family of problems in IR evaluation: insufficient resources lead to variability in evaluation scores which affects the reliability of the evaluation conclusions. Complex retrieval tasks of the present and certainly of the future (advanced user models/diversity/sessions/etc.) may only exacerbate this problem.
  • Expected reciprocal rank for graded relevance
    Olivier Chapelle, Donald Metlzer, Ya Zhang and Pierre Grinspan
    In Proceedings of the 18th ACM international Conference on Information and Knowledge Management (CIKM 2009), 2009, p 621--630
    This work is an example of an attempts to model user behaviour when interacting with a retrieval system, and integrate this behaviour in an effectiveness measure under the traditional TREC evaluation framework. I consider this paper as part of a broader effort to bridge the gap between Cranfield and interactive evaluation (as much as this is possible). Similar efforts are illustrated in other recent measures of effectiveness (including measures of diversity/novelty/sessions/etc). Understanding and modelling the end user of retrieval systems will remain an interesting problem especially as the search interfaces change.
  • Evaluating Retrieval Performance Using Clickthrough Data
    Thorsten Joachims
    In Text Mining, 2003, p 79-96
    This is one of the early works in online evaluation. It illustrates an experimental setup that allows separating the signal from noise and bias and leads to new directions of utilising user behaviour over search results to evaluate retrieval systems. Given that IR moves towards complex tasks, personalisation and new environments and that resources will continue being scarce, online evaluation and user implicit feedback will continue playing an important role.
Jussi Karlgren
  • Hyperdimensional computing: An Introduction to Computing in Distributed Representation with High-Dimensional Random Vectors
    Pentti Kanerva
    Cognitive Computing 1(2):139-159 2009.
    This paper presents a family of representations, capabable of modelling language data in a more sophisticated manner than unigram frequencies or probabilistic language models, built to be computationally practical and sound in terms of engineering. Any future application which depends on large volumes of linguistic data as a base will be moving in this direction eventually. Key quote: "A major challenge for cognitive modeling is to identify mathematical systems of representation with operations that mirror cognitive phenomena of interest. This alone would satisfy the engineering objective of building computers with new capabilities. The mathematical systems should ultimately be realizable in neural substratum. Computing with hyperdimensional vectors is meant to take us in that direction."
  • Constructions: a new theoretical approach to language
    Adele Goldberg
    TRENDS in Cognitive Sciences Vol.7 No.5 May 2003
    This paper presents a theory, or rather an overview of a new family of theories, for modelling the obviously observable regularities of human language. Most such theories involve multi-level processing which all fail in face of non-standard non-edited input or innovative language use. Construction grammar (CxG) shows promise to be a theory (or, again, a family of theories) which actually can be used to account for the facts as we observe them. Most importantly, it does not - in some of its instantiations, at least - require the distinction between LEXICON and CONSTRUCTION. This enables us in the information access field to index constructions along with terms as just other another type of communicative object. No practically implemented model of language will be able to disregard this approach. Key quote: "Constructionist theories set out to account for all of our knowledge of language as patterns of form and function. ... The inventory of constructions, which includes morphemes or words, idioms, partially lexically filled and fully abstract phrasal patterns, is understood to be learned on the basis of the input together with general cognitive mechanisms."
  • A Unified Architecture for Natural Language Processing Deep Neural Networks with Multitask Learning
    Roman Collobert and Jason Weston
    Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland, 2008
    This paper shows how a data-oriented representation for language built to accommodate several "NLP" tasks achieves better results on the most challenging one, without leaning on crutches from the past few hundred years of linguistics. It is more important for its insightful introduction to the problem space in general than the technical detail in its proposed solution to the question it addresses in specifics. Any future application will need to have a base for representing language with this type of base. Key quote: "In particular, when training the SRL [semantic role labeling] task jointly with our language model our architecture achieved state-of-the-art performance in SRL without any explicit syntactic features. This is an important result, given that the NLP community considers syntax as a mandatory feature for semantic extraction."
Diane Kelly
  • Accurately Interpreting Clickthrough Data as Implicit Feedback
    T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay
    Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR), 2005
    This paper was one of the first to demonstrate that people are more likely to click on the result ranked in the first position regardless of its relevance. Continuing to document and understand people's search biases are important directions for future research and this paper nicely illustrates how do to that using alaboratory study.
  • Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips
    B. Sparrow, J. Liu, and D. M Wegner
    Science 5 August 2011: 333 (6043), 776-778
    This study shows that "when people expect to have future access to information, they have lower rates of recall of the information itself and enhanced recall instead for where to access it."
  • Is Google making us stupid?
    N. Carr
    Atlantic 302, 56 (2008)
    Although this is not a research article, I believe it is provocative and raises some issues that are relevant to the future of IR, including the long-term intellectual effects of quickly serving-up results (and answers) on demand.
Matt Lease
  • Kochhar, S. and Mazzocchi, S. and Paritosh, P.
    The anatomy of a large-scale human computation engine.
    KDD Workshop on Human Computation (HComp), 2010.
    Many papers have now been published which try to improve quality of crowdsourced labels collected via Mechanical Turk without having questioning basic assumptions of MTurk's labor model, and the impact of those assumptions. In contrast, this paper describes a very different and very successfully applied alternative labor model which likely better resembles how quality editorial judgments are collected by commercial search engines. As such, this is a "must read" in the crowdsourcing literature.
  • Yan, T. and Kumar, V. and Ganesan, D.
    CrowdSearch: exploiting crowds for accurate real-time image search on mobile phones.
    Proceedings of the 8th international conference on Mobile systems, applications, and services, pp. 77--90, 2010.
    This paper investigates design of a hybrid system combining automation with near real-time crowd labeling for improving search quality. While commercial search engines already improve quality via collective feedback from their many users, this paper gives a flavor of a new school of creative hybrid systems design, re-thinking old limitations of what is possible with purely automated approaches.
  • Horowitz, D. and Kamvar, S.D..
    The Anatomy of a Large-Scale Social Search Engine.
    KDD 2010.
    Sometimes the best information isn't written down, and a good mixed-initiative design exploits what man and machine are each best at to maximize what they can achieve together. The authors describe an impressive system in terms of both architecture and human factors considerations for addressing information needs not satisfied well by traditional search engines.
Jimmy Lin
  • Earlybird: Real-Time Search at Twitter
    Michael Busch, Krishna Gade, Brian Larson, Patrick Lok, Samuel Luckenbill, and Jimmy Lin
    ICDE 2012
    This paper provides an architectural overview of real-time search at Twitter. It discusses some of the challenges and solutions to low latency/high throughput query evaluation, while simultaneously ingesting thousands of tweets per second.
  • Large-scale Incremental Processing Using Distributed Transactions and Notifications
    Daniel Peng and Frank Dabek
    OSDI 2010
    An oversimplification of this paper would be "Bigtable meets database triggers". It describes Percolator, the architecture behind Google's (most recent publicly known) indexing and document processing pipeline. This, combined with the previous paper, provide nice examples of how IR really works "in the real world", in two very different parts of the design space.
  • MapReduce: Simplified Data Processing on Large Clusters
    Jeffrey Dean and Sanjay Ghemawat
    OSDI 2004
    IR is about managing large data. To manage large data we need the right tools. MapReduce (via Hadoop) should be one tool in every IR researcher's toolbox.
Donald Metzler
  • Search Needs a Shake-Up
    Oren Etzioni
    Nature, 476: 25-26, August 4, 2011
    This short, provocative commentary piece is a call to arms that encourages more creative, risky research at the intersection of information retrieval and natural language processing, with the ultimate goal of building search capabilities far beyond Google (and Watson).
  • Future Directions in Learning to Rank
    Olivier Chapelle, Yi Chang, and Tie-Yan Liu
    JMLR 14: 91-100, 2011
    Learning to rank algorithms have undoubtedly revolutionized our community's view of relevance ranking. However, the line of research has begun to stagnate. This paper discusses a number of potentially fruitful directions that could reinvigorate this line of research by tackling a number of fundamental information retrieval problems beyond relevance ranking.
  • Computers and iPhones and Mobile Phones, oh my! A logs-Based Comparison of Search Users on Different Devices
    Maryam Kamvar, Melanie Kellar, Rajan Patel, and Ya Xu
    WWW 2009: 801-810, 2009
    As users begin to consume more content on mobile devices, via mobile browsers and apps, there will be a fundamental change in how they search, organize, and share information. The information retrieval community has almost completely ignored this important paradigm shift. Studies like the one in this paper provide basic insights into mobile search habits and lay the groundwork necessary for more detailed research efforts into this important emerging direction.
Stefano Mizzaro
  • Location-aware click prediction in mobile local search
    Dimitrios Lymberopoulos, Peixiang Zhao, Arnd Christian König, Klaus Berberich, Jie Li
    CIKM 2011: 413-422.
    Although printing this paper was a bit... difficult, it is a good example of the research on context-aware and location-aware search on mobile devices, a topic that will probably be very important in the next years. Also, it is recent, it provides useful results, and it belongs to IR literature, which is not always the case for papers on this topic.
  • In Search of Quality in Crowdsourcing for Search Engine Evaluation
    Gabriella Kazai
    ECIR 2011: 165-176.
    Crowdsourcing (Amazon's Mechanical Turk and similar services) is being increasingly used by the IR community, mainly for evaluation (e.g., for gathering relevance assessments). I believe that crowdsourcing will be another hot topic in IR in the next years. This is one among many recent papers, some of them still in press, that discuss quality issues in crowdsourcing.
  • CrowdSearch: exploiting crowds for accurate real-time image search on mobile phones.
    Tingxin Yan, Vikas Kumar, Deepak Ganesan
    MobiSys 2010: 77-90
    This paper presents an idea that combines my other two hot topics, mobile devices and crowdsourcing, in a somehow non-standard fashion -- i.e., leaving evaluation aside. Although I'm convinced that evaluation will be crucial, this paper is an insightful example of how crowdsourcing can be exploited not only for evaluation, but for providing a real time search service based on the workforce of a crowd.
Vanessa Murdock
  • Twitter Under Crisis: Can We Trust what we RT
    Marcelo Mendoza, Barbara Poblete, Carlos Castillo
    Social Media Analytics Workshop (KDD 2010)
    A nice discussion on the credibility of information, and the way it propagates, using a case study of the tweets surrounding the earthquake in Chile. One of the first papers to look at the spread of misinformation in Twitter.
  • Display Advertising Impact: Search Lift and Social Influence
    Papadimitriou et al.
    KDD 2011
    Interesting study about the effect of viewing display ads on search behavior, and on the querying behavior of the ad-viewer's social network. The experimental method was particularly nice, and not typical in computer science.
  • Meme-tracking and the Dynamics of the News Cycle
    Leskovec, Backstrom, Kleinberg
    KDD 2009
    A lot of work recently uses streaming data (such as twitter and news feeds) to identify events, people, trending topics etc. This paper looks at this area in a more general sense, from an algorithmic perspective.
Doug Oard
  • Worker types and personality traits in crowdsourcing relevance labels.
    Gabriella Kazai, Jaap Kamps and Natasa Milic-Frayling
    In Proceedings of the 20th ACM Conference on Information and Knowledge Management (CIKM 2011), 4 pages, 2011.
    Crowdsourcing for evaluation. For a community as focused on evaluation as ours, we surely need an evaluation paper. I have been impressed with the depth of thought that is starting to appear on crowdsourcing as something more than just replicating current methods with a new way of finding assessors. I recently heard Jaap Kamps present the work from this paper, and it seemed to me to capture that sense of new opportunities.
  • The Context-Aware Browser.
    Paolo Coppola, Vincenzo Della Mea, Luca Di Gaspero, Davide Menegon, Danny Mischis, Stefano Mizzaro, Ivan Scagnetto and Luca Vassena.
    IEEE Intelligent Systems 25(1): 38-47 (2010).
    Mobile Context-Based IR. Apple's recent release of Siri helps us to see that IR for mobile devices may ultimately turn out to be as different from typical Web search as Web search was from searching library catalogs. This seems to be an area in which commercial practice is somewhat ahead of academia, but of course the whole Information Interaction in Context (IIiX) community is interested in aspects of this problem. Here's a paper from a recent special issue on Mobile IR (which also helps to illustrate that much of this work at present is appearing outside the traditional IR literature).
  • Natural Language Processing to the Rescue? Extracting "Situational Awareness" Tweets During Mass Emergency.
    Sudha Verma, Sarah Vieweg, William Corvey, Leysia Palen, James H. Martin, Martha Palmer, Aaron Schram and Kenneth M. Anderson.
    In Fifth International AAAI Conference on Weblogs and Social Media, 2011.
    IR from Social Media. One consistent trend in IR has been to extend our techniques to new content types as they become accessible (Web, CLIR, speech, blogs, ...). The focus on microblogging, and on Twitter in particular, is emblematic of the current instantiation of this trend, so it seems appropriate to include a Twitter paper. If the TREC Microblog track overview is made public in time (unlikely) it would be an excellent choice. As would the the forthcoming ICDE 2012 Twitter paper on their real-time search architecture (which is not yet on the Web, but see http://www.lucidimagination.com/sites/default/files/file/Eurocon2011/Busch_twitter_realtime_search_eurocon_11.pdf for related slides). Failing either of those, something like the following may be useful (appealing to the classification side of the IR community despite the use of the evil term "NLP" in the title).
Tetsuya Sakai
  • IR System Evaluation using Nugget-based Test Collections
    Virgil Pavlu, Shahzad Rajput, Peter B. Golbus and Javed A. Aslam
    WSDM 2012, to appear
    This paper and a few other recent papers (Clarke et al. SIGIR 2008, Sakai et al. CIKM 2011 etc.) highlight the importance of going beyond DOCUMENT relevance and DOCUMENT retrieval - come on, this is the 21st century, let's start providing nonredundant INFORMATION to the user.
  • Evaluating Search Systems Using Result Page Context
    Peter Bailey, Nick Craswell, Ryen W. White, Liwei Chen, Ashwin Satyanarayana, and S. M. M. Tahaghoghi
    IIiX 2010 pp.105-114
    This paper exemplifies the fact that IR is no longer just about ranking documents, though there remain many open questions, e.g., how can we design repeatable and search-engine-independent evaluation frameworks for rich presentation interfaces?
  • Good Abandonment in Mobile and PC Internet Search
    Jane Li, Scott B. Huffman, and Akihito Tokuda
    SIGIR 2009 pp.43-50
    IR should advance towards query-free, context-aware mobile search - this paper motivated me to design the Once Click Access task at NTCIR, hoping that we will soon realise Zero Click Access which enables systems to provide relevant information to the user even before she clicks on the search button.
Mark Sanderson
  • Expected reciprocal rank for graded relevance.
    Chapelle, O., Metlzer, D., Zhang, Y., & Grinspan, P.
    Proceeding of the 18th ACM conference on Information and knowledge management (pp. 621-630), 2009. ACM Press New York, NY, USA
    This paper is listed as an exemplar of a wider trend particularly prevalent in the search engine community. The test collections here used thousands of queries; in this paper >16K queries. With this many different requests from users, IR systems that specialize to particular subsets of queries can be identified and tested. There is great potential for improvement to search from this examination of query specialization.
  • Breadcrumbs of interaction
    Lindroth, T., & Bergquist, M.
    NordiCHI '08 Proceedings of the 5th Nordic conference on Human-computer interaction: building bridges. ACM Press. 2008.
    See comment for next paper.
  • New Version of Google Maps Brings Indoor Floor Plans to Your Phone
    Isaac, M.
    Wired (2011)
    These two papers address the topic of physical location and its importance in search and information access. The first shows that information access is influenced by your location, your path to that location and the people you are with. Exploiting such information for IR is still in its infancy. In part this is because, there are areas of the world where our location is hard to track, such as indoor spaces. However, as the 2nd 'paper' (an article) describes, means of tracking people in shopping malls, airports, or other large spaces is coming soon. All of the personalisation of information access we have grown accustomed to in the online world, will come to us soon in the physical world.
Falk Scholer
  • The influence of caption features on clickthrough patterns in web search
    Clarke, C.L.A. and Agichtein, E. and Dumais, S. and White, R.W.
    Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, 135--142, 2007.
    Summaries are used in nearly all IR systems, and are an essential component in enabling users to find useful information; while creation of summaries is often studied in its own right, there are few papers on the role of summaries in the broader search process. This paper investigates various features of query biased summaries and how these impact on the useability of summaries in web search systems.
  • Test collection-based ir evaluation needs extension toward sessions--a case of extremely short queries
    Keskustalo, H. and Järvelin, K. and Pirkola, A. and Sharma, T. and Lykke, M.
    Proceedings of the 5th Asia Information Retrieval Symposium, 63--74, 2009.
    Search evaluation has typically focused on one-shot queries, yet analysis of query logs suggests that users are impatient, and will often enter follow-up queries rather than wading through long lists of search results. This paper carries ouf a simulation using very short web-style queries, and demonstrates that entering multiple short queries can lead to reasonable results.
  • A similarity measure for indefinite rankings
    Webber, W. and Moffat, A. and Zobel, J.
    ACM Transactions on Information Systems (TOIS), 28(4), 2010.
    Much IR research focuses on experimentation, and makes use of a range of analytical and statistical techniques. However, sometimes the statistics and measures used don't tell us quite what we think that they're telling us. This paper discusses the limitations of a widely used correlation measure, Kendall's Tau, and demonstrates why it may not be ideal for some of the uses that it's put to in the IR setting. A new measure that addresses some of the limitations of Tau is also proposed.
Luo Si
  • A User-oriented Model for Expert Finding.
    E. Smirnova and K. Balog.
    (ECIR 33rd European Conference on Information Retrieval, 2011)
    Expert search is the research problem of identifying appropriate people who are knowledgeable on a given topic. Most current research of expert search focuses on analyzing expertise information in textual data (e.g., publications and technical). This piece of work studies an additional factor as the time to contact an expert by using the distance between the searcher and expert candidates in a social network. It demonstrates the benefit for doing so.
  • Building Watson: An overview of the Deep QA Project.
    D. Ferrucci, E. Brown, J, Chu-Carroll, J. Fan, D. Gondek, A. A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J. Prager, N. Schlaefer, and C. Welty.
    (AI Magazine 31(3), 2010).
    This paper describes the system research and development efforts in IBM DeepQA question answering system, which had an impressive performance in the TV quiz show, Jeopardy. It is nice to see how some existing techniques are effectively integrated together into a successful system and the potential of new research to improve the performance.
  • CrowdLogging: distributed, private, and anonymous search logging.
    H. A. Feild, J. Allan and J. Glatt.
    (SIGIR 2011.)
    This paper proposes an approach for distributed search log collection, storage and mining. It compares several privacy policies showing the trade-offs between privacy guarantees and utility. It also describes a pilot study of the proposed work.
James Thom
  • Content redundancy in YouTube and its application to video tagging
    Jose San Pedro, Stefan Siersdorfer, Mark Sanderson
    ACM Transactions on Information Systems (TOIS), Volume 29, Issue 3, Article 13, July 2011
    IR needs to deal not just with text but with multimedia (such as video), however combining multimedia retrieval techniques with text retrieval offers a promising way forward. Also since the ability to generate data now exceeds the capacity store data - identifying duplicated information is an important challenge for IR.
  • From information to knowledge: harvesting entities and relationships from web sources
    Gerhard Weikum, Martin Theobald
    PODS '10 Proceedings of the twenty-ninth ACM SIGMOD-SIGACT-SIGART symposium on principles of database systems, pp 65-76
    A tutorial on extracting information and knowledge from the web (rather than just finding web documents as in conventional IR search query). The paper clearly articulates various problems and challenges in this area.
  • Linked Data - The Story So Far
    Christian Bizer, Tom Heath, Tim Berners-Lee
    International Journal on Semantic Web and Information Systems, Volume 5, Issue 3, pp 1-22
    This paper provides a good overview of linked data, often also referred to as the "web of data" as distinct from the "web of documents". The paper identifies several research challenges, some of which a very pertinent to the IR community, including: user interfaces and interaction paradigms; and trust, quality and relevance.
Paul Thomas
Andrew Trotman
  • ENSM-SE and UJM at INEX 2010: Scoring with Proximity and Tag Weights
    M. Beigbeder, M. Géry, C. Largeron, H. Seck
    Proceedings of INEX 2010, pp. 44-53
    This INEX paper is the last in a series of papers by the authors that examines the Relevance in Context problem. The paper shows quite convincingly that a good method of using document structure, combined with a good method of weighting query terms based on proximity, combined with a good method of choosing whole documents results is both better document retrieval and better identification of relevant passages within a document. The paper is good motivation for continuing research into both document ranking (based on structure and proximity) as well as continuing research into ad hoc passage retrieval. For example, will the technique work on the web? Can it be used to identify good document snippets? What about as a post-ranking technique?
  • Posting list intersection on multicore architectures
    S. Tatikonda, B. B. Cambazoglu, F. P. Junqueira
    Proceedings of ACM SIGIR 2011, pp. 963--972
    In the case of desktop and mobile search it is search latency (the response time) that is important, not throughput. In this respect an enormous amount of work has been done on efficient uni-processor based processing of the postings lists. But, modern desktop computers and smart phones are multi core. The most efficient (lowest latency) method for processing postings lists on a multi-core architecture remains a nearly unstudied research area; however this paper is an exception.
  • Web indexing on a diet: template removal with the sandwich algorithm
    T. Rowlands, P. Thomas, S. Wan
    Proceedings of ADCS 2009, pp. 115-117
    This short paper outlines a very simple way of removing the decoration from a web page leaving only the contents of the page. The authors show that the algorithm reduces the size of the document collection and consequently the size of the inverted index. The effect on precision is not significant. This is a good paper because the algorithm is so simple but the effect is substantial. It shows us that the prior (somewhat complex) algorithms are unnecessary sophisticated, and yet it leaves plenty of room for improvement.
Andrew Turpin
  • Examining document model residuals to provide feedback during Information Retrieval evaluation
    Laurence Park
    The Sixteenth Australasian Document Computing Symposium (ADCS 2011) December, 2011
    Given a ranking function and a test collection, this paper uses the basic idea of optimising regression residuals to improve the ranking function. The idea is principled and elegant; but will it lead to better ranking functions?
  • Statistical query expansion for sentence retrieval and its effects on weak and strong queries
    David E. Losada
    Information Retrieval 13(5), p485-506
    Some interesting experiments applying query expansion to sentence selection for snippet/summary generation. Is sentence ranking an important ingredient to snippet generation?
  • Language independent ranked retrieval with NeWT
    J. S. Culpepper, M. Yasukawa, and F. Scholer
    Proceedings of the 16th Australasian Document Computing Symposium, December 2011
    The inverted file has dominated as the data structure of choice for the last 10 (at least) years. This is an attempt to move away from a fixed lexicon defined at index time to a general pattern matching framework. There are some challenges, to be sure, but efficient data structures for IR are far from complete. See also Navarro and Makinen, Compressed Self-Indexes ACM Computing Surveys 2006.
William Webber
Chengxiang Zhai
  • The Web changes everything: Understanding the dynamics of Web content.
    E. Adar, J. Teevan, S. Dumais and J. Elsas (2009).
    In Proceedings of WSDM 2009.
    This paper was selected for its contribution in developing models and algorithms for analyzing dynamics of Web content in detail. Dynamics of information is an under-addressed, yet very important issue broadly touching all areas of information retrieval.
  • Time Challenges - Challenging Times for Future Information Search.
    Thomas Mestl, Olga Cerrato, Jon Ølnes, Per Myrseth, Inger-Mette Gustavsen.
    D-Lib Magazine, May/June 2009, Vol. 15, No. 5/6.
    This paper was selected for an excellent and broad discussion of the issue of time in information retrieval and many implied challenges for information retrieval.
  • How to build a WebFountain: An architecture for very large-scale text analytics.
    D. Gruhl, L. Chavet, D. Gibson, J. Meyer, P. Pattanayak, A. Tomkins, and J. Zien. 2004.
    IBM Syst. J. 43, 1 (January 2004), 64-77. DOI=10.1147/sj.431.0064
    This paper was chosen for its discussion of a large-scale text analytics project. Text analytics naturally extends the current search technologies to support analysis tasks, which is an important direction that can broaden the scope of topics addressed so far in the IR community.
Xiuzhen (Jenny) Zhang
  • Opinion integration through semi-supervised topic modelling
    Yue Lu and Chengxiang Zhai
    In Proc. of WWW'08, pp. 121--130.
    Sentiment analysis has potential applications for IR. This paper proposed a semi-supervised topic modelling approach to sentiment analysis.
  • Connecting the dots between news articles.
    Dafna Shahaf and Carlos Guestrin
    In Proc. of KDD'10, pp. 623--632.
    The paper studies an interesting problem -- given two news articles, finding a coherent chain linking them together. A new search system architecture of information needs by related articles (rather than keywords) and structured output and interaction was proposed.
  • Optimizing web search using social annotations
    Shenghua Bao, Guirong Xue, Xiaoyuan Wu, Yong Yu, Ben Fei, and Zhong Su
    In Proc. of WWW'07. pp. 501--510.
    Social annotations play important roles in IR, and this is one of the first papers on this topic.
Yi Zhang
  • Towards a Theory Model for Product Search
    Beibei Li, Anindya Ghose, Panagiotis G. Ipeirotis
    WWW 2011
    This paper proposes a theory model for product search based on expected utility theory from economics. The top ranked products are the "best value for money" for a specific user. This is one of the papers that leverages micro-economic theory for information retrieval. Integrating economic theory into search and recommendation is a promising direction. This might enable us to better model human behavior and intention, interaction between human and a retrieval engine, and to design better retrieval systems so that human and the system can accomplish hard retrieval tasks together.
  • The Economics in Interactive Information Retrieval
    Leif Azzopardi
    ACM SIGIR 2011
    Similar to paper_1, this paper integrates economic theory with search. It treats search process as an economics problem and conducted simulations on TREC data. The analysis reveals that the total Cumulative Gain obtained during the course of a search session is functionally related to querying and assessing. This is an interesting paper about modeling user behavior based on economic theory, unfortunately they used simulated user. Can existing economic model sufficiently capture properties of real search users? How to evaluate?
  • Find it if you can: a game for modeling different types of web search success using interaction data
    Mikhail Ageev, Qi Guo, Dimitry Lagun, Eugene Agichtein
    SIGIR 2011
    How to perform controlled, realistic, scalable, and reproducible studies of searcher behavior for various IR problems is an important problem. This paper proposed an interesting technique for a special search task (answering 10 questions within allotted time). It would be interesting to study how to evaluate/study other search scenarios (personalized search, social web search, navigational search, etc). As the web is changing from the web of documents to the web of people (social web), it is important to research IR algorithms that can be adapted to this new paradigm. There many related papers in recent IR conferences. However, the work is still limited. This is an important topic for discussion, although none of my 3 papers is in this direction.

Webmaster: Falk Scholer
Page last updated: 10 May 2012