Publications

Lipani, Aldo

On Biases in Information Retrieval Models and Evaluation Journal Article

SIGIR Forum, 52 (2), pp. 172–173, 2019, ISSN: 0163-5840.

Lipani, Aldo

On Biases in Information Retrieval Models and Evaluation PhD Thesis

TU Wien, 2018.

@phdthesis{PhDLipani2018,
title = {On Biases in Information Retrieval Models and Evaluation},
author = {Aldo Lipani},
url = {http://aldolipani.com/wp-content/uploads/2018/09/phd_thesis.pdf},
doi = {10.13140/RG.2.2.28623.74400},
year = {2018},
date = {2018-09-21},
school = {TU Wien},
abstract = {The advent of the modern information technology has benefited society as the digitisation of content increased over the last half-century. While the processing capability of our species has remained unchanged, the information available to us has been notably increasing. In this overload of information, Information Retrieval (IR) has been playing a prominent role by developing systems capable of separating relevant information from the rest. This separation, however, is a difficult task rooted in the complexity of understanding of what is and what is not relevant. To manage this complexity, IR has developed a strong empirical nature, which has led to the development of grounded retrieval models, resulting in the development of retrieval systems empirically designed to be biased towards relevant information. However, other biases have been observed, which counteract retrieval performance. In this thesis, the reduction of retrieval systems to filters of information, or sampling processes, has allowed us to systematically investigate these biases.

We study biases manifesting in two aspects of IR research: retrieval models and retrieval evaluation. We start by identifying retrieval biases in probabilistic IR models and then develop new document priors to improve retrieval performance. Next, we discuss the accessibility bias of retrieval models, and for Boolean retrieval models we develop a mathematical framework of retrievability. For retrieval evaluation biases, we study how test collections are built using the pooling method and how this method introduces bias. Then, to improve the reliability of the evaluation, we first develop new pooling strategies to mitigate this bias at test collection build time and then, for two IR evaluation measures, Precision and Recall at cut-off (P@n and R@n), we develop new pool bias estimators to mitigate it at evaluation time.

Through a large scale experimentation involving up to 15 test collections, four IR evaluation measures and three bias measures, we demonstrate that including document priors based on verboseness improves the performance of probabilistic retrieval models; that the accessibility bias of Boolean retrieval models quickly worsens for conjunctive queries with the increase of the query length (while slightly improving for disjunctive queries); that the test collection bias can be lowered at test collection build time by pooling strategies inspired by a well-known problem in reinforcement learning, the multi-armed bandit problem; and that this bias can also be improved at evaluation time by analysing the runs participating in the pool. For this last point in particular, we show that for P@n, bias reduction is done by quantifying the potential of the new system against the pooled runs, and for R@n, this is done instead by simulating the absence of a pooled run from the set of pooled runs.

This thesis contributes to the IR field by giving a better understanding of relevance through the lens of biases in retrieval models and retrieval evaluation. The identification of these biases, and their exploitation or mitigation, leads to the development of better performing IR models and the improvement of the current IR evaluation practice.},
keywords = {},
pubstate = {published},
tppubtype = {phdthesis}
}

Close

The advent of the modern information technology has benefited society as the digitisation of content increased over the last half-century. While the processing capability of our species has remained unchanged, the information available to us has been notably increasing. In this overload of information, Information Retrieval (IR) has been playing a prominent role by developing systems capable of separating relevant information from the rest. This separation, however, is a difficult task rooted in the complexity of understanding of what is and what is not relevant. To manage this complexity, IR has developed a strong empirical nature, which has led to the development of grounded retrieval models, resulting in the development of retrieval systems empirically designed to be biased towards relevant information. However, other biases have been observed, which counteract retrieval performance. In this thesis, the reduction of retrieval systems to filters of information, or sampling processes, has allowed us to systematically investigate these biases.

We study biases manifesting in two aspects of IR research: retrieval models and retrieval evaluation. We start by identifying retrieval biases in probabilistic IR models and then develop new document priors to improve retrieval performance. Next, we discuss the accessibility bias of retrieval models, and for Boolean retrieval models we develop a mathematical framework of retrievability. For retrieval evaluation biases, we study how test collections are built using the pooling method and how this method introduces bias. Then, to improve the reliability of the evaluation, we first develop new pooling strategies to mitigate this bias at test collection build time and then, for two IR evaluation measures, Precision and Recall at cut-off (P@n and R@n), we develop new pool bias estimators to mitigate it at evaluation time.

Through a large scale experimentation involving up to 15 test collections, four IR evaluation measures and three bias measures, we demonstrate that including document priors based on verboseness improves the performance of probabilistic retrieval models; that the accessibility bias of Boolean retrieval models quickly worsens for conjunctive queries with the increase of the query length (while slightly improving for disjunctive queries); that the test collection bias can be lowered at test collection build time by pooling strategies inspired by a well-known problem in reinforcement learning, the multi-armed bandit problem; and that this bias can also be improved at evaluation time by analysing the runs participating in the pool. For this last point in particular, we show that for P@n, bias reduction is done by quantifying the potential of the new system against the pooled runs, and for R@n, this is done instead by simulating the absence of a pooled run from the set of pooled runs.

This thesis contributes to the IR field by giving a better understanding of relevance through the lens of biases in retrieval models and retrieval evaluation. The identification of these biases, and their exploitation or mitigation, leads to the development of better performing IR models and the improvement of the current IR evaluation practice.

Close

Lipani, Aldo; Roelleke, Thomas; Lupu, Mihai; Hanbury, Allan

A systematic approach to normalization in probabilistic models Journal Article

Information Retrieval Journal, 2018, ISSN: 1573-7659.

Abstract | Links | BibTeX

Lipani, Aldo; Palotti, Joao; Lupu, Mihai; Piroi, Florina; Zuccon, Guido; Hanbury, Allan

Fixed-Cost Pooling Strategies Based on IR Evaluation Measures Inproceedings

Proceedings of the 39th European Conference on IR Research, pp. 357–368, Springer International Publishing, Aberdeen, UK, 2017.

Links | BibTeX

Lipani, Aldo; Lupu, Mihai; Hanbury, Allan

Visual Pool: A Tool to Visualize and Interact with the Pooling Method Inproceedings

Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1321–1324, ACM, Shinjuku, Tokyo, Japan, 2017, ISBN: 978-1-4503-5022-8.

Links | BibTeX

Lipani, Aldo; Lupu, Mihai; Palotti, Joao; Zuccon, Guido; Hanbury, Allan

Fixed Budget Pooling Strategies Based on Fusion Methods Inproceedings

Proceedings of the 32nd ACM SIGAPP Symposium On Applied Computing, pp. 919–924, ACM, Marrakech, Morocco, 2017, ISBN: 978-1-4503-4486-9.

Abstract | Links | BibTeX

Schillaci, Calogero; Acutis, Marco; Lombardo, Luigi; Lipani, Aldo; Fantappiè, Maria; Märker, Michael; Saia, Sergio

Spatio-temporal topsoil organic carbon mapping of a semi-arid Mediterranean region: The role of land use, soil texture, topographic indices and the influence of remote sensing data to modelling Journal Article

Science of The Total Environment, 601-602 (Supplement C), pp. 821 - 832, 2017, ISSN: 0048-9697.

Links | BibTeX

Lipani, Aldo

Fairness in Information Retrieval Inproceedings

Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1171–1171, ACM, Pisa, Italy, 2016, ISBN: 978-1-4503-4069-4.

Links | BibTeX

Lipani, Aldo; Lupu, Mihai; Kanoulas, Evangelos; Hanbury, Allan

The Solitude of Relevant Documents in the Pool Inproceedings

Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 1989–1992, ACM, Indianapolis, Indiana, USA, 2016, ISBN: 978-1-4503-4073-1.

Links | BibTeX

Lipani, Aldo; Lupu, Mihai; Hanbury, Allan

The Curious Incidence of Bias Corrections in the Pool Inproceedings

Proceedings of the 38th European Conference on IR Research, pp. 267–279, Springer International Publishing, Padua, Italy, 2016.

Links | BibTeX

Lipani, Aldo; Zuccon, Guido; Lupu, Mihai; Koopman, Bevan; Hanbury, Allan

The Impact of Fixed-Cost Pooling Strategies on Test Collection Bias Inproceedings

Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval (ICTIR), pp. 105–108, ACM, Newark, Delaware, USA, 2016, ISBN: 978-1-4503-4497-5.

Links | BibTeX

Piroi, Florina; Lipani, Aldo; Lupu, Mihai; Hanbury, Allan

DASyR(IR) - document analysis system for systematic reviews (in Information Retrieval) Inproceedings

Proceedings of the 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 591-595, 2015.

Links | BibTeX

Lipani, Aldo; Lupu, Mihai; Hanbury, Allan; Aizawa, Akiko

Verboseness Fission for BM25 Document Length Normalization Inproceedings

Proceedings of the 2015 International Conference on The Theory of Information Retrieval (ICTIR), pp. 385–388, ACM, Northampton, Massachusetts, USA, 2015, ISBN: 978-1-4503-3833-2.

Links | BibTeX

Lipani, Aldo; Lupu, Mihai; Aizawa, Akiko; Hanbury, Allan

An Initial Analytical Exploration of Retrievability Inproceedings

Proceedings of the 2015 International Conference on The Theory of Information Retrieval, pp. 329–332, ACM, Northampton, Massachusetts, USA, 2015, ISBN: 978-1-4503-3833-2.

Links | BibTeX

Lipani, Aldo; Lupu, Mihai; Hanbury, Allan

Splitting Water: Precision and Anti-Precision to Reduce Pool Bias Inproceedings

Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 103–112, ACM, Santiago, Chile, 2015, ISBN: 978-1-4503-3621-5.

Links | BibTeX