trec_eval is a command line based tool used to evaluate search engines. This tool implements most of the standard retrieval evaluation measures like P@n, MAP, nDCG, etc.

In this post I’ll review how to install it, use it, and I’ll provide you a list of known (mis)behaviours that may affect your evaluation.

Installation

To install trec_eval you need first to clone the official git repository:

git clone https://github.com/usnistgov/trec_eval.git
cd trec_eval

OS X (High Sierra)

In order to compile this code in OS X, you first need to install the XCode tools:

xcode-select --install

Now you can compile trec_eval by executing:

make

You should now find the executable trec_eval in the same folder. You can then move this file into your bin folder:

mv trec_eval /usr/local/bin/

Now executing trec_eval should return this output:

Usage: trec_eval [-h] [-q] {-m measure}* trec_rel_file trec_top_file

   -h: Give full help information, including other options

   -q: In addition to summary evaluation, give evaluation for each query

   -m: calculate and print measures indicated by 'measure'

       ('-m all_qrels' prints all qrels measures, '-m official' is default)

Ubuntu

You can compile trec_eval by executing:

make

You should now find the executable trec_eval in the same folder. You can then move this file into your bin folder:

mv trec_eval /usr/local/bin/

Now executing trec_eval should return this output:

Usage: trec_eval [-h] [-q] {-m measure}* trec_rel_file trec_top_file

   -h: Give full help information, including other options

   -q: In addition to summary evaluation, give evaluation for each query

   -m: calculate and print measures indicated by 'measure'

       ('-m all_qrels' prints all qrels measures, '-m official' is default)

Usage

TODO

(Mis)Behaviours

TODO