trec_eval
is a command line based tool used to evaluate search engines. This tool implements most of the standard retrieval evaluation measures like P@n, MAP, nDCG, etc.
In this post I’ll review how to install it, use it, and I’ll provide you a list of known (mis)behaviours that may affect your evaluation.
Installation
To install trec_eval
you need first to clone the official git repository:
git clone https://github.com/usnistgov/trec_eval.git
cd trec_eval
OS X (High Sierra)
In order to compile this code in OS X, you first need to install the XCode tools:
xcode-select --install
Now you can compile trec_eval
by executing:
make
You should now find the executable trec_eval
in the same folder. You can then move this file into your bin folder:
mv trec_eval /usr/local/bin/
Now executing trec_eval
should return this output:
Usage: trec_eval [-h] [-q] {-m measure}* trec_rel_file trec_top_file
-h: Give full help information, including other options
-q: In addition to summary evaluation, give evaluation for each query
-m: calculate and print measures indicated by 'measure'
('-m all_qrels' prints all qrels measures, '-m official' is default)
Ubuntu
You can compile trec_eval
by executing:
make
You should now find the executable trec_eval
in the same folder. You can then move this file into your bin folder:
mv trec_eval /usr/local/bin/
Now executing trec_eval
should return this output:
Usage: trec_eval [-h] [-q] {-m measure}* trec_rel_file trec_top_file
-h: Give full help information, including other options
-q: In addition to summary evaluation, give evaluation for each query
-m: calculate and print measures indicated by 'measure'
('-m all_qrels' prints all qrels measures, '-m official' is default)
Usage
TODO
(Mis)Behaviours
TODO