Web Adhoc Track
RIRES: Russian Information Retrieval Evaluation Seminar

 Call for participation 
 General principles 
 Test collections 
 Relevance tables 


Web Adhoc Track

Adhoc search in a Web collection


The purpose of this track is to evaluate adhoc search methods on a Web collection. The dataset that is used imitates Web documents and Web queries.

For this track the standard procedure is used.

Test Collection

The source dataset is the Narod.ru collection.
Documents from all the archives narod.* and narod_training.* must be searched.

Task Description for Participating Systems

Each participant is granted access to the Narod.ru document collection and a set of queries which is a selection of queries from the logs of two Russian search engines (Yandex and Rambler). The number of queries in the set is 24250 (including those which were used in the previous ROMIP cycles).

Expected result is an ordered list of document URLs. Maximum list size is 100 per query.

Evaluation Methodology

  • instructions for assessors:
    Assessors evaluate document relevance to the query basing on the extended description of the user information need.
  • evaluation method: pooling (pool depth is 50)
  • relevance scale:
    • yes / probably yes / perhaps yes / no / impossible to evaluate
    • yes / no / impossible to evaluate
  • official metrics:
    • precision
    • recall
    • TREC 11-point precision/recall graph

Data Formats