Mixed Adhoc Track
RIRES: Russian Information Retrieval Evaluation Seminar

 News 
 About 
 Manifesto 
 Call for participation 
 General principles 
 Participation 
 Tracks 
 Participants 
 Test collections 
 Publications 
 Relevance tables 
 History 
 2004 
 2005 
 Forum 

По-русскиПо-русски
 

Mixed Adhoc Track

Adhoc search in a mixed collection of documents

Overview

The purpose of this track is to evaluate adhoc search methods on a heterogeneous document collection.

For this track the standard procedure is used.

Test Collection

The source dataset is a union of three collections: BY.web collection, KM.ru collection and Legal documents collection.

Task Description for Participating Systems

Each participant is granted access to the document collections (BY.web, KM.ru and legal documents collection) and a set of queries which is a union of queries from the Web adhoc and legal adhoc tracks. So the total number of queries is 34424.

Expected result is an ordered list of document URLs. Maximum list size is 100 per query.

Evaluation Methodology

  • instructions for assessors:
    Assessors evaluate document relevance to the query basing on the extended description of the user information need.
  • evaluation method: pooling (pool depth is 50)
  • relevance scale:
    • yes / probably yes / perhaps yes / no / impossible to evaluate
    • yes / no / impossible to evaluate
  • official metrics:
    • precision
    • recall
    • TREC 11-point precision/recall graph
    • bpref

Data Formats