Relevance tables used in ROMIP tracks
RIRES: Russian Information Retrieval Evaluation Seminar

 News 
 About 
 Manifesto 
 Call for participation 
 General principles 
 Participation 
 Tracks 
 Participants 
 Test collections 
 Publications 
 Relevance tables 
 History 
 2004 
 2005 
 Forum 

По-русскиПо-русски
 

Relevance tables used in ROMIP tracks

For calculation of the evaluation metrics we use relevance tables most of which are based on the marks put down by the assessors.

The are several different types of relevance tables:

  • AND - "strong relevance" (all assessors mark the document as "relevant" to the given query)
  • OR - "loose relevance" (at least one assessor marks the document as "relevant" to the given query)
  • SINGLE - each document/query pair was evaluated by exactly one assessor
  • IDEAL - comparison with a reference catalog (without assessors)

Relevance tables by track: