Relevance tables used in ROMIP tracks

For calculation of the evaluation metrics we use relevance tables most of which are based on the marks put down by the assessors.

The are several different types of relevance tables:

• AND - "strong relevance" (all assessors mark the document as "relevant" to the given query)
• OR - "loose relevance" (at least one assessor marks the document as "relevant" to the given query)
• SINGLE - each document/query pair was evaluated by exactly one assessor
• IDEAL - comparison with a reference catalog (without assessors)

Relevance tables by track:

• Web adhoc track
  • 2003: AND OR
  • 2004:
    • with extended descriptions of user information needs: AND OR
    • without extended descriptions of user information needs: AND OR
  • 2005: AND OR
  • 2006: AND OR
• Legal adhoc track
  • 2004: AND OR
  • 2005: SINGLE
  • 2006: SINGLE
• Mixed adhoc track
  • 2005: AND OR
  • 2006: AND OR
• Similar documents search
  • 2005: AND OR
  • 2006: AND OR
• Legal documents classification
• 2004: IDEAL40 AND OR
• 2005: IDEAL40
• 2006: IDEAL40
• Web site classification
• 2003: AND OR
• 2004: AND OR
• 2005: AND OR
• 2006: AND OR
• Web page classification
• 2005: AND OR
• 2006: AND OR
• Fact extraction from news
  • 2005: AND OR
• News clustering
• 2006:
• Query-biased summarization
  • 2005: AND OR
  • 2006:
    • for the relevant-minus level AND OR
    • for the relevant-plus level AND OR
    • for the vital level AND OR
• Question answering
  • 2006: