Manifesto of Russian Information Retrieval Evaluation Seminar (ROMIP)

In recent years a significant progress has been made in information retrieval theory as well as in development of industrial information search engines.

Continuous evolution of information space and applying search methods in new contexts fosters further research in information retrieval theory.

Impressive development pace is partially stipulated by fast Internet growth and escalating interest in providing search functionality within application systems from different application fields.

Another important driving force is evaluation, which is the dominant instrument of identifying efficiency of alternative approaches. In addition to numerous specialized investigations on this matter, there are a number of initiatives on independent evaluation (for example, TREC, SUMMAC and CLEF).

At the moment, initiatives of the kind are not held in Russia, and research in the field of information retrieval is fairly uncoordinated. Participation in world initiatives is complicated and allows making evaluation of systems working exclusively with foreign languages.

The goal of Russian Information Retrieval Evaluation Seminar (ROMIP) is creation of the platform for independent information retrieval evaluation methods which are Russian language oriented.

• consolidation of community of Russian researchers and developers working in information retrieval;
• cooperation with foreign initiatives on testing of information retrieval systems;
• expansion of contacts with foreign researchers and developers working on multilingual information retrieval;
• attraction of attention to problems and perspectives of information retrieval and fostering of research in this area in Russia;
• creation of necessary prerequisites and stimuli for further development of participating systems in order to improve the quality of information retrieval.

It is a question of long-term perspectives, not a one-time event. In the future it is planned to create a number of test corpora and evaluation methods for different tasks of information retrieval.

At the first stage it is planned to create one testing corpus (a collection of data and test tasks) and make evaluation on its basis. Participants are not required to provide their systems or access to them, but the results of the fulfilled test tasks only. A separate seminar meeting is planned for discussion of results.

The approaches used for evaluation are going to be open-access, formulated in accordance with opinions of participants. It is intended to provide results of the evaluation in a open space; test corpora will also be available to non-participants of the initiative after evaluation. The results of testing are intended exclusively for research and cannot be used for marketing/commercial purposes unless agreed with the participant.

The initiative is strongly non-commercial. Expenses for preparation and conduction of testing is going to be covered by participation fees. Specific conditions are currently under discussion, the more detailed information will be available from the website later on.

The initiative is open for participation for all those interested, the commercial application developers as well as researchers. If you are interested in participation, feel free to contact us by e-mail romip[AT]romip.ru.

If you are willing in participating in the discussion concerning the seminar, you can join ROMIP mailing list which contains the list of group members.