Strategies for Evaluation of Interactive Multimedia Information Retrieval Systems

The standard criteria for evaluation of information retrieval (IR) systems: effectiveness, efficiency, usability, satisfaction, cost-benefit seem as applicable to the interactive multimedia context as to the non-interactive, text-based context in which they have been developed. However, the operationalizations, measures and methods developed in the traditional context are, for a variety of reasons, almost wholly inadequate for the new context. This paper discusses some of the problematic aspects of evaluation in this new context, and suggests some strategies for developing new measures and methodologies for the evaluation of interactive multimedia IR systems.