Process and outcome: On the evaluation of IR systems in the age of interaction, GUIs and multimedia

The question I want to address in this talk has to do with the relation between traditional ideas of retrieval system evaluation and the present concern with highly interactive systems, maybe involving other media as well as or instead of text. For reasons that I hope to make clear, my major concern is with interaction rather than with the media themselves. Particularly, I want to consider the questions: Why has the notion of relevance been such a powerful device in the traditional model of evaluation? And: Why is it not so useful in the evaluation of interactive systems?