Design experiment on two information retrieval learning environments

Design experiment on two information retrieval (IR) learning environments is presented. The participants in the study (n=57), were undergraduate students of information studies at the university. The pedagogical design of the experimental learning environment was based on the ideas from anchored instruction and intentional scaffolding. Anchoring related the search exercises to a simulated journalistic work-task situation. Scaffolding, i.e. various ways of supporting learners in proceeding with their task was provided by an instructional tool, the Query Performance Analyzer, or by a teacher. The pedagogical design of the traditional learning environment consisted of search exercises on several operational IR systems with unintentional scaffolding. The effect of the learning environments on the students’ learning experiences, performance and learning outcomes was evaluated. The evaluation was based on seven different datasets gathered in the course. Students in the experimental learning environment took the view that anchored instruction increased meaningfulness of learning tasks. The overall effectiveness of queries in the search exercises was slightly better and the students made far fewer semantic knowledge errors and change in conceptual IR know-how was also larger than in the student group in the traditional learning environment. Students from both environments made quite the same number of syntactic knowledge errors.


INTRODUCTION
Information retrieval (IR) has become a commonplace activity in our networked world.People constantly use various search services in order to find relevant documents to satisfy their information needs.These search services may, for example, be web-based library catalogues, archives of newspaper or journal articles, or web search engines.Information retrieval as a concept is related to the user and user's information needs.Based on these needs the user selects appropriate IR systems and sources constructing a query that represents her information need.Search results are then evaluated by their relevance to the user of the IR system.Users' interaction with IR systems i.e. information searching, is in many cases a complicated phenomenon.
There are some special aspects related to IR activities from an educational and instructional viewpoint.These include firstly, task dependence of information searching.In the real-world, searching is bound to various task situations as described above.Therefore IR instruction is seldom successful as a de-contextualized activity.Secondly, users of IR systems encounter uncertainty in various phases of searching.Information searching is basically motivated by uncertainty in a situation i.e. the need to find information to reduce uncertainty.Uncertainty also relates to selection of information sources, search keys and evaluation of search results.Thirdly, although we can outline the main phases of searching, we seldom can provide definite rules on how to proceed in IR tasks.These ill-defined rules for proceeding are important elements of IR instruction.Learners should be supported with motivating cues and hints, without providing too ready-made solutions.IR instruction should also equip learners with transferable skills to manage searching in different operational IR systems and interfaces.
The present study, however, concentrates on the pedagogical design and evaluation of two IR learning environments.Novelty and importance of the study is based on the following aspects.First, the study is based on a pedagogical design of an experimental IR learning environment, which took into account the conceptions of learning and characteristics of learning environments.Second, special aspects related to IR activities, described above, have been taken into account in pedagogical design.Third, innovative pedagogical solutions, within IS, anchored instruction and scaffolding were used to support learners.Finally fourth, a novel instructional application, the Query Performance Analyzer (QPA), was also used to scaffold and support the learner.The present workshop presentation is based on the design experiment reported detail in doctoral thesis (Halttunen 2004) and four original research articles (Halttunen & Sormunen 2000, Halttunen 2003a, 2003b, Halttunen & Järvelin 2005).

RESEARCH DESIGN
Understanding how technology and pedagogical solutions can best support student learning in diverse learning environments remains a crucial line of educational research and development.Finding a suitable approach to rapid technological change and the identification of best practices are core ideas of "design experiments".Collins (1992) describes an educational research experiment carried out in a complex learning context, which explores how a technological innovation affects student learning and educational practice.The goals of design experiments are to design and implement innovative learning environments and simultaneously understand salient aspects of human cognition and learning involved in those innovations.Design experiments: address learning programs involving important subject matter are usually mediated by innovative technology are embedded in everyday social contexts which are often classrooms, homes and workplaces where it is hard to control unanticipated events account for multiple dependent variables do not attempt to hold variables constant, but rather identify many variables and the nature and the extent of their effects evaluate different aspects of the design and develop a profile to characterize the design in practice.
The present IR design experiment was based on varying the instructional design of tutored exercises within two IR learning environments in naturalistic educational context.The main variation of these environments was the implementation of anchored instruction, intentional scaffolding and the use of the QPA as an instructional tool.Anchored instruction was used to create a macro-context for IR activities.We used the full-text database of a local newspaper, an image database of a national press agency and a national bibliographic database on journal and newspaper articles.Based on these tools we created a context of journalistic practice.In other words, IR activities were situated in a simulated work-task situation, where search-tasks were based on the idea of a journalist searching the text and image databases available in order to find information on certain topics for a forthcoming article.In addition, some topics required the reporter to search national databases on certain topics.
Scaffolding was offered both by tutors and the QPA.The teacher can model the search process by providing examples focusing not only on end products (efficient query formulations), but also on the search process.The teacher provides hints, either initially or on an ongoing basis, on query formulation.The provision of examples of possible search keys serves as a scaffold in certain situations.Coaching comments are intended for motivation, providing feedback and advice on performance, and provoking reflection.Different kinds of questions can be set to enhance reflection, for example pointing out weaknesses and asking about motivation.The provision of a timeline with fixed timing and goals provides support for goal direction and reflection.The gradual removal of scaffolding, i.e. fading, was based on student performance in the exercises.When the students were able to construct queries, scaffolds such as examples and hints were removed.Coaching comments and questions were used throughout the exercises to improve reflection and articulation.Relevance feedback provided by the QPA was removed in one of the search-tasks, when students selected their own viewpoints on the topic.
The basic idea and functionality of the QPA is that the query performance feedback scaffolds the learner by providing information on query performance.The learner receives feedback on query construction, which enables her/him to evaluate different search strategies and tactics.Learners can concentrate on the analysis of effective query formulation without spending lots of time in analyzing results.The "Give a hint" function provides the learner with one non-retrieved relevant document.This document serves as a cue to the selection of appropriate search keys or document structures usable while searching.The possibility of identifying the retrieved relevant documents also serves as a cue in the same respect.The hall of fame provides examples of best queries on the current topic over time.The opportunity to see one's own performance compared to the query performance of other users provides an area of comparison and feedback.Search topics can serve as a scaffold, providing examples of search keys and informing searchers about applicable restrictions.
The software-based scaffolds in the QPA are based on the basic idea of "knowing the right documents" for each topic.In a way this approach pays attention to the end product of the process, to effective query formulation with good precision and recall, but not interactive learning process support as such.The face-to-face scaffolding described above is needed in this situation.The QPA was used both in traditional and experimental learning environments.In the traditional environment it was used in a one-hour -long tutored exercise to demonstrate concepts like relevance, precision and recall with five exercises.In the experimental environment the QPA was used in most of the exercises (five hours), along with anchored instruction and intentional scaffolding.
In the traditional learning environment different operational search systems (an OPAC, a union catalogue, article reference databases and full-text databases) were used to demonstrate basic functions or IR systems.The same two article reference databases were also used in the experimental setting as in traditional instruction.

Research problem
The research problem of the design experiment was: "In what ways does the experimental IR learning environment affect learning experiences, performance and outcomes on a basic course in IR compared to more traditional instruction?"In order to answer the main question the following four sub-questions were outlined: What are students' prior conceptions of IR? What are students' learning experiences in the two IR learning environments?What is students' performance in the two IR learning environments?What are the learning outcomes as assessed through a) the overall conceptual change and b) development of IR skills in the two IR learning environments?and answered with the aid of seven datasets and various methods of analysis as described below.

Research data and analytic methods
The course "Introduction to Information Retrieval" (6 ECTS credits) at the Department of Information Studies at the University of Tampere was intended for first year undergraduate students.The course provided an overview of information storage and retrieval as practice and as a research area.Themes like the production and structure of databases, matching, metadata, subject description, query languages and formulation, search strategies and tactics, and evaluation were covered in the course.
The course consisted of four instructional elements.First, lectures were given on basic concepts of information storage and retrieval.Along with the themes covered above, lectures presented advance organizers of topics and provided interactive feedback and summaries of exercises and learning tasks completed during the course.Second, weekly web-exercises concentrated on putting the themes covered in lectures into practice.Every participant worked on these exercises making use of web-based tools and resources.Exercises were reported on web-forms, which were mailed to the teacher.Feedback on these tasks was given during the lectures.Third, tutored exercises in the classroom covered various aspects of information retrieval systems and their effective use.In these sessions, groups of 8-10 students worked in pairs.These excercises formed the basis of variation in the design experiment as described above.Fourth, a course feedback web-form was filled out at the end of the course.This feedback covered three main areas: 1) course design and teaching methods, 2) the learner's self evaluation and role in the course and 3) the teacher' s role in the course.The course feedback was an integral part of the course and it provides questions to support learners' self-evaluation and reflection.
About 120 students attended this course, and a lecturer and two tutors taking care of a part of the tutored sessions provided instruction.The course outline, a bulletin board and exercises were provided via web-pages.Exhaustive handouts covering the lecture material were provided in print.Despite the number of participants, the course can be regarded as highly interactive and task oriented.The learning-by-doing approach was used in several situations.Assessments of learning outcomes were based on weekly web-exercises and tutored exercises.The dichotomy pass/fail is used in course evaluation as such without grading to indicate that a student had fulfilled/not fulfilled the learning objectives.Multiple datasets on the design and evaluation of the IR learning environment were gathered during the course: (1) Short essays describing students' conceptions of IR in the beginning of the course (2) questionnaire results on students' prior conceptions (3) a learning style inventory (4) search logs in tutored exercises (5) short essays describing students' conceptions of IR at the end of the course (6) empathy-based stories describing students subjective learning experiences (7) course feedback.
There is a complete collection of datasets from 57 students.Two IR learning environments were evaluated at three levels.We focus on students' learning experiences, performance and learning outcomes.
Students' subjective learning experiences concerning the learning environment and the QPA were studied using the method of empathy-based stories (Eskola A., 1988) and the course feedback questionnaire.Stories and course feedback were analyzed qualitatively by theme coding, categorization, and with the aid of matrices (Miles & Huberman, 1994).
The development of students' conceptions of IR know-how during the course was examined through student essays at the beginning and the end of the course.This data was first analyzed through the phenomenographic approach (Marton, 1988(Marton, , 1994) ) in order to ascertain students' conceptions of IR and how it may affect the design of learning environments.These conceptions were also used as a baseline for the evaluation of learning outcomes.Prior conceptions of information retrieval were also studied through the questionnaire, which presents visual analogue scales (VASs) of different kinds of conceptions of IR.
Student performance in the two learning environments was analyzed by assessing their search sessions both in terms of query construction and modification and the overall effectiveness of queries.Query construction and modification was analyzed with the scheme concentrating on (1) the construction of facets and the use of operators, (2) the use of truncation and masking, (3) the use of field restrictions.The overall effectiveness of best queries of each student within the four exercises was measured by average precision and recall.
Learning outcomes were analyzed on two levels.First, the learning outcomes were assessed by conceptual change.Essays at the beginning and end of the course were analyzed with the aid of concept-maps.These maps describe the essence of concepts, and the number of, and connections between, concepts (Novak & Musonda, 1991;Novak, 1998).Second, the development of IR skills was analyzed with performance assessment criteria based on search-logs in an assessment situation during the last tutored exercises.This assessment was based on analyzing problems and errors in queries and the effectiveness of queries.A scoring scheme was devised for logfile analysis.It identified all possible errors which students had made during each session.

RESULTS
Students' prior conceptions of IR proved to be quite rich.Learners' conceptions of IR know-how cover a broad spectrum of IR activities.The analysis of short essays revealed 12 categories of description.These different conceptions are scattered over the phases of the search process.Students paid special attention to the initial phases of the search process, i.e. problem formulation and analysis of information needs.Students fell into five qualitatively different groups, namely: process identifiers; source identifiers; searchers; problem formulators; assessors.These prior conceptions suggest that pedagogical design of IR learning environments should take in to account the broad spectrum of activities, although it is impossible to cover them within one course.Creating anchors, simulated work tasks, and other contextual elements within IR instruction is one solution to enhance learning of new aspects of IR by relating them to learners' prior conceptions.Making of this connection is of vital importance according to modern views of learning.
Students' learning experiences in the two IR learning environments can be outlined in two levels.First, descriptions of good and poor learning experiences revealed general aspects affecting learning.The most important aspects related to learning experiences were: meaningfulness of learning tasks and their relation to prior conceptions and know-how; instructional design with process approach, clear structure and interplay between theory and practice; and feedback on performance in the learning environment.Second, learning experiences related especially to the present experimental pedagogical design consisting of anchoring, scaffolding and the QPA can be summed up as following: Participants in the experimental learning environment considered that the process approach and interplay between theory and practice were particularly good elements of their learning environment.They also received enough feedback on their performance.These results indicate that modeling of search process and other scaffolds were able to support the learners to acquire knowledge and skills to manage their search process and relate basic phenomena of IR to their searching.
Students' performance in the two IR learning environments proved to be slightly different.Results indicate that students in the experimental learning environment took into account the variety of possibly effective search keys.The overall effectiveness of queries was better in the experimental learning environment.Students in the traditional learning environment applied more gaming strategies, i.e. artificial strategies to improve precision.Results indicate that anchoring IR activities to journalistic context and usage of fewer full-text databases in the experimental learning environment enhanced students' learning about the analysis of information needs and identification of possibly relevant search keys in relation to information sources available.It seems also that anchoring and scaffolding were applicable pedagogical solutions to overcome the limitations of the QPA, i.e. it was possible to hinder the artificial gaming approach.
Students' learning outcomes in the two IR learning environments proved to be slightly different.The evaluation covered students' learning outcomes regarding both conceptual change and development of IR skills.Students in the experimental group changed fundamentally, or dismissed, more concepts while studying than students in the traditional learning environment.The traditional group was somewhat more stable in their conceptions throughout the instruction.The analysis of conceptual change that relates to students' prior conceptions revealed successful learning outcomes.The fact that there were not so much clear differences between students' conceptual change in the two learning environments is understandable since lectures given were the same in both environments.It seems that students in the experimental environment could enrich their conceptual structures with the help of instruction given in tutored exercises, but students from both environments could reach successful learning outcomes when compared to their prior conceptions.Performance assessment on IR skills revealed statistically significant difference between the groups in semantic knowledge errors.The traditional group made much more semantic knowledge errors, i.e. errors related to the process of transforming a search assignment into a query, than participants studying in the experimental learning environment.Students from both learning environments made quite the same number of syntactic knowledge errors.It seems that both groups were able to overcome problems with syntactic errors with active exploration, but semantic problems affected their overall performance, since students in the traditional environment were not able to achieve as good search results as participants in the experimental group.Results indicate that the experimental learning environment was able to support knowledge creation and skills development especially in relation to semantic aspects of IR.
Along with the results of the study describing prior conceptions, learning experiences and performance, and learning outcomes in the two IR learning environments, the novel approaches to IR instruction are contributions of the study.The contributions of the study are summed up as follows: -The design and implementation of a novel information retrieval learning environment based on pedagogical design.The design took account of the special features of IR.These include, task dependence, uncertainty, and ill-defined rules of proceeding.Pedagogical approaches like anchored instruction and scaffolding were used to support the learner.
-Using the QPA as one component of the IR learning environment.The QPA gives the user query performance feedback based on a recall-base of known relevant documents.This feedback served as a scaffold in the learning environment.
-The evaluation of learning experiences and outcomes in the two IR learning environments.Evaluation was based on various forms of qualitative and quantitative data analysis.Evaluation was performed in the naturalistic context.
-The evaluation results indicate that anchoring and scaffolding are promising strategies to support learners in various phases of searching.The results and reflection of the design process contributed to our understanding of the role of various forms of human and computer supported scaffolding.

DISCUSSION
The importance of the present study lies in its attempt to combine some of the special elements of the IR processes, i.e. uncertainty, ill-defined rules on proceeding, to pedagogically motivated frameworks for the design of learning environments.Although, interfaces of IR systems have become easier to use, they still seldom provide support for a searcher or learner of these systems.It can be argued that problems in learning to use IR systems successfully relate today more to semantic aspects of IR, and connections between work tasks, information needs and management of search processes, than to syntactic aspects of IR systems.Today's graphical user interfaces can support information searching at the level of moves, but they seldom provide support to the user at the levels of tactics, stratagems, and strategies identified by Bates (1990).
The evaluation results indicate that anchoring and scaffolding can provide usable solutions to implement in IR instruction.It seems that anchoring can be used to overcome the problems of de-contextualized IR instruction.Artificial search assignments, concentrating on operational, syntactic skills often produce learning that remains inert.In this situation it is difficult to transfer the learned skills and knowledge to real-world situations.(Mandl et al., 1996.)The results indicate that students were able to transfer the IR skills developed in the learning environment in to operational IR system.The uncertainty and ill-defined rules on proceeding in information searching situations often produce situations where learners could proceed on their own, if they received additional support.Scaffolding, is a powerful pedagogical approach in IR instruction, both in pedagogical design and informal instruction given to users of IR systems in schools, libraries etc.The ideas of scaffolding might also inform the interface designers of IR systems.Designing learning environments for IR, instructional interventions during IR instruction, user support in IR systems, and interfaces of IR systems share similar problems.Reducing uncertainty and supporting learner or searcher to the next phase or level are the main aspects in all of these design fields.
There are some limitations to the study that ought to be noted.The design experiment in tutored exercises is only a small part of the whole learning environment and its effect on learning experiences and outcomes is not as obvious as it could have been in more extreme situations where the elements of learning environments are totally different.Anchoring was implemented in a way that was possible without substantial technical and economic investment on, for example, video-based interactive tutorials of journalistic practice.We used classroom discussions, modeling and background stories to anchor the IR activities into a context.These designs can also be seen as strength in the present research.We operated in an operational, authentic environment in an exploratory way to identify the effects of scaffolding and anchored instruction on learning experiences, performance and outcomes in the two IR learning environments.On the other hand, it is very difficult or even impossible to control for all possible factors in an authentic field experiment and in the case of a pedagogical design that tries to be naturalistic.It would also be artificial to try to differentiate between factors in a very detailed manner.The results of the study, in terms of the benefits of anchoring and scaffolding, are not categorical because of the range of intervening variables and the difficulty of setting up a field experiment which tried to be naturalistic but at the same time tried to focus on a specific aspect.