Design in the Wild: Interfacing the OER learning journey

Open Educational Resources (OERs) such as talks, lectures, texts, slideshows, and online activities, offer much potential for people to study a broad range of topics. However, the choice among millions of OERs can be overwhelming and many learners give up on their goals prematurely. Our ongoing research aims to help self-directed learners study effectively and enjoyably by providing a personalized route through appropriately prioritized OERs. However, there are many aspects that can be considered including motivation, getting lost, determining whether to continue, and what resource to look at next. How can we begin to design an interface that can support these? To this end, we describe our iterative ‘design in the wild’ approach, showing how it helped us to determine how to operationalise and support these aspects and, in doing so, provide us with a way of assessing the overall learning experience.


INTRODUCTION
Adult learners have a diversity of prior skills and knowledge, together with preferences for how they want to learn. Such individual needs have been recognized by MOOC research in recent attempts to make them more adaptive. For example, features that have been added to enable more personalisation included course-specific features, such as intelligent tutoring systems [1] , recommender systems [4] , and A/B testing [6] . However, it is difficult to know how effective these approaches are as they typically focus on evaluating their success through largely measuring completion rates based on log data. While such online data can be informative, it does not take into account other 'off-platform' learning activities [5] , such as the learners' knowledge before starting the MOOC course, what other online resources they use in parallel, and importantly the actual experience of learning. How much do students enjoy the course, what do they get out of it? How can these aspects be more extensively taken into account alongside what learning activities can be logged, in order to inform the next generation of self-directed learning -OERs -that are now becoming more mainstream?
Open Educational Resources (OER) provide a new opportunity to create personalised learning pathways for each learner, by connecting resources of various sizes, types, and origins. OERs may include MOOCs as well as smaller units, such as tutorials, books, lectures, etc and fragments thereof. With AI-based recommender systems beginning to outperform human recommendations , there is great potential for designing systems that analyse someone's long-term learning history in order to infer their knowledge and preferences and give informed recommendations regarding what to study next [2] . By allowing the learner to change paths and adapting continuously, this approach promises more flexibility for self-determined learners to take ownership of their journey. At the same time, it is far from obvious to what extent people's learning habits and preferences transfer naturally to the unfamiliar context of personalised learning. For instance, how will the abundance of choice of learning materials affect planning, engagement, sense of progress and achievement, reflection, and perseverance? How can peer interaction be orchestrated when no two learners' journeys are the same? How will people handle moments of getting sidetracked, stuck, overwhelmed, distracted, etc in this novel kind of environment? Will their usual strategies suffice or will new strategies emerge? What measures of success and support are appropriate in this context? In the following we describe our research approach.

AIMS
The aims of our ongoing research are: • to collect empirical evidence of how learners use OER recommendations in the wild, their needs and expectations; • establish appropriate learner-centric definitions of success and progress; • inform the design of user interfaces and learning platforms around OER recommender systems.

METHODOLOGY
When determining how to design an effective, efficient and satisfying learning experience, we decided to focus on the experience of real learners under real conditions, rather than assumed scenarios or personas. To begin, we designed a minimalistic web interface, based on the results of initial interviews suggesting that learners need to (1) find OERs, (2) preview and compare search results, and (3) keep track of selected resources over days and weeks. Many interviewees said they were happy with the results from Google search but expressed frustration about having too many browser tabs open, which made it difficult to preview, compare and keep track of things over time and across devices. Consequently, our initial design started with a Google search for a popular topic, in this case "Introduction to machine learning". We selected a number of resources, and decided how best to structure and present these at the interface and how they should adapt to the user. The details of the initial design are presented later.
The reasons for choosing Machine Learning (ML) as a topic included the practical benefit of having domain expertise represented in our team. Furthermore, ML is practically applicable and of high relevance to the general public, formal education and professional training. It is a fast-changing knowledge domain: new OERs are published every day and courses can quickly become outdated.

DESIGN IN THE WILD
The participants were recruited based on their expressed interest in an introduction to ML. Rather than just ask them to use our system, we asked them to explicitly help us iterate and evolve our OER system in a real world context. 8 participants (5 male, 3 female, age 23-57) were invited to use the interface for as long as they found it useful, in their free time, using their own laptop or smartphone. They were also asked to provide feedback during a weekly interview. These were directed towards building an in-depth understanding of the learner's experience over time (including usability but also getting stuck, sidetracked, etc, see introduction) and towards informing design changes that address the learners' explicitly stated needs.
One-on-one sessions with participants have constituted the primary data source. Also taken into account was the extent to which the participants used the interface for collecting resources and taking notes. In addition, feedback from 3 experts in the area was elicited at various stages. This 'design in the wild' iteration process took place over two months. Below we describe the initial design of the interface and how it evolved.

Initial design
The initial OER design presented users with 37 manually selected, ML-related OER recommendations, including several online courses, articles, podcasts, tutorials, meetup groups, etc. We decided on a small number to begin with, in order to be flexible in case of any need for manual processing. We also took inspiration from familiar websites like Netflix and Amazon, that people are very familiar with using, to present the items, as horizontal icon-based lists, categorised by media type and other criteria, such as "Because you are interested in learning Python". By selecting an item, users could see more information, take notes and mark the item as started or completed. Two dedicated lists allowed learners to easily find started and completed items again. Basic functionality for overall note-taking and diary was also provided. "Bookmarks" instead of "started" In response to user feedback, the categories "started" and "completed" were renamed as "bookmarks" and "archive", in order to better apply to certain types of resources, such as meetups, short videos, and articles intended for continuous reference. This seemingly minor change highlighted the potential for journeys to be made open-ended.

Support for journeys
As their collections of OERs they had selected for viewing grew over the weeks, several participants expressed a need to organise their bookmarked resources into meaningful groups. Different designs were considered and tested with participants, including "tags", "folders", "projects", "problems" (perceived as sounding too negative), "workspaces" (too static), and "journeys". The latter was found most appropriate by the majority of participants, as it matched best how they perceived what they were doing when interacting with the OERs. A journey can be short or long, focused or exploratory, completable or open-ended. Moreover, the idea of inviting others to join a journey was perceived as compelling and intuitive. When creating a new journey, the user is required to give it a name. Surprisingly, this feature was also used as a planning tool by one participant who created multiple journeys at once, like a todolist of subtopics to study.

Peer involvement
Several participants suggested the possibility to share a journey, or multiple journeys, with other people. To get an indication of the real demand for such a feature, it was implemented as a mockup and participants were asked whether they would use it in practice and if so, what for and how. A variety of reasons were expressed. For instance, one participant suggested "to have a real person check in on my progress once in a while -as opposed to automatic email reminders, because those I just ignore". Another participant said: "Just to have someone who is also trying to achieve the same thing [...] that would be motivating." Three participants said that they could imagine inviting their friends, partners, fellow university students, etc who are also keen to learn about the topic, but not necessarily strangers.
Overall, participants expressed considerable interest, mostly in regard to increasing their own motivation and perseverance. Conversely, one participant suggested that by sharing her bookmarks she could help other learners. Finally, two participants expressed that they would probably prefer to study alone, at least initially or most of the time.

Need for previews
To avoid opening too many browser tabs, four participants asked for a more convenient way to preview OERs. Because in our case OERs are represented as URLs of web pages, we experimented with embedding the target web pages in our interface in small boxes (HTML iframes). Surprisingly, the result was found quite usable with many target websites, including Wikipedia and websites that appeared in a mobile-friendly version. However, a considerable number of popular websites showed nothing but blank space in response to being embedded, indicating that this solution would not easily generalise. A variety of alternatives are currently being considered, including screenshots, extracted features and services that provide page previews.

Definitions of progress
One participants suggested that journeys could indicate the learner's progress in some form, "[...] to know how close you are to completion". However, when asked where and how they would like this information displayed, difficulties became apparent. Concerns were raised that the size of a journey is rarely knowable in advance and may vary as new things are discovered. Moreover, some journeys may be open-ended. Further questions were raised regarding appropriate metrics, e.g. number of resources, time investment, level of understanding or types of engagement. One participant suggested: "maybe you specify progress in terms of a feeling."

OBSERVED USE OF OER
Participants were interested in a variety of media types, including online courses. However, choosing between courses was generally found to be difficult. Given the time investment involved in trying out a course, none of the participants tried out more than two courses and some participants settled for a suboptimal choice. For instance, one participant was content with their course except that they found the programming examples to use an older version of the programming language (Python). The question was raised whether our system could be used to find alternative examples that also matched the course lectures but were more up to date.
Participants' reflections on the importance of following a recommended OER from beginning to end were mixed. For the most part, our findings indicated that self-directed learners care very little about completing or not completing a MOOC or other OER. What mattered more to them was using their time efficiently. In the words of one participant, who said that she generally avoided MOOCs for this reason: "I tend to avoid learning things that I wouldn't immediately apply in one of my projects, because then they don't stick and I would have to learn them again. I would probably watch a video to get an introduction but then I use Google to look up very specific problems that I need to solve." Another participant explained: "[...] like in a supermarket, you're not expected to buy everything. It's OK to leave some things on the shelf. Or with a textbook, sometimes you want to read the whole thing and sometimes you only care about a specific chapter." The participants agreed that our prototype OER interface was functional, usable and attractive. Some participants reported using it less frequently after a target website had drawn them away from it. They kept on that site. For instance: "I found this Udacity course through the interface. Now most days I go directly to Udacity, rather than going through the interface, because it's faster. But occasionally I come back here to take notes." Similar cases were reported where participants found themselves being drawn to playlists on YouTube for extended periods. While this switching to other sites makes it harder to keep track of what learners are doing online, it was not seen as a problem for their learning experience as long as the content was relevant. The tendency to go to other sites, however, does raise questions about how we bring them back and how they sit with our accompanying personalised learning resources.
As a result of the comments and feedback made by the participants we were able to reflect upon how small changes at the interface would impact upon the higher level pedagogical goals we hoped to foster. Below we discuss the results of our 'design in the wild' approach with respect to our aims.

DISCUSSION
Our ongoing design and in-depth evaluation with real users produced a number of insights that we acted upon in changing aspects of the interface. Making these changes with a live audience enabled us to get iterative feedback on the fly -providing us with a way of testing the new features and discovering how they impacted on the learning experience. For example, the concept of "journey" suggested by the participants to describe what they were doing, was found to be useful for us as an analytical tool to describe the experience and selfperception of self-directed learners using OER recommendations. While the notion of "learning pathways" as used in [2] represents a contentcentric view on a series of resources, the "journey" is the experiential counterpart, supporting a complementary, learner-centric view.
At a more general level, we identified 3 areas when considering the next iteration of the interface design: peers, previews, and progress. By these are meant: (1) leveraging peer involvement to support learners' motivation, perseverance and help-seeking, (2) enabling learners to efficiently preview OERs of diverse media types, (3) helping learners reflect on and manage their progress. Our next steps require operationalising these in terms of the learner journey and to be able to implement these at the interface.
Another future direction is to consider how much of the interface to leave open for learners to choose from available OERs and how much to scaffold through using automated recommendation. Given the ever increasing number of resources for learners to choose from, leaving certain things unfinished might be preferable. An implication for design is that learners might value an interface that helps them mix and match different resources and parts thereof.
Our next steps also will be to address how to design aspects of an interface that can help with supporting learners' strategies for dealing with obstacles and impasses -such as getting stuck, side-tracked, overwhelmed, or distracted. Recent MOOC research has begun to recognise these issues [3] and insights gained in this context could in some cases transfer to self-directed OER journeys as well.

CONCLUSIONS
Our 'design in the wild' approach has enabled us to iteratively design a user interface for OER recommendations that adopts as its central metaphor -the learner journey. Using the "journey" as our primary conceptual unit of analysis, also enabled us to begin to consider how to design functionality that can support learner engagement and reflection. We have also found through this process that small changes at the interface can enable us to explore more generally what impacts they can have on learning at large.

ACKNOWLEDGMENTS
The research described in this paper was conducted as part of the X5GON project www.x5gon.org which has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 761758.