Design Discovery Practices: Engaging professional design communities with Ketso

Decisions and assumptions made during design sessions, when teams are formulating their design objectives and their understanding of the problem they intend to solve, can be essential to the outcome as they fundamentally shape and direct the design of the product or service that is delivered. Current practice in these crucial design discovery activities is under-explored in the academic literature. To address that shortfall, and answer the research question of how UX practitioners approach and perform discovery, we used the Ketso workshop format to explore the design discovery process and its challenges with 12 user researchers and designers from a university and a large retail organisation. Our thematic analysis of the workshop outputs showed that practitioners valued an empirical data-led approach, where they could have confidence in the coverage and validity of the data, and achieve a shared understanding of the user research findings across the organisation. Key challenges included the mindset of stakeholders, with whom practitioners wanted deeper engagement, and constraints on time which may require HCI research to develop practical solutions.


INTRODUCTION
Choices made during design sessions may be based on an incomplete, and possibly mistaken, understanding of the problem, but still have the potential to fundamentally shape and direct the design in ways that will be apparent in the product or service that is delivered to customers. Some choices are conscious decisions, others may be unchallenged assumptions. In order to have confidence in our delivery processes, we first need to have confidence in how we decided what to build.
Teams formulate their design objectives, and their understanding of the problem they intend to solve, by establishing that there is a need to be met, that they know how to build it, that potential users will want it, and that stakeholders will support it. The data required for this comes from user research and business analysis confirming the viability of a product. These activities are collectively termed 'design discovery' (Brown, 2017) or 'product discovery' (Cagan, 2018) or most commonly in the UX practitioner community simply 'discovery' (Government Digital Service, 2019). Use of the term 'discovery' in academic literature is more limited, typically it is used to refer to business models, as in 'discovery driven' (McGrath, 2010), or when discussing Lean start-up approaches (Shepherd and Gruber, 2020).
Academic literature describing design discovery practices in the UK software industry, and the rationale for their choice, is sparse and not specific to a geographic community of practice. The available papers take their focus from particular segments of the client community rather than the practitioner community, such as addressing particular issues for UX with children (Sim et al., 2017), or are concerned more with the integration of design with development than with design itself, for example examining the relationship with Agile as described by Salah et al. (2014) or Gregory et al. (2015).
To address that shortfall, the research question for this study is How do UX practitioners approach and perform discovery?
In order to address that question, workshops were held in summer 2019 and early 2020 to gather information on current design discovery practice by exploring the objectives that practitioners aim to satisfy. The scope of the workshops was specifically the design discovery stage of development, covering the planning and conduct of user research and the initial presentation and shared understanding of the research findings by the project team as a whole. While recognising that discovery implies a learning curve, and design choices may be made at different points of that curve, no assumptions were made about the distribution of discovery activity over time, and it was not assumed to occupy a distinct phase of development.
This paper contributes an initial thematic analysis of the key aspects of design discovery used by practitioners, the best practice that practitioners aspire to, and the organisational challenges that they identified.

RELATED WORK
The desire for rapid iterative deployment of software and other digital products has placed challenging demands on the user research and design activities that precede and support the product development. Various attempts have been made to integrate User Experience (UX) design with continuous development in ways that achieve greater agility. Some have taken a dual track approach where design activity runs in parallel but synchronises at key points (Cagan, 2018, Trieflinger et al., 2021, or periodically inserts a short design sprint to answer key design questions (Knapp et al., 2016). Other approaches, drawing on Lean manufacturing ideas (Gothelf and Seiden, 2016), have posed incremental outcome hypotheses to be tested by each development sprint.
As the available methods differ so significantly, it is important to understand which are being used and how they are being applied in practice. An international survey of understandings of User Experience (UX) design by Lallemand et al. (2015) found no clear consensus on quantitative versus qualitative approaches to UX, nor whether UX is an individual or social phenomenon, and importantly identified both a gap in the way UX is understood between industry and academia, and geographical differences in the way that it was related to marketing, usability, and emotion. This implies a need to understand communities of practice within their national context, as the differences may alter design outcomes.
A case study of the role of UX design professionals in Agile development by Bruun et al. (2018) used semi-structured interviews to gather data on industry practice in a single case company in Denmark. They found that UX professionals had a wide range of responsibilities, which made it more difficult to characterise their activities. The focus of their research was how well UX tasks integrated into Agile development, rather than the detail of those tasks, and they identified a need to better understand the relationship between customer centric and user centric needs as that may affect the willingness to fund UX activity. Our choice of open guiding questions for the workshop was intended to draw out information in a role-neutral way and give room for any such conflicting demands to emerge.
An interview study of practitioners attitudes to design methods, techniques, tools and processes by Gray (2016) identified that method use by practitioners departed from the projected use in the HCI literature, having an increased reliance on professional judgement and an appropriate mindset for tailoring the method to the specific problem. The lack of direct engagement between the research and practice communities was raised as a concern. Our choice of methods for this study was motivated by a desire to make participation in our study interesting and enjoyable to encourage further engagement.
The most challenging situation for our research question was expected to occur if non-Agile practices of design discovery were conducted in an environment tailored for Agile software development. A study addressing a matching problem by Kuusinen et al. (2016) used online surveys and Ketso workshops to ask practitioners about their experience of applying Agile methods in a non-Agile environment. The focus of their study was on the environment of practice rather than the nature of the practices themselves, but it showed the utility of the Ketso method with practitioners. Their analysis identified management buy-in and a supportive organisational culture and structures, the team environment itself, and effective application of practices as key themes.
No prior work was identified that specifically explores design discovery practice in the UK software industry. This study addresses that research gap.

THE RESEARCH STUDY
Our purpose in conducting this study was to gather information on current design discovery practice, and to better understand the context of that practice by exploring the objectives that practitioners aim to satisfy. Face-to-face workshops were chosen as the means of data collection to allow a free exchange of ideas between the participants and gain richer information than might be obtained by surveys. A Ketso (Tippett et al., 2007) workshop format was chosen to facilitate this. Ketso is a technique for engaging communities in discussion around specific topics. A Ketso session builds up a picture of participants' ideas written onto 'leaves' that are placed on 'branches' on a felt background (see Figure 1). This picture emerges through a structured discussion about the topic, in this case practices used during design discovery. This approach also fosters a safe environment where all participants are able to contribute equally without any one individual dominating the discussion. Ketso achieves this by combining individual idea generation and group discussion, structured by the 264 workshop materials and by the guiding questions asked by the facilitator.
Two workshops were held. The first workshop ran on university premises in late June 2019 with three participants. The second session ran with a large retailer and was held in their offices in mid-January 2020, with nine participants in three groups of three. Both took approximately 90 minutes including set-up, briefing, and clear-up. Relevant participant characteristics are listed in Table 1. None of the participants had used Ketso before.

Ontological and epistemological position
Our assumption is that the data gathered describes only how things seem to be and any conclusions drawn may only be applicable to the participating communities of practice, and we therefore take a bounded descriptive-relativist ontological position. The findings of thematic analysis are reflexively created as the research progresses, and our understanding of professional practices arising out of social interactions and sense-making is itself a social construct, so our epistemological position is pragmatic social constructionist with an interpretivist theoretical perspective (Moon and Blackman).

Research question and relevant themes
The research question and sub-questions can be stated as: How do UX practitioners approach and perform discovery?
 What is done in practice?  What would improve practice?  What are the challenges?
Any pattern of response in the workshops that addressed these questions was taken as an emergent theme.

Scope, intent, and depth of meaning
The aim was to develop a rich description of the whole, rather than a detailed account of one aspect, and our intention was an inductive analysis linked closely to the data, not a theoretical model of practice. The data corpus for this study implicitly included many hours of conversation with practitioners prior to the workshops, so although extracts for coding were identified at the explicit semantic level in the transcript, the themes under which they were grouped were necessarily based on our interpretative understanding of their latent meaning.

Ketso general details
Participants were asked a guiding question, and asked to write their own ideas onto leaf shapes. The Ketso leaves are colour coded to represent the kinds of ideas that are wanted at that stage, and have a letter in the corner of the leaf for those without full colour vision. Only one kind of leaf was provided for each question. The standard Ketso conventions were used: Taking turns, they introduced and explained their ideas to the group, and the leaves were then placed on to a felt workspace. The felt has a space at the centre, from which narrow coloured strips radiate out, representing branches. Oval label shapes were used for a reminder of the overall question, placed in the centre of the felt, and for labelling the branches. Each leaf was placed either onto a new branch or onto an existing branch that they seem related to. After introducing their individual ideas, the group discussed them, and were able to add more ideas or move them around if they saw more relevant connections. The facilitator then asked the next question. At any stage, a collectively agreed label could be written and added to a branch. Part of the workspace from the first session is shown in Figure 1. Recruitment was by internal communication within the organisations involved once initial contact had been made. In the case of the large retail organisation, this was based around their internal community of practice and the workshop took place in one of their regular meeting slots, at which they were accustomed to trying out new methods.
Participants were given a brief general overview of the format of the workshop and the Ketso materials. More detailed information on the use of the materials was given as each guiding question was introduced by the facilitator. Around each Ketso workspace, three seated participants sat at each table. There was sufficient room for all to have spare leaves and writing space, without any having to view the text upside-down or from an uncomfortable angle. Five minutes were allocated for idea generation, and 10 minutes for group discussion, of each question. Each guiding question was supported with prompts for the kinds of things we would like them to consider, and written up on a poster in the list form shown.

Guiding questions
Question 1 (Yellow) asked about their success criteria. They were prompted to think about how they would recognise a 'good' discovery session, what it should look like and feel like, and what the discovery should produce as an outcome or output.
Question 2 (Brown) asked what was currently working well. Based on the findings of Gray (2016), we explicitly prompted them to include mindset as one of the things to consider, in addition to methods and materials.
Question 3 (Green) asked what they would like to do differently if they could. Time was then allowed for reflection on which things mattered most, and how the ideas were related to each other.
Question 4 (Grey) asked what obstacles and challenges they had, and Question 5 (Green) asked how they might solve them.
Question 6 (Yellow) asked if they wanted to update their success criteria in the light of the discussion, adding any additional goals they felt were appropriate.

Data collection and processing
The Ketso leaves have adhesive strips on the back that allow the whole felt workspace to be folded up and packed away for later transcription, with only minimal movement of the shapes against the background. To assist transcription, photographs of each workspace were taken with a smart phone before packing them away.
The content of each leaf was transcribed into a spreadsheet to capture the raw textual content before preparing a document for analysis, using the standard template from the ketso.com website. This was shared with the participants within 24 hours of the workshop, as a courtesy and for their own use should they wish to. The spreadsheet also captured which felt each idea was from, which branch it appeared on, and what type of leaf had been used.

Artefacts generated
To provide a permanent copy of each workspace, a digital version was created and checked against the photographs, with the exact text and the same relative positions of the leaves on each branch. A copy of this was provided back to the participants for their own use, accompanied by a reminder of the questions they had been asked, in a summary legend sheet. The Portable Document Format (PDF) copy of each workspace was used as the input document for coding and thematic analysis using the NVivo tool.

Thematic analysis
The ideas gathered at the workshops were analysed from the perspective of the framing questions, using the thematic analysis approach suggested by Braun and Clarke (2021). Only the text of the idea was used in the analysis, not information on which felt it came from, or the branch label that participants had applied to it. As such, all leaves were treated equally, and themes allowed to emerge from the text as a whole rather than from any structure imposed by the participants or implied by the guiding questions. The kind of leaf used was not generally taken as significant unless it helped distinguish a goal from a challenge.
Representative labels for the ideas were chosen by in vivo coding from the words used by the participants, or synthesised from the underlying concepts if their words were not sufficiently general, and then relabelled or merged as broader themes emerged from the data. Ranking of the themes, by the number of contributing participant groups and the number of textual references, was used to identify the most prominent ideas for the purposes of consistent presentation and communication. As the data gathered was insufficient to be considered representative of the wider community of practice no other significance should be attributed to the ordering.

Results
A total of 250 Ketso 'leaves' were completed by 12 workshop participants, of which 9 were UX practitioners from a large retail organisation, and 3 were academic staff with an equivalent background in product design or ethnographic research. These provided a total of 74 statements of current practice that participants considered to work well, 61 statements of aspirational practice, 64 statements of challenges to successful discovery, and 51 statements of what constitutes successful discovery.
We asked practitioners for their ideas on how a successful design discovery could be characterised. The analysis identified common terms of reference for challenges, constrained resources, attributes of success, and means of successful discovery. These were then further collected into key aspects of practice needed to meet operational goals in Table  3, organisational aspirations in Table 4, and organisational challenges in Table 5.

Coding examples
Phrases used by the participants were preferred as the initial representative of that idea. These were then progressively merged together under more general codes until the differences between their meanings had sufficient significance to keep them distinct. Examples of the approach to coding are given in Table 2. The codes 'Enabling others' and 'Empowering teams' were grouped with seven others into theme 'Empowering'. Ideas about high quality artefacts to capture learning, research libraries, and sharing insights with other teams were coded as 'Exchanging knowledge', which was grouped with code 'User led' and four others into theme 'Knowledge led'. The 'Empowering' theme was grouped with 'Curiosity' and 'Knowledge led' themes under the top level theme of 'Organisational aspirations, as shown in Table 4. Methods were referred to in general terms, with no specific method being named. A preference for prototypes and experiments "allowing for randomness and unpredictability" was present in both sessions, but more pronounced in the academic setting. The retail organisation emphasised being certain "what to do next" and being able to assess whether to continue or stop. Being data driven by "using data to identify customer problems" and if necessary having "evidence to stop further progress" was as prominent as the use of ethnographic methods for "observation of users in real-world settings". Goals of "having enough time" and making "efficient use of what you have available to you" were taken as a desire to be efficient, and the leaf type was used in that case to distinguish time as a goal from time as a challenge.
A practitioners mindset may affect the efficacy of the design discovery activities. Among the mindsets that were mentioned most were being confident and having "confidence in how to progress", and "thinking laterally" to gain insights, and being inspired so that "there is a buzz around the success of the discovery". An open attitude to "advertising challenges/progress" and "open sharing communication" was recognised as something that worked well, as was a purposeful mindset with an "emphasis on action/doing above all else'" and a "strong process". An engaging mindset "involving others" and "sharing", and a "collaborative mindset" where "the whole team has a shared understanding and has participated" were both identified as things that worked well.
Outcomes that were valued were that the problem should be understood, expressed for example as "the team understand the audience", that this understanding be shared and aligned across the team so that the "team is on the same page regarding outcomes", and that problem be defined and bounded so that they have a "clear scope for the next stage" and validated by "asking the right questions". They wanted a detailed understanding that was "in-depth, not vague", and some expressed a desire for data that could be visualised, for example by "displaying our work within our workspaces".

Aspirational practices
Participants were asked what they would like to do if they were not constrained by the challenges they identified. The aspirational practice themes that emerged are given in Table 4. There was a strong theme of empowerment and autonomy, and both a desire to spend more time with stakeholders but also to "take stakeholder objectives out of the equation". A desire to conduct both a broader and deeper discovery was expressed, to do more "in the wild" work and "have time to explore the whole ecosystem".
As project teams move from periodic software delivery to something closer to a continuous delivery model, there will be a need for discovery activity to integrate with development in ways that better support that, such as the practices described by Torres (2021). One participant expressed an interest in "rolling discovery to explore new areas", suggesting that user researchers might have other reasons for wanting continuous discovery.
The university session included participants with a physical, rather than software, product background. Their aspirations included "trying lots of new technology to consider solutions" and "loads of money and people for prototypes".
No organisational challenge associated with the diversity of user research participants was raised, but an aspiration for "easy access to a diverse audience" suggests that this might sometimes be a problem.

Organisational challenges
The challenges identified were wider ranging, with weaker themes, but prominent among them were internal and external communication issues, constraints on time, wrong mindset, and inefficient processes, as shown in Table 4. Time pressure was associated not just with deadlines, but also having "no time to collaborate". Recruitment of necessary expertise was noted as a problem for understanding complexity and a problem of timing as they could not "recruit fast enough". References to an unhelpful mindset, particularly "solutionising" and "solution-led thinking", were common. Equally prominent were references to inefficient processes related to governance and sign-off.
The solutions that the participants discussed for the challenges given in Table 5 are reflected predominantly in the aspirational practices listed in Table 5, but also to an extent in the key aspects listed in Table 3.

Discovery practices
The research question asked in this study was How do UX practitioners approach and perform discovery?
This was refined in the questions that primed the thematic analysis, and the guiding questions that participants were asked, to three questions:  What is done in practice?  What would improve practice?  What are the challenges?
In answer to these questions we found that methods were referred to in general terms, with no specific method being named. A preference for prototypes and experiments was more pronounced in the academic setting than the retail organisation, which emphasised knowing how to proceed. Practitioners valued understanding that was shared and aligned across the team, a problem that was defined and bounded in scope, and research outputs validated by input from multiple sources. There was preference for data that was detailed, not vague, and which could be visualised. A good discovery was recognised by feelings of inspiration and excitement in the team. The breadth of factors discussed indicates a diverse experience of discovery, and an ad-hoc definition of success with no widely shared criteria within the organisation In the descriptions of the practices they aspired to, there was a strong theme of empowerment and autonomy, and interestingly both a desire to spend more time with stakeholders but also to be less constrained by their objectives. A desire to conduct both a broader and deeper discovery was expressed, which suggests that exercising greater autonomy and achieving the desired 'user-led' process might require a more time efficient approach.
One of the factors most frequently discussed was time pressure. A greater focus on customer value and agility in development (Clarke et al., 2018) implies similar demands for agility in user research and other discovery activities. The mention of inefficient processes in the organisational challenges discussed may also be related to a feeling of insufficient time. Currently, discovery and development are often separate streams of activity such as the dual-track approach described by Cagan (2018). Research is needed to identify the extent to which continuous discovery methods (Torres, 2021) have been adopted in practice and to assess whether they successfully -avoid-sharp peaks in demand.
An interesting omission from the data was vocabulary associated with rigour and challenge. This was missing from both of the sessions, and was not a point of difference between the participants with retail and academic backgrounds. If challenge is not considered an important part of discovery, that might be because it is more strongly associated with later stages of development.

Tailoring the workshop format
The standard Ketso pack assumes up to eight people per workspace, but our experience with the first session suggested this would be too many, so in order to limit the number of people around each table to three or four, for comfort and viewing angle, an additional workspace was purchased. The number of leaves written by the participating design professionals, who were experienced in similar activities if not with Ketso, was sufficient that freedom to arrange them as they wished might have been curtailed if we had not done so.
For a complete cycle of questionsstarting with a definition of done, covering what works or does not, and revisiting our definition of successa period of 90 minutes was barely sufficient to allow proper discussion. If the availability of meeting spaces is limited, the ease with which the felt workspace can be folded and packed up without disturbing the leaves could be exploited to hold a follow-up discussion session at another time or with the workspace mounted vertically on a convenient wall space rather than on a table.
The physical writing and manipulation of the leaves may help the thought process, so it would be interesting to compare the face-to-face use of the physical Ketso materials with a fully digital equivalent on a Miro board or any similar online workshop platform.

Use as a research output
Capturing the raw text of each leaf was relatively quick and easy. Producing a high quality digital version of the workspace for use as a research output was more time consuming. Companies are becoming more interested in building research libraries as part of the operationalisation of user research and design activity, referred to as ResearchOps and DesignOps respectively, as described in Metzler (2020) and Dörnenburg (2018) or Malouf et al. (2019). High quality research outputs was identified in post-workshop discussions as an important part of their practice by the retail organisation that hosted the second workshop, so the relative ease and quality of artefact generation may remain an important consideration regardless of other changes in working practices brought about by greater use of remote working.

Limitations
Limitations of the approach used were that we only ran two workshops with participants from two companies, so the sample size was small. However, as this was an explorative, inductive study that aimed to identify an initial set of thematic categories to characterise aspects of current practice, a small sample was appropriate. The written data collected on the Ketso leaves was brief, and we did not audio record the workshop conversations as we wanted to encourage a relaxed atmosphere. As a result of this a more detailed understanding of some aspects of the written comments was not captured.
Due to limited time, participants were not asked to explicitly rank the relative importance of the factors discussed. The prominence of themes reflects the content of the workshop discussions but taken alone may not capture an accurate picture of working practices.
We asked participants what a 'good' discovery looks like, and captured their ideas about the criteria that should be applied to judge it, but we did not explicitly ask who should make that judgement.

CONCLUSIONS
This study investigated design discovery, the crucial information gathering and sense-making part of the design process, which was under-explored in the academic literature. Using the Ketso workshop format, 12 user researchers and designers from a university and a large retail organisation were invited to share ideas about their design discovery process and its challenges. Our thematic analysis of the workshop outputs showed that practitioners valued an empirical data-led approach, where they could have confidence in the coverage and validity of the data, and achieve a shared understanding of the user research findings across the organisation. Their aspirations for future practice focussed on greater depth and breadth of user research, team empowerment, and knowledge exchange. Key challenges to successful discovery included a lack of clarity in communication, constrained time and materials, and an inappropriate mindset. Research is needed to identify the extent to which continuous discovery methods (Torres, 2021) have been adopted in practice and to assess whether they successfully address these challenges.
The Ketso community engagement method was used in 90 minute workshops, and was found to be an efficient and effective means of asking questions about working methods and sharing ideas within a community of practice. For use with professional designers, groups of no more than three or four people are recommended, as an adaptation to participants skilled in idea generation.