Bring Something to the Potluck: A System for Inclusive and Reciprocal Online Discussion

We present Potluck, a structured online commenting system that aims to promote the inclusion and reflection of differing views. It features a discussion flow that (1) elicits independent viewpoints through shielded discussion (2) collates and displays divergent viewpoints using automatic summarisation-aggregation and (3) encourages reflection through recursive question-and-answer. This paper describes work in progress, outlining the design motivations, features and user flow of a working prototype. A preliminary user study has shown encouraging results on the usability of the proposed system.


INTRODUCTION
Online commenting systems -from standalone web forums, to embedded tools under media and blogs -enable public discussions with an unprecedented number of participants. These platforms have the potential to present multiple viewpoints on the same topic, and to lead to complex forms of groupundertaking such as large-scale deliberation (Wright and Street 2007).
However, their potential is undermined by longstanding design issues. These systems have users interact with what is typically an unstructured list of disjointed comments. As the list grows, the format suffers from "information overload" in which users are unable to consider all of the comments (Jones et al. 2004). To help make sense of the comments, platforms enable comment re-ordering by time and rating -but this gives less exposure to (and effectively excludes) the comments in the middle of the list. These design issues provide little incentive to write well or courteously. As a consequence, online commenting systems are often subject to low-quality and uncivil comments (Chen 2017).
Related work have gone beyond the list format to better display online comments, using techniques such as dimensionality reduction to visualise opinions as points in a vector space (Faridani et al. 2010;Kim et al. 2021), and topic modelling to identify and group related comments (Dave et al. 2004;Hoque and Carenini 2016). However, these systems are mainly concerned with the visualisation and exploration of comments, rather than the discussion. To enable more constructive discussion, systems such as the Deliberatorium (Klein 2007) have scaffolded the discussion flow, but these solutions are focused on argumentation more so than general discussion.
We sought to design an online commenting system that enables more constructive and inclusive discussion. We propose the discussion flow for online commenting systems shown in figure 1, designed to (1) elicit independent viewpoints through shielded discussion (2) collate and display differing viewpoints using automatic summarisation-aggregation and (3) encourage reflection through recursive question-and-answer. Potluck's answer summarisation-aggregation mechanism. When a user submits an answer a, it is automatically summarised to 2-3 sentences (a.s). Summary a.s is compared to all existing summaries in the database. If a semantically-similar summary exists (b.s), a.s is discarded and a is assigned b.s. If a summary like a.s does not exist, then a is assigned a.s. This discussion flow is demonstrated in a working prototype of Potluck 1 , a novel online commenting system. It is named after potluck-style gatherings where each participant must contribute something to be shared by everyone. In other words, Potluck requires users to actively participate in discussion. We describe a user study of Potluck to assess the usability of the proposed discussion flow.

DESIGN RATIONALE
Before describing the system, we explain the motivations behind Potluck's key design features.

Pseudonymity and active participation
The distribution of participation on social platforms reveals that a small fraction of users make up most of the contributions (Shirky 2008). To include more voices in the discussion, users will need to be encouraged to participate. Pseudonymity provides a middle-ground between the civility associated with real-name identities and the self-disclosure afforded by anonymity (Rowe 2015;Graf et al. 2017;Moore et al. 2021). Potluck has users register with a system-generated username before they can start to engage with the system. To maintain the privacy of anonymous participation, the system does not ask for any identifying information.
To increase inclusion and diversity of contribution, a first-time user of Potluck is only shown a discussion prompt: a question or topic. To advance through to the discussion area, the user must actively participate by contributing a written answer to the prompt. Further answers are required in the same way for the user to advance through the discussions.

Shielded discussion
Reading the existing comments before posting can influence how users proceed. Incivility or lack of representation in the comments could potentially 1 potluck.cim.rhul.ac.uk inhibit participation, causing users to self-censor and fragment off to other systems (Walther and Jang 2012;Springer et al. 2015). To retain users, the user must post before they can read the answers already submitted to Potluck. This design ensures that a user's answer is not influenced by the answers submitted by others. This complements the feature of active participation in section 2.1 by giving the user another incentive to participate.

Summarisation and aggregation
Users cannot see what answers have already been submitted to a discussion prompt. Since each answer is submitted without knowing what has already been said, the same information may be repeated. Without a way to organise the answers, the discussion area would suffer from the problem of redundant and disjointed comments found in existing systems. Potluck synthesises the information by automatically summarising and grouping similar answers as depicted in figure 2. Each summary is displayed in the discussion area, thus presenting the user with a quick overview of unique viewpoints.

Recursive question-and-answer
Question-asking is a listening tool that can be used to deepen understanding and solve problems (Murphy 2020). It can also be employed as a communication tool for argumentation and rhetoric (Gowdy 2020). Potluck allows users to interact with each other by recursively asking and answering questions -users cannot advance through the system without doing so. By asking a question, users create a new lower-level discussion area within the system. This was implemented with the goal of adding structure to the system, while encouraging deliberative norms such as reflection, ideal turntaking and reciprocity between users.

Question trail
After several rounds of recursive question-andanswer, the user may get lost within the system. To address this concern, we implemented a "question trail" (figure 3c) stickied to the top of each page: a navigation feature that displays the prompt for the current top-level discussion area and the prompts for lower-level discussion areas (if any) in a tree. The links indicate whether the user has previously answered the discussion prompt. From the question trail, the user can jump back into the discussion area where they already have access, or contribute an answer to a locked discussion prompt to gain access to a new area. The user can also contribute a new answer to a question they have already answered before.

Usage
Potluck has two types of users: Hosts and Guests. Both Hosts and Guests can partake in the system as outlined in section 3.1.1 but Hosts have additional capabilities as described in section 3.1.2.

Guest user
View top-level questions. When a user registers or logs in, they are directed to the home page (figure 3a) where links to top-level discussion prompts are displayed in descending chronological order.
Answer questions. Selecting a discussion prompt from the home page directs the user to the answer input page (figure 3b) where they must pseudonymously contribute an answer to access the discussion. When the user submits their answer to the discussion prompt, the system processes it with summarisation-aggregation as per figure 2. The answer is automatically screened for toxicity -"rude, disrespectful, or unreasonable comment that is likely to make someone leave the discussion (Perspective API n.d.)" -as described in section 3.2. If the answer's toxicity is above a certain threshold, the answer is posted but hidden behind a "mask": an opaque layer that requires users to tap to reveal the answer ( figure 4).
View summaries. After the system assigns a summary to the submitted answer, the user is directed to the summaries page ( figure 4). This page displays cards for each of the summaries associated with the discussion prompt. Each summary has two buttons: an answer count button and a question count button. The answer count button indicates how many answers are under that summary. Tapping the answer count button takes the user to the full answers page (figure 5) where the user can read and interact with each answer under that summary. The question count button displays the number of questions asked on the answers under  If the user finds an answer constructive to discussion, they can "respect" it by tapping the handshake button, as implemented in previous work (Faridani et al. 2010).
Only the representative answer (the first answer in the grid of cards) is used to create the summary but this is subject to change to reflect the most respected answer. For example, if answer A receives more respect votes than the current representative answer B, then A swaps places with B to become the representative answer. The summary is regenerated using A and a  notification is sent out to the author of A to commend them.
The thumbs-up button can be used if the user agrees with a comment. Inappropriate and non-constructive answers can be acknowledged with the flag button. If an answer is flagged above a threshold number of times, it is automatically hidden. The respect, agreement and flag counts of an answer are not displayed to the user. In Potluck, they are used as internal metrics to determine what answers should be displayed, and in what order.

Ask questions. Tapping the question mark button
under a full answer reveals an input area that allows the user to enter a question to the author of the answer (figure 5). Once the user submits their question, the system semantically compares it to the other questions asked on the answer, as explained in section 3.2. If it is too similar to an already-submitted question, the question is not added and the user is shown a link to the existing question. If a similar question has not been asked on the answer, the question is appended to the bottom of the answer (figure 5) and the author of the answer is notified that a new question has been asked. The system then creates a lower-level discussion area from the new question. Tapping a question under an answer directs the user to the answer input page (figure 3b).

Host user
Set top-level discussions. The user can create toplevel discussion prompts which are displayed on the home page (figure 3a). The user can pin prompts that they wish to prioritise or archive prompts that are no longer relevant.
Override moderation. The user can hide Guest user-submitted questions and mask answers that are not conducive to discussion. The user can also override the decisions of the auto-or crowd-sourced moderation and reveal answers that were hidden.

Implementation
To build a reproducible prototype of Potluck, we chose technologies that are freely available and require minimal set up. Potluck is a Python with Flask web application. The Transformers library 2 is used with a pre-trained DistilBART model 3 to generate abstractive summaries. To compare the semantic similarity between new and existing user input (such as summaries and questions), Potluck converts the input into sentence embeddings using the SentenceTransformers library (Reimers and Gurevych 2019) with the pre-trained all-MiniLM-L6-v2 model 4 and then measures the cosine similarity between the embeddings to determine whether aggregation is needed. The auto-moderation of user input is powered by Perspective 5 , a machine learning API that returns an estimated toxicity score for textual inputs.

USER STUDY
We had the opportunity to have alumni of the World Health Organisation infodemic management training programme (WHO IMTP) trial Potluck remotely for 10 days. The main aim of this study was to understand the usability of Potluck in a real-world setting.

Participants
Nine participants were recruited from a rolling invitation to alumni of the WHO IMTP through WhatsApp groups. Two of the participants did not complete the post-study survey by the deadline so their survey results have been omitted from analysis. The remaining seven participants (45-54 median age group; four men, three women) are labelled P1 to P7 in the findings.

Procedure and tasks
Before the study, participants were emailed a video on how to use Potluck. At the start of the study, all participants were granted Guest roles in a shared private instance of Potluck, initialised with four top-level discussion prompts and seed answers. The second author was assigned the Host role.
Participants were required to spend at least five minutes a day using Potluck for the first five days of the study. After the fifth day, this requirement was removed. Post-study, participants provided longform text responses and 5-point Likert responses in a 10-minute survey about their demographic and experience using Potluck.

Usability
In the post-study survey, participants rated the easeof-use, usefulness and engagement of Potluck's features, and the ease-of-use and comprehensibility of the system overall.
The results from feature evaluations suggest that the usage of Potluck is feasible ( figure 6). In particular, the summaries screen was rated highly for usefulness and it was perceived as easy to use and engaging ( figure 6). Five out of seven participants agreed that the summaries presented on Potluck were relevant (figure 7). These results show that Potluck can provide an overview of small to mediumsized discussions, motivating further studies with larger discussions.
The question trail was the most divisive in perceived usefulness ( figure 6). In the future, we should provide clearer instructions on the question trail within the system or update the feature with a more familiar look-and-feel, such as a sitemap.
Five out of seven participants agreed that Potluck was overall easy to use (figure 7). P6 said: "It is very simple to use and everyone having basic knowledge and IT can use it." However, only three out of seven participants agreed that the system was overall easy to understand (figure 7). P2 said: "I can see what it is trying to do, but overall, I found it a bit confusing" but also noted that they did not think they had enough time to evaluate it. This suggests that future deployments should be longer than ten days so that participants have sufficient time to understand the system.

Overall impressions
Participants provided their impressions of Potluck by rating statements and writing long-form responses in the post-study survey.
Only three out seven participants agreed that Potluck presented diverse viewpoints, with the same participants agreeing that the system presented viewpoints that they had not previously considered before (figure 7). The participants were professionally homogenous and from these results, we can infer that they likely had the same ideas on the discussed topics.  Nevertheless, participants recognised the potential of Potluck to expose differing opinions. P4 said: "Very useful idea which needs more field testing in the intended environment, eg. covering topics where people disagree the most." Overall, six out of seven participants reported a positive experience on Potluck. Notably, the system was found to be informative (figure 7). P6 described the system as "very user friendly, educative and informative" and P5 said that it was a "good tool to navigate." Participants agreed or were neutral that Potluck was suitable for online discussion (figure 7).

LIMITATIONS AND FUTURE WORK
Potluck restricts the exchange between participants to the asking and answering of questions, making it unintuitive to communicate certain types of information. Solutions on how to maintain structure and reciprocity while allowing different communicative forms will be investigated in future work iterations.
The summaries on Potluck are currently ordered chronologically. The number of answers with unique viewpoints (and thus unique summaries) were expected to be small in the preliminary study. But as this number grows, the summaries will need to be organised to display salient viewpoints to the user. We will implement a weighting system in future work.
The summarisation and similarity detection are done without human intervention. Although convenient, an automatic summary could be suboptimal compared to a human summary. The similarity detection might not detect all similar cases or mistakenly group dissimilar cases. These concerns could be addressed in future work by allowing users to review and fix the automatic summaries, and move answers to a better-suited summary.
Due to the asynchronicity of the system and the small group size of participants, we found it difficult to sustain and observe engagement. Moreover, the participants expressed similar views on the discussed topics. In future studies, we will recruit larger heterogenous groups and select topical or controversial discussion prompts. This will enable us to observe how Potluck handles the display of many differing opinions.

CONCLUSION
We designed and implemented Potluck, a structured online commenting system that aims to encourage the inclusion and reflection of differing viewpoints. It features a discussion flow that requires users to actively answer and ask questions to advance through the system. Although the usage of Potluck is more restrictive than existing commenting systems, our preliminary user study shows that it is feasible for small to medium sized groups. In future work, Potluck will be evaluated with larger heterogenous groups.