Did That Stone Move? Staging stone swarms in galleries and virtual reality

Elena M. Vella Daniel A. Williams Aleks Michalewicz Airlie Chapman Robert Ellis Walton The University of Melbourne, Australia The University of Melbourne, Australia The University of Melbourne, Australia The University of Melbourne, Australia The University of Melbourne, Australia evella@student.unim eld.edu.au williamsd4@student.u nimelb.edu.au aleksm@unimelb.edu .au airlie.chapman@unim elb.edu.au robert.walton@unimel b.edu.au


INTRODUCTION
To change the world, we need to combine ancient wisdom with new technologies.

-Paulo Coelho, Warrior of the Light: A Manual
In this paper we explore relationships between human audiences, ancient stonecraft and modern technology by examining interactions with groups of robots ('swarms') disguised as monoliths in virtual reality simulations ('VR') and in actual reality ('in real life' or 'IRL'). Multimedia performance installation Sacrifice requires the audience to share a gallery space with a swarm of robotic ground vehicles disguised as replica standing stones from a network of globally significant archaeological sites. We anticipate that as the stone swarm moves autonomously through space, the audience's curiosity and engagement with the performance will be stimulated.
Initial development efforts have focused on VR as a performance medium. This has enabled the tuning of key parameters of the robotic swarm's behaviours and physical appearance in order to inform the IRL exhibit. As we transition from virtual simulation to physical realisation we find ourselves in a paradoxical mixed reality of ancient, immobile stone technology strangely animated by computer controlled, semi-autonomous robots that exposes and reframes long-held appreciation of the vibrant autonomy of stony materialities in human life. Responding in part to metaphysics and objectoriented ontologies, and works on speculative realism as developed by Shaviro, we consider the perspective of the robotic stones (Shaviro 2014). In environments where humans interact with objects outside their own experience, both beliefs and physical states influence successful collaborations (Setter et al. 2016) and the new media and new material dramaturgies of performance (Eckersall et al. 2017). It is our aim to better understand how humans collaborate with stone swarms in different performance environments, both in VR and IRL.

ROBOTIC SWARM INTERACTIONS WITH HUMANS
Robotic swarms are groups of robots that collaborate to achieve a common goal. Coordinating agents' movements using discrete logic and communication protocols (such as mesh networks), a robotic swarm can act in multiple locations simultaneously. As the capabilities of a single robot multiply with the size of the swarm, this property of robotic swarms may be advantageous for tasks involving human-robot interactions, e.g. search and rescue or caregiving (Bartneck et al. 2004, Kozima et al. 2005, Sabanovic et al. 2006). Fundamental to the success of such missions are notions of trust, reliability, safety, and intelligibility.
In this light, an art exhibit provides a valuable opportunity to develop robotic swarm systems that interact with humans. Through this work we aim to temporarily unite human participants with the swarm in performance, ultimately to better understand how a robotic swarm's behaviours influence participant states of [dis]engagement.  Challenges arise when integrating robotics into a performance installation due to temporal, financial, and computational constraints associated with prototyping and refining the setup (Afzal et al. 2020). Furthermore, these challenges exponentially grow when scaling from a single robot to a swarm of robots (Sahin et al. 2008). To sidestep some of these challenges, we have developed a novel VR environment for prototyping human-robotic swarm interactions for the performance. The key components involved are portrayed in Figure 1.
As depicted in Figure 2, the stages of building and refining the performance have been primarily conducted in VR to date. The continually evolving VR application has provided an immersive environment in which the team can test and observe the effects of changing certain swarm design parameters; at each iteration of development the team has been able to concretely visualise proposals for the performance, overcoming interdisciplinary communication barriers. In addition, the VR environment has enabled the collection of data regarding human behaviours and preferences. This has been used by our artists to train the robotic swarm and learn from its surroundings. We envision that the swarm will progressively learn how to interact with live audiences, adapting its behaviours to stimulate human engagement.

Figure 4: Robotic stone prototypes (miniatures).
We shall now shift our focus from a swarm of robots to a swarm of standing stones found in archaeological sites from across the world. We have worked together with cultural custodians to develop photogrammetric models of the stones. Integral to the process of exchange, we have sought to engage with the custodians' cultural heritage, specifically the representative digital data, with care and to protect the sovereignty of these materials. Cultural considerations have included what it means to create photogrammetric models of physical stones and to transport these from their original contexts, degrees of fidelity in replicating stones, and the implications of creating a new type of site. The intentional aggregation of heterogeneous stones moving sympathetically with human participants places the latter at the heart of an evolving process of intercultural exchange. In doing so, we anticipate that the experience will provoke contemplation about the common elements between humanity's relationships with ancient and modern technologies: robotic swarms and ancient standing stones.
Having concentrated on a purely VR-based performance experience initially, we have begun to transition to a mixed reality experience (Benford & Giannachi 2011). Following the workflow summarised in Figure 3, our team is currently prototyping the designs of the standing stone carapaces and tuning the physical robots' controllers accordingly; differences in virtual development are highlighted in blue. At the same time, we are continually refining the VR experience to keep it closely aligned with the forthcoming IRL gallery experience. The common structure of the VR and IRL simulation systems (depicted in Figures 1 and 4 respectively) has allowed us to rapidly transfer behaviours implemented in VR to the robotic platforms with a degree of confidence. By implementing parallel VR and IRL experiences for the gallery space, we will be able to experiment with the volume and format of information to convey to human participants. This will result in a multilayered and multivocal IRL and VR experience for audiences. Our ultimate goal is to use these observations to yield insights into how human trust can evolve in response to changes in the swarm experience.

CONCLUSIONS
Preliminary findings from the development of Sacrifice provide evidence for a greater crossover between swarm robotics, art and mixed reality experiences. Using modern photogrammetric techniques we have sought to document and aggregate elements of cultural heritage from multiple archaeological sites globally. In addition, we have underlined the importance of extending the use of robot swarms in performances for the general public. In turn, this prompts a wider discussion about humans collaborating with robotic technologies in performance both in heightened arts settings and more mundane quotidian encounters in society. We continue to develop new ways of interpreting and structuring human-robot interactions in VR and IRL settings. We anticipate that Sacrifice will evolve further as the gallery performance takes shape, allowing us to explore the expressive potential of hybrid mixed reality experiences.