Creation of an Integrated Multisensory Wearable Story: Costume embedded haptics-VR using a considered eco-ethical design approach

In this paper, Burge will summarise the journey of iterative technological and creative development to produce a multi-sensory story called ‘GhostNet’ developed as part of the WearSustain network to create sustainable wearable technology (EU Horizon 2020 grant No 732098). GhostNet is an interactive artistic wearable experience, which raises the awareness of ocean plastic pollution. GhostNet refers to the problems caused by plastic fishing net, lost or discarded, so-called ‘ghost gear and ghost nets’. This multi-component system has been developed with ethical considerations at every step of the design process (including circular design methodology, supplier, material and process choice) without compromising on creative or technological innovation. The dystopian interactive underwater VR scene is animated in Unity with visualisation of scientific data (Imperial College) is integrated with in-house crafted pneumatic haptics which inflate and according to the VR in the scene and via the controller. Couture style jackets with prints created in VR hold the electronics and have been produced by immigrant charity Heba using organic cotton and seaweed fabric. The costume aspect is central to the experiential narrative to create enclothed cognition providing a strong mental imagery adding to altering self-perception (Adam & Galinsky 2012) and the experience. Burge will reflect on the pros and cons of using this approach on sustainably designed user-centred emergent technology and its impact on multisensory story-telling. This was Burge’s fourth costume-spectacle wearable VR-haptic experience developed as a continuation from experiential themes around costume embedded wearables as a reference for ‘transformation and exploration of the pathways to narrative and ludic (playful) experiences (Isbister & Abe 2015).


INTRODUCTION
With GhostNet, BB Studio reflected on learnings from previous in-house VR-haptics projects to develop a more sustainable and complex innovative framework with an objective to create an engaging multisensory story to raise awareness of ocean plastic pollution.
In contrast to BB Studio's previous work, this was a wireless interactive VR experience integrating untethered headsets of 6 degrees of freedom with improved proprietary inflatable wearable haptics developed using a circular design methodology.

VR EXPERIENCE AND ARTWORK
The sci-fi dystopia VR animation, created in Unity, abstractly describes ocean plastics based on extensive research about issues from team member Hunt, audio proof from the Marine Conservation Society, and data visualization as particles in VR directly from 'Plastic Drift' simulating the probability of ocean plastic pollution. The visual style was driven by in house textile print within the constraints of untethered android headset capability, with an added processing complexity of multiplayer and haptic interactivity and animation tracks.
The constraints led BB Studio to research optimised and generative 3D assets in particular cinder medusae squid alongside Burge's painted Google Blocks assets, which were developed into fabric prints to create a visual association across the different mediums. A haptic tentacle animation appeared in the scene and was also crafted as a real silicon inflatable tentacle made by Cohrt.
Inevitably, with every addition of asset or complexity, the compiled scene on the headset would become suboptimal and glitch or output unpredictable errors, so there as a constant editing and deleting with development. Consequently, the VR story was iteratively developed with animation tracks structured around the audio script. The headset controller was coded to trigger increase and decrease of the ghost gear animation in VR alongside the haptics inflating and deflating.

GARMENT
A critical part of the multisensory story has been the design and development of the 2 outer garments. The original inspiration came from polygonal style of VR combined with origami-esque and sculptural kimono silhouettes of the previous BB Studio VR-haptic-costume story, The Great Wave, which in itself was inspired by an art residency Burge completed in Ikisan, Japan in 2016. Evaluation from the previous BB Studio projects showed that the costume wearable was a reference for 'transformation and exploration of the pathways to narrative and ludic (playful) (Isbister & Abe 2015) experience and people wanted:  The garment to be literally represented and abstracted in the VR story;  To see touch both virtually and in real life through the garment (Burge 2018).
The role of the wearer's outer garment was deliberately captivating providing a performative experience for onlookers was explored, leading to a more structured garment design complete with reflective tape creating a shape of a ghostly net.

Figure 2: GhostNet Garments with VR assets as textile prints developed by Burge on organic cotton
and seaweed fabric produced partly by Heba Women's Charity. Photo: Burge.
Burge painted VR assets in Google blocks, which were developed into textile prints and digitally printed onto sustainable fabrics such as organic cotton and seaweed fabric and the garments were part-produced by immigrant charity Heba. An inner garment to hold electronics and tubing and haptics were made out of reclaimed fabrics. This componentization allowed for easier reuse and repair in line with circular design methodologies.

HAPTICS
BB Studio team experimented with materials, surface design and types of construction for inflation and deflation as well as testing different effects achieved through inflation (triggered by bespoke electronics through flexible hosing), placement on the body and size and shape variations. With a lack of industry haptic vocabulary to refer to, BB Studio created an enhanced loose narrative to connect it to a meaningful event within the VR (Israr et al. 2014).
The bespoke electronics were created from reclaimed blood pressure machines with extra booster elements for increased pumping and additional electronics for messaging from the VR controller.
Haptic type 1 was created by PVC cured (not stitched) into different shapes referencing soft robotics with an outer sleeve to prevent air breakages and allow for fabric manipulation and different textures which would create its own haptic sensation. Bespoke connectors were 3d printed to the connect air pipes. PVC as a material is not sustainable and the pouches were restrictive in shape-construction always requiring an outer sleeve.
Haptic type 2 was created to 'See touch', to show movement and a more visually engaging haptic, Cohrt with a background in soft robotics and wearables crafted different coloured tentacles using silicon. The silicon tentacle haptics were highly performative, unique with a biomorphic quality, which made an immediate impact during demonstrations. The silicon itself felt quite heavy and had to be adjusted for best placement.
After multiple uses at demos, there were issues of air breakages with both haptic types. For future developments we would add air pressure sensors or code a maximum point with the Unity in the VR scene to prevent future damage.

ELECTRONICS, INTEGRATION, ARCHITECTURE
From the outset, Burge wanted to create a rich visual story, which interacted with BB Studio electronics in an untethered headset. The platform used was daydream on an android operating system with a Bluetooth paired controller. Having worked with Bluetooth, we researched within our network for a more robust solution and were recommend the WiFi-protocol XIO-board to send messages to the pump to inflate the haptics. The board also held other parameters such as accelerometer output so one-story development was having a VR body, which moved independently to the VR headset accelerometer. The board connected through a hotspot on an in-house app. Additionally the app was developed to be able to test and override the controller.
For multiplayer, a Unity package called Photon Room was used, which allowed players to enter interact in an internet VR room. For efficiency, the players would effectively spawn and appear as the connection happened through the Wi-Fi (either through mobile network of the phone or a normal Wi-Fi). The multiplayer aspect was a challenge during public demos, sometimes because of firewall issues and sometimes because of crowded bandwidth. BB Studio tried some workarounds like a dedicated modem or networked phone but it would be still be unstable.
The final architecture of GhostNet has been revised to be much simpler, and currently shown as a single player experience.

SUSTAINABILITY
The IDEO and Ellen MacArthur Foundation 'Circular Design Guide' had been referenced throughout the project particularly in choice of materials (reclaimed, reused, organic and seaweed fabrics (Sea-cell, biodegradable), componentization wherever possible; and choice of processing such as digital printing which uses less water wasting and polluting than traditional printing methods. Bio plastics such as hemp were investigated in the production of electronics casings and will be used in surface design of future iterations. BB Studio had partnered with HEBA women's charity to produce a large portion of the garment sewing (the most technically advanced aspects have been carried out in house by fashion designer).
Sourcing sustainable electronic components proved challenging for small quantities. Excess factory stock and discontinued lines were a potential source for reclaimed components and also buy from second hand outlets such as eBay.
Other options were to source electronics made by reputable suppliers with associated accreditations and standards.
Another aspect was activism through the story itself which was to raise awareness of a sustainability issue and data was visualised from public datasets (downloaded to CSV) shown to us by Dr. Erik Van Sebille, Faculty of Natural Sciences, The Grantham Institute for Climate Change who told us about http://plasticadrift.org/ which simulates the probability of plastic pollution in the sea at a particular longitude and latitude.

CHALLENGES
The sheer complexity of each component of the experience from couture style garments to multiplayer caused unpredictable issues. For instance, production of the garment required revisions because of quality problems with the production house. Another example was that the soft-robotic silicone tentacles would contort appropriately in test but when demoed would burst, and over time, the PVC air pouches developed holes. BB Studio is currently developing solutions such as including with pressure sensors and more robust expandable materials.
There were also issues regarding the lack of interoperability between the app and the latest boards across the multiplayer systems. The boards required unique IP addresses. Furthermore, the system worked well in testing but in a public event environment with lots of people the multiplayer Wi-Fi systems became unstable. It has resulted in reducing the demos to single player as the multiplayer versions are further trouble-shooted and developed. The biggest challenge was constructing a cohesive story-experience from so many bespoke components created by a semi-remote team. The BB Studio team attempted to keep synergies through shared virtual channels with Burge driving creative the vision.

CONCLUSION
With GhostNet, the BB studio team incorporated their previous findings (Burge 2018) and developed areas of innovation and creativity under the mentorship of V. Frascolla (Intel, Director Research and Innovation) and Dr. R. Hunt (researcher in sustainability) with team members Dr. H. Boys (researcher in smart materials for enhanced human computer interaction), A Baker (creative Unity developer), F. Akmal (Wearables Consultant) and E. Cohrt (design futurist). This resulted in an immersive, multisensory storytelling experience using a fully integrated and aesthetically striking wearable technology garment, and Virtual Reality (VR) using sustainable design methodologies where possible. This project allowed BB Studio to build a considerable experience (Table 2) and a range of solutions, including electronics, sensors, actuators, haptic elements, communications, VR assets, wearables and garments which they are recombining and adapting to other experiences and stories.
BB Studio demoed the experience at different stages of development at a number of events such as ICT Austria, December 2018 and Imperial Lates, February 2019 as well being shortlisted for '2019 SXSW Alternate to Finalists at the Style & Wearable Tech in the Interactive Innovation category' and awarded the Worth Grant (EU) for a future VR-craft experience.