Designing an augmented reality exhibition: Leonardo's Impossible Machines

This paper discusses the origins, development and results of the animated and augmented reality aspects of the exhibition ‘Leonardo’s Impossible Machines’ that was developed at Ravensbourne University London and Birkbeck, University of London, with support from the Museo Galileo. The exhibition included novel reconstructions and visualisations of Leonardo’s perpetual motion machines from the Codex Forster, and the process is explained here, along with the challenges of mounting a combined physical and AR show.


INTRODUCTION
This paper is a record of the process of designing and developing augmented reality and animated content for an exhibition, "Visualising Perpetual Motion Machines", which opened in The Peltz Gallery, Birkbeck University of London, 6th February to 12th March 2019. Marking 500 years since Leonardo's death in 1519, this free exhibition explored new ways of analysing, reconstructing and contextualising his designs. As part of the international celebrations marking the 500th anniversary of Leonardo's death in 2019 the prototype of this installation was then incorporated into a larger exhibition at the Museo Galileo in Florence (October 2019-January 2020).
The exhibition involved specialists from three main areas: academic consultants and curators Dr Juliana Barone, Honorary Research Fellow at Birkbeck; and Andrea Bernardoni from the Museo Galileo in Florence, funders of the project; museum and heritage practitioners from the Leonardo Da Vinci Society, and specialists in 2D, 3D and Augmented Reality techniques from Ravensbourne University London. The project benefitted from the support of Dr Joel McKim, Director of the Vasari Research Centre for Art & Technology. Each stage of the visual process was agreed upon in consultation with academic and curatorial consultants, in particular, In May 2017 the Research team at Ravensbourne was invited to create installations to celebrate two of Leonardo's Perpetual motion studies. The team included experienced animator Mike Smith, Nick Lambert as project coordinator and developer of a 3D virtual archive of Leonardo's drawings, and Jazz Rasool with an extensive background in developing applications for the Microsoft Hololens, our preferred augmented reality interface.
The studies chosen for the augmented reality experience were Folio 91 and 92, on the front and back of the same paper (verso and recto), within the Codex Forster II, part of the collection of three codices of Leonardo's drawings, and bequeathed to the Victoria and Albert museum by John Forster. The second installation request, f.1062r, is a manuscript by Leonardo from the Codex Atlanticus, which is now in the Ambrosiana library in Milan.
Research by Bernardoni and Barone revealed the development of perpetual motion theories across three continents. Originating in India in the 7th century, the theme of perpetual motion arrived in Europe via manuscripts transcribed in Toledo in the mid-13th century; (Bernardoni, A 2019 p 14). Designs of overbalanced wheels are found in the notebooks of European engineers from the 14th century onwards. However, the only medieval manuscript that has been passed down to us from this period is by the architect and engineer, Villard de Honnecourt. The Codex Atlanticus, f. 1062r-v includes a re interpretation of De Honnecourt's design. It clearly indicates Leonardo was familiar with this earlier design and that he considered it worthy of further analysis and reinterpretation. However, in itself this does not establish whether Leonardo believed this or other earlier designs had the potential to work. In fact evidence suggests he was of the opinion all shared a general lack of prior empirical knowledge. This is supported by his comments in the 1490's; "Among the superfluous and impossible delusions of man there is the search for continuous motion, called by some the perpetual motion wheel"; (Bernardoni, A 2019 p 24). All the more understandable then that he did not bother to build any of his designs.
Nonetheless, Leonardo maintained an enduring interest in perpetual motion. He continued to work on possible solutions while employed at the Sforza court in Milan between 1481 and 1499, as well as for some of his time, spent between 1500 and 1508, in Florence. In total Leonardo spent twenty years of his life pursuing perpetual motion solutions. However, the evidence suggests that Leonardo did not consider perpetual motion possible. Therefore he must have had other incentives justify this extraordinary commitment of time and resources. One possible reason might be the period of history Leonardo lived. It has been suggested the renaissance triggered a "cultural revolution", when technical knowledge became most valued". (Bernardoni, A 2019 p 19). It was also a pivotal point in scientific rigor and professionalism. For the first time engineers began to write scientific treatise. Perhaps Leonardo used perpetual motion as a vehicle to explore and test his overall understanding of physics and engineering.

INITIAL DEVELOPMENT AND ENQUIRY
The following questions informed our research framework. Firstly we asked, in the long tradition of realising Leonardo's "thought experiments", and in the light of contemporary knowledge, how can any further visual adaptations, add to their understanding?
As an example of this tradition are the models of Leonardo's mechanical drawings built by Giuseppe Schneider in 1929, that possess a tactile, real world immediacy. By contrast, a filmic, televisual, or virtual alternative might; "make everything visible but nothing accessible" and as Heidegger would suggest contribute to; "the worlding of the world as picture" (Causey, M. 2006 p 49).
With this in mind how more effectively might new technologies inform the visitor of Leonardo's analogical thinking and design process? Are they appropriate? A digital representation might not fully convey the tactile nature of a process such as the casting of Leonardo's bronze equestrian monument. However, it excels when representing the scale and complexity of this subject, demonstrating temperature change, movement and the consistency of liquid metals.
As curator of the exhibition, Juliana Barone agreed to the suggestion we attempt to identify an interactive solution for the Forster Verso and Recto designs and create a screen based solution for the Atlanticus manuscript. However, our initial findings suggested virtual reality could deliver a solitary and isolating experience. An example of a VR learning experience in a similar exhibition was installed at the Modigliani exhibition at the Tate in 2017; https://www.tate.org.uk/whats-on/tatemodern/exhibition/modigliani/modigliani-vr-ochreatelier Whereas at MoMar's inaugural show in 2017 we see disruptive AR content that responds to specific elements within the museum space, providing a shared and collective experience for all the visitors.
Based on this research the consensus of the three collaborative institutions was that we should design a fully inclusive, situated, augmented reality experience. AR allowed us to offer the viewer a virtual, collective experience while still within the reality of the exhibition. At the same time the viewer could potentially study Leonardo's animating drawings from any angle or distance. It had the potential to revolutionise the museum experience, be inclusive and make learning fun and meaningful. In this respect it perfectly aligned with our ethical framework.
Having established some of the reasons why Leonardo continued his perpetual motion studies, whilst believing that perpetual motion was impossible, the team considered whether he evaluated how and why his own designs did not work. Reviewing these designs in the context of contemporary physics and engineering, we considered whether it was possible to construct a plausible working model of his designs. Working with the historical scholars, the team extrapolated from the 2D drawing in the Codex Forster a series of three-lobed turbine blades revolving inside a cylinder, with two balls moving around as illustrated by Leonardo. Of course, they also function in the idealised space of a computer animation where forces such as friction and gravity can be altered to suit the needs of the visualisation. In this way, perpetual motion can be made to work. We then set about planning how to situate this augmented experience within the Peltz Gallery exhibition space. The Forster perpetual motion designs were drawn by Leonardo on the front and back of one page, Recto and Verso. To celebrate this tactile anomaly we set about planning how to encourage the visitor to walk around the page and view both designs from many angles.
This suggested experimenting with a free standing object, like a portable and reusable exhibition system, that could be placed anywhere in the room. It offered the potential to playfully use the available space around itself, to allow the audience to move round it and appreciate the augmented experience from many different angles. It involved accommodating the requirements of the curatorial team and then exploring how best to choreograph visitors around the available space in the Peltz. Below are images demonstrating initial concepts for this choreography and the final outcome.

PROGRAMME
Once the proof of concept was approved by our stakeholders, we began the 2D animations and the 3D modelling of Leonardo's designs. Completed by Tomas Koza, a recent Ravensbourne Animation graduate, these models were built using the strict specifications supplied by our academic consultants. We made a series of assumptions about the geometry and placement of these turbine blades, as they needed to contain curved channels for the balls to run in. As can be seen in the models below, the 3D blades had rounded corners compared to the sharp corners of the original drawing, in order to enable free running of the balls. Koza and Jazz Rasool both highlighted the need for some changes so that the turbines would work as planned. This revealed a disagreement between the academic consultants, designers and animators. The underlying question being continually asked was, at what speed should the turbine rotate, and what speed accurately depicted Leonardo's intention? Despite appreciating the design could never work the consultants insisted on a certain process based on the order of representation on Leonardo's drawings. However, when animated using real world gravity the balls would not reach the speed to make them pass through the point "M" as indicated by Leonardo (Fig. 10). Instead of "M" being at 90 degrees, real world gravity indicated the balls would hit the edge of the turbine at 120 degrees (Fig.  11).. This suggested that in order for Leonardo's design to work, irrespective of the interaction of the balls and the turbine, an acceleration of energy had to take place to have the ball reach the point "M" where Leonardo wished it to be. In fact they run from N to M, but not in that exact position and because the balls cannot be in the places their required to be by Leonardo, they can never produce the energy to achieve the perpetual motion he speculates. Despite not being able to see this in motion Leonardo understood, for this very reason that in principle, this design did not work. His extraordinary mind achieved this without the assistance of the tools, knowledge and calculators we now have at our disposal.
Further testing revealed additional intriguing anomalies. As an example; despite using gravity simulators to animate the balls in the turbines and changing the speed of the turbine rotation, the balls would not form the pattern illustrated in the Forster II manuscript. Eventually, this was achieved by stretching the clips of the real world gravity animated layers until they all finally aligned to the secondary circle. These inconsistencies, defying the principles of physics and engineering, underscored the temporal nature of these designs and that they should be understood more as thought experiments than working designs.
These production tests ultimately broke the deadlock that had arisen. By finding a route that looked plausible from an animation point of view we could build the object in 3D so that it produced Leonardo's pattern of movement. Now the object could be properly reconstructed, incorporating many of the consultant's arguments without breaking any of Leonardo's ideas.

NARRATIVE AND SCRIPT
The template for both moving image pieces rested on the writing of a robust, lucid script. This became the defining framework on which the narrative content. The scripts were provided by our academic consultants, Barone and Bernardoni. From these an overhead narration was recorded. It was used a guide to more accurately time out the content. Any updates became iterative versions until the storyboards could then be signed off and an animatic produced. This cycle of creative development continually informed the decision making and allowed all parties to discuss and approve each stage.
With this structure in place work started on planning and animating the 2D representation of how Leonardo sketched the Forster verso design. Continual tests and iterations were closely informed and reviewed by our academic consultants. With the 2D look established animation commenced. Once the rotational speed of the turbine was agreed the 3D models could then be animated to align with the 2D animation. These animations were then sent to our AR specialist Jazz Rasool for the augmented reality experience. Rasool used the open platform developed for WEKIT to inform the structure of the Hololens experience for Leonardo's Impossible Machines. This involved importing the 2D and 3D animations into Unity and syncing the timings with the narrated script and music. The animations had to fit the perceived dimensions of the physical prints from the Codex Forster, and the Hololens's camera had to recognise the images in the correct sequence (recto then verso) so the experience would unfold in the correct order. Once tested in Unity, the app could then be exported to the Hololens. Real-world conditions required much work to overcome variations in light and image quality. The sensors in the Hololens (derived from the Kinect camera) had some problems with recognising symmetrical images, so had to be trained to see the two Leonardo drawings in different settings. The correlation between the print and the resulting 3D image had to be carefully managed.

AUGMENTED REALITY DEVELOPMENT
Although a working version was ready some months before the exhibition's launch in February 2019, one unforeseen issue emerged shortly before the opening day. A reproduction of the Codex Forster and several other rare folios were loaned by the British Library for the exhibition, and these required reduced lighting in the gallery for conservation purposes. At normal light levels, the Hololens had performed well, but the dimness caused problems.
Following various experiments, Rasool resolved the issue by instructing the camera to look for a 2D image captured from the darker room instead of trying to see the original print, as before. After that the app was successful.
Another concern was the usability and security of the expensive Hololens hardware in a gallery environment. So long as trained assistants were available, the Hololens could be utilised properly and kept secure. However it was too complex for untrained visitors to operate, and too fragile to leave on display. Therefore a version of the Unity app was developed for Android smartphone and tablets, with an iOS version also possible. The tablets had noticeably better recognition response and visitors with compatible mobile phones were encouraged to download the app for their devices. The tablets were tethered to the stand that housed the two Codex Forster prints, along with printed instructions for their use. Although visitors who had the opportunity to try both devices generally preferred the immersive experience of the Hololens, the tablet had the advantage of being able to share the 3D visualisation with groups of visitors. Some people also cavilled at the prospect of wearing a headset and preferred the tablet for that reason. In three later iterations of the exhibition (at the Victoria and Albert Museum and at the International Broadcasting Conference in Amsterdam, both in September 2019, and at the Ravensbourne Degree Show), similar responses were noted. Ultimately the tablets were more accessible due to user familiarity with the devices, whilst the Hololens was a more exotic piece of hardware that provided an absorbing approach to Leonardo's concepts.

EXHIBITION DESIGN AND CONTEXT
The augmented reality exhibit fitted into a multimedia space that also included the reproduction Leonardo codices, a physical Timeline that connected Leonardo's paintings and engineering designs, and a large print of the Codex Atlanticus drawings arranged sequentially. Other explanatory wall panels detailed the evolution of Leonardo's thoughts on perpetual motion. There was also a glass vitrine containing 3D prints of the Codex Atlanticus and Codex Forster machines derived from the 3D models created by the Ravensbourne team.
Mike Smith also produced a 2D animation about the Codex Atlanticus manuscript that explored the operation of the varied devices Leonardo invented to investigate perpetual motion. A screen-based animation allowed a greater freedom to engage with a multitude of filmic solutions to demonstrate the operation of these machines. The quality of the image, its texturing, lighting, cinematography and rendering, could be significantly controlled and finessed. Near the end of the process Alexander Collinson, a 3 rd year student on Ravensbourne's Sound and Music BA course joined the team to compose copyright free sound effects and music. These had to be appropriate to the subject and in keeping with the needs of every user in and around the public space of the exhibition. Mike Smith suggested a minimalistic approach similar to Philip Glass to effectively compliment the repetitive rotational cycles of Leonardo's turbines and establish a contemplative soundscape, suitable for this shared, public space. This resulted in an harmonious score that embodied the cyclical rotation.
A further exhibit was also developed in Unity: a 3D virtual gallery showing Leonardo's prints relating to perpetual motion along with those of his contemporaries, and other Leonardo drawings and artworks that were relevant to the discussion. This was made by Nick Lambert using the metaphor of a first-person 3D game and presented on a desktop computer, although it has subsequently been compiled for the Microsoft Mixed Reality headset. This builds on Lambert's previous work to create Imaginearchive, a 3D archival space; and prior research on 3D libraries and associated knowledge environments that invites interactive engagement with 3D knowledge assets.  Further evaluation of the visitor experience was gathered from the visitor book. The overwhelming number of comments were very positive. As an example, one visitor commented: "A great little exhibition. The reconstructions and visuals in motion, with their separations of Leonardo's drawings, really clarified his intentions".

CONCLUSION AND OUTCOMES
Coinciding with the Peltz exhibition, a symposium entitled, "Holograms in the Museum"; was held on 1st March 2019, at Birkbeck school of Arts. This interdisciplinary discussion brought together creatives and cultural studies scholars to consider the changing significance of holograms in artistic and cultural practice.
On 12th March 2019, Ravensbourne Research Office organised a symposium at Ravensbourne University to present the projects findings. The symposium aimed to evaluate how effectively our methodology had informed the final work.
The animation forms the nucleus of a larger exhibition at the Museo Galileo in Florence (October 2019 -January 2020). Along with the Virtual Archive and AR exhibit, it was restaged at the "Digital Weekend" that took place at the Victoria and Albert Museum, September 21st-22nd 2019.
Discussions are currently underway with Google Arts and Culture to have elements of the exhibition displayed on their globally recognised site.