Projection Augmented Relief Models ( PARM ) : Tangible Displays for Geographic Information

Gary Priestnall School of Geography The University of Nottingham, Nottingham NG7 2RD United Kingdom gary.priestnall @nottingham.ac.uk Jeremy Gardiner Ravensbourne 6 Penrose Way London SE10 0EW United Kingdom jeremy.gardiner @rave.ac.uk Jake Durrant Ravensbourne 6 Penrose Way London SE10 0EW United Kingdom jake.durrant @rave.ac.uk James Goulding Horizon Digital Economy Institute, The University of Nottingham, Innovation Park Nottingham NG7 2TU United Kingdom james.goulding @nottingham.ac.uk


INTRODUCTION
A key aim of geographic visualization is to help people develop a greater understanding of spatial and temporal patterns.Recent developments have exploited digital technologies, including the power of the web and mobile devices to reach new audiences, and to explore new applications in support of both research and pedagogy (Dodge, McDerby and Turner, 2008).Alongside this there has been greater awarenesss and use of locational references across many discplines, with such 'spatialization' demonstrating an appreciation of the power of taking a spatiotemporal perspective (Goodchild and Janelle, 2010).The use of Geographical Information Systems (GIS) can help capture, store and manipulate large volumes of spatial data, often using the web for visualization.An example is shown by Cooper and Gregory (2011) with their 'Mapping the English Lake District' initiative, which used geospatial technologies to reconstruct tours made by poets Thomas Gray and Samuel Taylor Coleridge from spatial references found in textual accounts.
Many geographic visualization techniques are designed to present information to individuals via their computer monitor or the printed page, and some of the research challenges here focus on how to convey spatial patterns effectively using a 2dimensional interface.Where the visualization requires some kind of landscape representation to offer a recognisable frame of reference for the viewer, the potrayal of the 'vertical dimension' becomes one of the design challenges.For example the distribution of mountains and valleys may have to be represented either cartographically, or via pseudo 3-dimensional rendering, before other features are superimposed.However, in the context of using geographic visualization as part of a situated display in a gallery, visitor centre or collaboration space, the use of a flat display device is no longer a constraint, and other techniques can be considered.This paper explores the development of a geographic visualization technique where the vertical dimension of landscape is represented literally in the form of a physical relief model, and where the dynamic or interactive element can be provided by projecting map-based data vertically down onto the model, referred to as the Projection Augmented Relief Model (PARM) technique.We discuss the merits of the PARM technique in the context of a gallery setting, and describe how it might offer not only an engaging and clear geographic representation, but may also work to help visitors place materials relating to themes or events on display in the gallery into a spatial context, and through that help make connections between those themes or events.We discuss the broader potential for the technique and describe ongoing work to explore the measurable benefits of using such an approach in terms of people's ability to understand and recall spatial patterns.

BACKGROUND
The use of geographic visualization in a gallery context can aim to place artefacts 'on the map', or to provide reconstructions of past times which can help to offer a broader spatial and temporal context.The map on the wall remains a common technique when attempting to give spatial context to cultural heritage material, often acting as secondary data associated with a particular object.Whilst often effective for locating a single object, or distribution of related objects, such maps can struggle to represent a range of interconnected themes, or to demonstrate relationships between objects in space and time.The use of dynamic maps, featuring animated and interactive elements goes some way to representing multiple map layers and patterns through time, but requires careful cartographic and interaction design to ensure visitors are given a useful frame of reference.
One element of promoting learning within museums has been the provision of engaging, often interactive installations to draw visitors in and to help them understand the information being presented.Such an approach follows a model where visitors are seen as active participants in a process of learning, rather than passive recipients of information (Hein, 1998).The use of 3D visualisation is often seen as useful in engaging visitors, and many different modes of delivery can be considered, including situated Augmented Reality kiosk displays in outdoor situations, and interactive 3D environments projected onto large screens, both of which were explored at the archaeological centre at Ename, Belgium (Pletinckx et al, 2000).Technologies such as Head-Mounted Displays can offer compelling visually immersive experiences in a museum context (Hall et al, 2002) but remain largely experimental in such settings due to complexities of setup and the inevitable emphasis on the single user experience.
In parallel, many museums and visitor attractions have used physical scale models of a landscape, which offer both regional overviews and close examination from multiple angles, unmediated by computer display technology, allowing spatial relationships to be explored in a natural manner, some examples being shown in Figure 1.However whilst often being more engaging than static maps they suffer similar limitations in terms of their ability to represent multiple themes or changes through time.

Figure 1. Examples of physical models in public settings:
Central London present and future, at the Building Centre, London (left); Historical London docks, Science Museum, London (right).
Combining the affordances of dynamic mapping and physical models may be one solution when attempting to create engaging displays that also serve to communicate spatiotemporal patterns and interrelationships.
The development of the PARM stemmed from collaboration between Priestnall (geographer) and Gardiner (artist) as part of a residency on the 'Towards Pervasive Media' project at the University of Nottingham.The residency focussed on a shared interest in landscape representation and digital media, using the English Lake District as the subject.The aim was to explore gallery-based virtual representations of landscape, based upon Gardiner's previous studies of the Jurassic Coast (Gardiner, 2009).Here an immersive virtual environment utilised airborne laser-scanning terrain data (Light Detection And Ranging -LiDAR) for the ground model, with live weather data influencing visual and audible environmental effects implemented using a games engine.Gardiner's paintings and prints informed the content of the virtual space and were illuminated through realtime calculations of how sunlight and ambient light reflect, scatter, and refract through the atmosphere.The combination of the scientific data and human interpretation resulted in an engaging and multidimensional experience of a virtual coastline.
During the residency, visits to the Lake District reemphasised the complex interrelationships between landscape, geology, industry, art, literature and tourism.Also of inspiration was the physical relief model of the area created in 1834 by Joseph Flintoft (see Figure 2), which is housed in the Keswick Museum.The fidelity of the Dorset relief models, in terms of both topographic representation and painted overlay, offered a form of display which demanded close inspection from many angles, but clearly offered only a static representation of landscape.
The desire to explore the more complex interrelationships between themes observed in the Lake District suggested the use of variable overlays, whilst striving to maintain the degree of fidelity created in the Dorset models, and so to the use of vertical projection of digital map data.
The use of projected images to texture physical models of objects is termed the Projection Augmented (PA) model technique, a form of Spatial Augmented Reality (Bimber and Raskar, 2005).Vertical projection onto landscape models has been used within the physical sciences, the prime examples being the Illuminating Clay project (Piper et al, 2002) and follow-on developments (Mitsova et al, 2006), where a small clay relief model could be manipulated by hand, triggering algorithmic changes to water flow which were re-projected down onto the model.Whilst technological novelty is an important factor in holding visitors attention in a science museum context (Sandifer, 2003), complex interaction designs such as those above may not normally be appropriate for public settings.
In order to explore the viability of the PARM approach, a simple prototype had to be developed, with a reasonably sized high fidelity physical landscape model, onto which various geographic data layers could be projected.

THE PARM PROTOTYPE
Traditional techniques for creating 3D landscape models include manually building up layers contour by contour, or various types of 'sand table' as used for the planning of military operations, although the resulting surfaces can lack detail.Experience of displaying the Dorset relief models suggested that the high fidelity in topographic representation made possible through rapid prototyping technologies directly from digital data was important, as was the fact that they were mounted horizontally and could be easily viewed from different angles.
The physical model for the PARM display was to be of a size suitable for a public display, which we considered to be at least 50cm by 50cm, and as the focus was the upper surface given the vertical projection of data then a technique to remove material above the topographic surface was appropriate.This is termed subtractive fabrication, a process where material is taken away or reduced from a solid in order to reveal a new shape.This subtraction can take place using any combination of tooling techniques such as drills, lathes, grinders, and lasers.Such 'subtractive' techniques offer several advantages over additive '3D printers' and other rapid prototyping systems, which are more expensive, fragile and time consuming.Many different materials can be considered, including wood, metal or plastic, but for this prototype a lightweight high-density foam board was chosen.
After experimentation with a range of techniques a Computer Numerically Controlled (CNC) milling machine was chosen (specifically a Roland MDX 540) which was well suited to the type of surface model being created.
The source data for the terrain surface was a gridbased Digital Surface Model (DSM) of the English Lake District derived from airborne radar (Figure 4, left) with a ground resolution of 5m.This was cropped using ArcGIS software and exported as a simple text file.The Mayka Expert software was used to convert the grid of heights into a series of drill paths to drive the milling process.After experimentation a model size of 80cm x 80cm was chosen, which required 6 separate tiles to be produced (Figure 4, centre).A vertical exaggeration of 20% was applied to the elevations, informed by the results of a series of trial runs.Each model section took around 6 hours to mill using two passes, the second pass requiring a finer tool bit.
The tiles were joined and mounted on board, forming the base of the prototype PARM.A metal rig was constructed to support a data projector above the model, the average throw of projectors requiring this to be around 2.2m above the model surface.For some display spaces this meant the PARM display had to be placed on the floor (Figure 4, right), although from the outset the intention was to be able to have the model at desk height to allow people a more oblique view onto the terrain surface.
The purpose of the prototype was to help us explore how people interacted with the display, what they did or said, and what they wanted to do with the system.It was also an opportunity to test the system, and to develop design ideas through discussion between the research team and those viewing the PARM display.In many ways this shared characteristics with a technology probe, as used, for example, by Hutchinson et al. (2003).
Here simple graphical communications technologies were deployed with real families as part of a cooperative design process.One difference is that the context for deployment of the PARM prototype was not a specific real world situation, but was to be a series of demonstrations to members of the public and museum curators.As such the content for projection could not focus on a set of themes that would be directly relevant to one gallery context for example.The approach taken was to develop a series of projections that demonstrated different types of overlay, both static and animated, showing a variety of places, routes, events, and a series of alterative graphical styles.Through giving viewers a flavour of what was possible, it was hoped it would promote discussion around their own particular interests.
The content for projection comprised: Alternative backdrops.These covered the whole area and included: satellite imagery, aerial photography, geological mapping, conventional 1:50k scale mapping, historical mapping from 1900 and hillshade imagery derived from the data used to build the physical model (as shown in Figure 4, left).Specific features of interest.These highlighted particular places, routes or areas of interest in a variety of ways and included: animation of the route taken by Edward Lear in 1836 on his painting tour of the Lake District, a series of key stopping points and the areas visible from those points (viewsheds, derived from the terrain data in ArcGIS), and animated sequences showing the advance and retreat of the ice sheet which covered the area over 20,000 years ago.
The content above was sequenced together and could be run as a continuous loop, or played one stage after another.The milling of one tile (centre); The prototype PARM system (right).mmm

OBSERVATIONS
The first demonstration of the prototype was at a community open day event called Mayfest, held at the University of Nottingham in May 2011 (Figure 5).Further demonstrations and discussions were held with museum curators and collections managers from the Keswick Museum, the Wordsworth Trust in Grasmere, Cumbria, and the Imperial War Museum in London.The interests of people viewing the model varied enormously, from casual observers with no knowledge of the Lake District, to curators working with cultural heritage material related to that area.

Figure 5. The PARM prototype: A selection of data layers used for projection (upper left); Demonstrating at the Mayfest community event, University of Nottingham, 2011 (upper right and bottom).
Through direct observation and discussion the following findings emerged: General impact.Almost without exception viewers experienced an engaging 'holographic' effect, provided the room was dimly lit.There was a general fascination with the raised relief effect, but no comments specifically on the level of exaggeration applied to the terrain.One viewer commented on the relative height difference between two of the major lakes in the centre of the model, which is quite subtle and less noticeable on 3D visualizations such as Google Earth.
Viewing position.People often viewed the model from many different angles and distances.When the model was placed on the floor, many viewers crouched down to obtain a more oblique view point, or commented that it was effective from the more distant first view they had experienced, reinforcing the need for tabletop mounting.
Interaction.Many people pointed features out at close range, and traced out familiar routes they had taken when visiting the actual landscape.On some occasions viewers touched the actual surface but more often than not had to be prompted to do so.For many the passive sequence of projected layers was engaging enough, but on occasions people wanted to revisit a previous sequence or step to a particular sequence when attempting to describe something to another person.
Design of projection content.Projecting the hillshade image emphasised terrain detail.The satellite imagery was particularly effective and much more so than the aerial photography.This is likely to be due to the way the satellite data is processed to give more exaggerated colouration for different land cover types.The map was found useful although lacking the clarity necessary on many occasions.Not surprisingly people's attention was held when something specific was being pointed out, and the animated routes and key view points were particularly effective.There were many occasions where people asked for more information about what was being projected.Curators identified specific artefacts or themes that were on display in the gallery to which projected content could relate.
These observations were taken into account when designing further developments and refinements to the PARM system.The aim was not only to develop PARM to a point where it could form an installation in a gallery space, but to provide a platform for further research into the measurable benefits of this method of geographic visualization.

REFINING THE PARM SYSTEM
As a result of the findings from prototype demonstrations the following specifications were defined for the next iteration of the PARM system: • Basic interaction was required and a touch screen interface was chosen.• High Definition projection was needed to add clarity to display.• Secondary monitor providing information about projected layers was required.• Map layers and the portrayal of features of interest need careful design and testing, for specific use with the physical model.• Models should use more robust material to withstand public display.
• Models need to be mounted at tabletop height rather floor level.
The configuration of the refined PARM system is shown in Figure 6.A vital component of the PARM design is the software to allow the multiple screens to work together and to give flexibility to manipulate the portrayal of projected content.The software to support the PARM configuration shown in Figure 6 was written in HTML5 for the following reasons: • HTML5 is built around a model of user interaction, and its ability to integrate navigational facilitates with content presentation is well suited for exhibits.
• It provides a rapid prototyping environment; essential for ongoing experimentation and to configure future exhibits.• Extensive multimedia support including integrated audio and video tags.• A long history and an abundance of development tools.• Rapidly developing animation frameworks such as paper.jsand processing.js.• Ability to quickly switch CSS styles allows rapid rebranding of presentation styles.• Simple mechanism to communicate between different presentation windows.
• All platforms now have browsers that support HTML5.This means that the system can be easily extended in the future, adopting mobile devices such as smartphones and tablets if necessary.
The next step was to create a gallery-based installation, providing an opportunity to develop site-specific content based upon the requirements of a heritage organisation, and for longer-term observation of the visitor experience.

THE GALLERY-BASED PARM
A gallery-based PARM has been developed for the Wordsworth Trust gallery, which exhibits a range of archival material related to William Wordsworth (1770-1850), including connections to other poets, artists and importantly the landscape itself.This builds on previous collaborative work in designing a gallery-based virtual tour to accompany an exhibition of work by Edward Lear (Priestnall and Cowton, 2009).After detailed discussion at the Trust a focus for the installation became 'Spots of Time', key events in Wordsworth's childhood that had connections with specific places in the landscape but which also related to poetry created in adulthood, notably 'The Prelude'.Four key spots of time were identified which related to material on display elsewhere in the gallery.A location was identified in the gallery (a mock-up is shown in Figure 7) central to the physical artefacts, where the location of displays showing physical artefacts related to the spots of time is shown in green.

A RESEARCH AGENDA FOR PARM
In addition to being a highly engaging form of display, we believe there is a strong research agenda related to the effectiveness of the PARM technique as a form of geographic visualization, both as a mechanism for placing objects and events into their spatial context in a gallery or visitor centre, but also more generally as a tool for communicating spatiotemporal patterns by offering an intuitive frame of reference.

The role of PARM in the gallery
The placement of the PARM system in a gallery or visitor centre in the context of physical artefacts and other displays, presents an opportunity to study how it might influence peoples movements around the space, in addition to its effect as a standalone installation.Wineman and Peponis (2010) describe the importance of spatial layout and movements of visitors through that space in shaping the nature and extent of informal learning in a museum environment.An important goal when evaluating the PARM display over a longer period of time in a gallery setting will be to assess its role as a central hub in the space, potentially influencing how visitors move between displays and how they learn from exhibits as well as what they actually learn.Beginning with the 'Spots of Time' installation we are making detailed observations of peoples movements around the space as well as their interactions with the PARM system, scoping the measurable benefits of such interactions in relation to the particular goals of that exhibition space.The aim is to contribute to the study of how people extract meaning from situated geographic displays and to consider the implications for their design, which resonates with the broader call by Dodge et al. (2009) for more research into how maps actually function in certain everyday settings.
Careful attention needs to be paid to the design of the media used in the display, both in terms of its effectiveness as standalone graphics or audio, but also how it connects people to related media and artefacts around the gallery.A version of the PARM system features an authoring environment to allow rapid experimentation with the portrayal of projected content for a particular situation, including new forms of backdrop mapping designed specifically for use with physical 3D models.
Whilst the design of media could help people make connections to other displays, there are possibilities for exploring various forms of embedded tags, perhaps using mobile devices as the linking mechanism.We are also considering mechanisms for delivering locative media mementos via the display in some circumstances where the gallery is situated within the actual landscape in question.

Interaction design
Currently users are able to interact with PARM via a touch screen that forms part of the system's overall display ecology.However, in keeping with the physicality inherent in the PARM display, we plan to explore more tangible means of interaction, including gesture recognition via technology such as Microsoft's Kinect framework.Such approaches are attractive given the challenges posed by implementing a direct touch interface on the model surface, and the occlusion of projections this would cause.We intend to explore the experiences that are made possible when users are able to swipe, drag and point above the PARM model.We are particularly excited about the potential to extend this interaction to multiple-users interacting with the model simultaneously and potentially through different forms of user-input.Our work in this area will be informed by research on chained displays (Koppel et al. 2012), multi-person display ecosystems (Terrenghi et al. 2009), and multiparticipant behaviour around shared displays (Marshall et al. 2008).

Spatial knowledge acquisition
Many of the proposed benefits of using PARM in the gallery context stem from the belief that it offers an intuitive display of landscape, which removes much of the cognitive load associated with viewers constructing a mental picture of a landscape to provide their frame of reference.In terms of promoting more effective spatial knowledge acquisition clearly there are many contexts where the PARM technique could be of value beyond the gallery, and we are undertaking controlled studies to explore measurable benefits, initially within the domain of pedestrian navigation (May et al, 2003).
In particular, we are testing the hypothesis that delivering navigational instructions using a physical 3D interface, in comparison to other forms of geographic representation, improves performance in route traversal time, confidence, path accuracy and post-navigation recall of both the route and the broader survey knowledge of how features relate in space.These studies are using large-scale models from airborne laser scanning data where buildings and trees are represented in addition to the ground.
We aim to isolate the specific issue of how people orientate themselves within a PARM-based landscape model, drawing influence from wellfocussed experimental psychology methodologies.This will support the development of PARM displays for a range of collaborative planning activities as well as justifying the approach in communicating spatiotemporal patterns in the geosciences, both as a research tool (linked to GIS in some cases) and in a public context.

CONCLUSIONS
We believe that the Projection Augmented Relief Model (PARM) technique offers an engaging form of situated geographical display due in part to the high fidelity of representation in both surface model and projected overlays.When used in a gallery or visitor centre context it offers the ability to place artefacts into their spatial context but also to explore how connections between different themes covered elsewhere in the gallery space can be promoted using the PARM display as an integrating hub.Our agenda for ongoing research also includes an exploration of interaction design both for individual queries and as part of a multi-user environment.More generally there are exciting opportunities to isolate the measurable benefits of using physical 3D models, combined with projected digital mapping of various kinds, for communicating patterns over space and time as compared to other forms of geographic visualization.

Figure 2 .
Figure 2. Relief model of the Lake District by Joseph Flintoft, 1834, Keswick Museum, Cumbria.Gardiner's ongoing experiments with rapid prototyping technologies had involved the use of additive fabrication systems to forge new forms utilising digital design systems and manufacturing processes.Of most relevance were the landscape relief prints of the Dorset coast using LiDAR elevation data.The models, measuring around 30 cm in diameter, were carefully hand painted to replicate the detail found in aerial photography, an example (St.Aldhelm's Head) is shown in Figure 3.

Figure 3 .
Figure 3.An example of a hand painted landscape model derived from airborne laser scanning data.

Figure 4 .
Figure 4. Creating the PARM prototype: Hillshade view of the DSM of Cumbria (left);The milling of one tile (centre); The prototype PARM system (right).

Figure 7 .
Figure 7.A generalised re-construction of the gallery space showing the 'Spots of Time' PARM display (July 2012 onwards), related physical artefacts in green.