TESTING THE LIVING CANVAS ON STAGE

The Living Canvas initiative aims to use a performer on stage as a dynamic projection surface. Ultimately, the system aims to create the illusion of a completely dynamic costume. The collaborative work brings together the disciplines of computer graphics with the performing arts. This paper briefly introduces the concept and presents the results from test-bedding the technology during a performance workshop. The initiative focuses on the challenges of exploiting a completely new kind of projection technology in an artistically meaningful way, avoiding the pitfall of just introducing a technology gadget without actively supporting the narrative of the performance.


INTRODUCTION
The Living Canvas initiative aims to use a performer on stage as a dynamic projection surface.Using machine vision in the near-infrared spectrum enables the digital projection system to follow and adapt to the performer, particularly avoiding background spill by detecting the silhouette and masking the video stream accordingly.Ultimately, the initiative aims to create the illusion of a completely dynamic costume created and controlled in real-time by a video artist.The collaborative work brings together the disciplines of real-time computer graphics and machine vision with the artistic vision of Cryptic within the performing arts.It aims to create and explore a new expressive medium by adding interactivity to the projection on stage.
The Living Canvas is an ongoing initiative that has received initial funding from the UK Arts & Humanities Research Council (AHRC) to conduct a feasibility study during summer and autumn 2007.This first study generated a prototype implementation of the core computing elements for restricting a projected video stream to within the silhouette of a performer on stage.While the primary goal of this study was to establish the technical feasibility and limits, the technology has proven to be robust enough to start evaluating the artistic potential and to identify issues for deployment on stage.
This paper briefly introduces the concept and presents the results from test-bedding the technology during a performance workshop involving artistic director, multimedia technician, stage manager and a visual artist.It focuses on the challenges of exploiting a completely new kind of projection technology in an artistically meaningful way, avoiding the pitfall of just introducing a technology gadget without actively supporting the narrative of the performance.

RELATED WORK
Video projection has become ubiquitous in contemporary multi-media culture.Its applications range from projected backdrops or moving scenery [1], dynamic lighting [2] to performances where performers directly interact with the video [3].Increasingly, artists are working with real-time captures of the performers on stage [4], letting performer interact with video.Projection onto the performers has been explored previously [5,6].
The latter projects required the performer to strictly follow the choreography as the video can not react to deviations from the originally designed positions and poses.Allowing the projection to spill onto the floor or a background (preferably dark) would relax those requirements, but would impose limits for stage design and the video artist.The "Jew of Malta" production [7] introduced a much more flexible projection system using similar machine vision techniques as the Living Canvas, the latter however trying to push the idea significantly further by exploiting the latest technologies, in particular by adding further pose recognition later in the project.
Cameras track performers in projects such as "Sensuous Geographies" by Alistair MacDonald, allowing them to interact with the projection or sound during the performance.These projects are not overly demanding in terms of latency or precision, as they show the visuals reacting to the performance, whereas for the Living Canvas, projection and performer has to become one and any visible delay would detract from that unity.
None of these previous projects go as far as detecting the exact posture and silhouette of the performer.However, motion capturing has been used successfully in projects such as "Quartet" [8] to drive real-time visuals.Performers wear a range of sensor or markers for tracking, but the system does not acquire the silhouette that would enable precise restriction of the projection.
Technology-wise, the Living Canvas is built upon a wealth of previous research, optimizing existing machine vision algorithms [9] to work as fast as possible and borrowing ideas from projector-based augmented reality projects [10] as well as motion tracking.

TECHNOLOGY
The Living Canvas prototype system implements a pipeline starting with image acquisition using a near-infrared camera, processing the images to identify the performer against the background in real-time, and generating a masked video to be projected.The video or texture to be projected is generated using a stand-alone tool to be controlled by a video artist.

Image acquisition and segmentation
The initial stages of the processing pipeline acquire and segment a camera image of the performer on stage.A high-speed digital greyscale camera is used to acquire the initial image.The camera operates in the near-infrared spectrum that is invisible to the human eye; the visible spectrum is blocked by an IR-pass filter.Lighting the stage with infrared illuminators enables the system to "see" even in complete darkness, while at the same time being totally independent from other projection systems that might be used on the stage concurrently.
The acquired image is then segmented into foreground and background to extract the silhouette of the performer.The segmentation algorithm is a variant of the widely used background subtraction idea where the current image is compared against a previously learned empty background [11].Image regions differing from the original background are labelled as "foreground".Segmentation parameters can be modified in order to optimise the result, balancing susceptibility to noise against reliably detecting all parts of the performer and adapting to the lighting system.
A final post-processing step consisting of morphological image operators is applied to remove noise and holes in the mask.This step also allows expanding or shrinking the silhouette, giving the artist additional creative control over the system.

Warping and projection
The segmentation mask is then applied to the incoming video stream to black out those parts where projection would spill onto the background.As the camera and the projector have neither the same origin nor identical lens configurations, a warping step is required to match the camera image into the projection space.The prototype system uses a 2D correspondence established during a calibration step.Such a 2D method inherently introduces a mismatch error as it is only correct at the calibrated distance.

Video generator
The content to be projected onto the performer is created through a separate, standalone video generator tool.The video generator tool sends the video frames to the projection system through a network link.Decoupling the two systems has two major advantages.While the machine vision is implemented using very efficient but proprietary low-level code, the video generator is implemented using Max/MSP/Jitter [12], a tool commonly used by video artists, therefore enabling immediate reuse of the vast amount of existing knowledge and video processing tools.It also enables running the two systems on separate machines, providing independent processing resources for the time critical components.

Speed
Any delay between the movement of the performer until the projection changes accordingly results in visible errors, either in the form of spill onto the background ("shadows") or not fully covering the performer.The total system latency from camera acquisition until projection therefore defines the maximum speed of motion of the performer before artefacts become disturbing.
While the vision processing software is able to fully keep up with the 200 Hz input rate of the camera used, the inherent delay introduced by the various hardware buffers and transmission delays particularly between the graphics hardware and the digital projector results in approximately 30 to 40ms latency in the prototype.

ANALYSIS
The Living Canvas prototype as described above was tested and explored in a range of settings in order to identify the strengths, weaknesses and limits of the idea.From an engineering point of view, the questions were focused on the analysis of the performance and precision.While it became evident early on that the technical goals and expectations are met by the system, the attention shifted towards exploring the potential and analysing how the performance parameters might limit the artist.

System errors and limitations
The current prototype implementation has a range of errors and limitations, some of them inherent in the approach or hardware used, others due to the early stage of the implementation.
Built using standard hardware components, there are fundamental limitations to the achievable speed.System latency is dominated by the buffering required by the graphics card and projector and could only be improved significantly through a direct integration of the vision processing pipeline into the projector itself, requiring a custom hardware design.The latency of 30 to 40ms means that for an object moving at 1 m/s, projection lags by 3 to 4cm.This speed is readily exceeded by individual limbs even during a casual walk.Fast movements typical for dance performances are therefore clearly beyond the system capability.
The high-speed camera used has a resolution of 640 by 480 pixels, which defines the spatial precision of the mask.If a single camera was to cover a stage width of 5m, this would result in a resolution just below 1 cm -roughly the width of a finger.Higher resolution cameras are available, but they tend to be either very expensive and/or significantly slower.If resolution becomes an issue, it is more practical to deploy several independent systems covering separate parts of the stage.
The calibration between camera and projector is currently implemented as a 2D correspondence, a gross simplification for a 3D problem.The magnitude of the resulting error, however, is dependent on the ratio between the distance between the camera and projector origins, and the distance between the camera and the performer.The actual error can therefore be minimized by placing the camera close to the projector lens.

Handling the errors
The sum of the calibration and system delay errors results in "shadows" that are cast onto the stage backdrop.Depending on the background, these can be nearly invisible to the audience, or they can show up rather obvious.While the system development efforts concentrated on minimising such errors, the artist actually became quite enamoured to them and considered the shadows as an attractive design element.During the workshop, we therefore spent some time actively exploring the system errors and quirks.While changing the extent of the silhouette enabled increased shadow effects, future versions of the software will include additional controls to introduce system latency to have the projection follow the performer in time, as well as geometry operators applying offsets, rotations and scale to further detach the projection in a controlled manner.Although system development will continue trying to achieve maximum precision, it became evident that systematic errors -once understood and controlled -can be just as interesting as absolute correctness.Figure 2 shows a selection of such happy accidents.

Costumes and fabrics
Successful projection onto the performer requires a suitable costume.A white fabric without specular highlights (shininess) and few folds is required for an even projection.We successfully tested a matt white lycra suit as a near-optimal surface.However, the artist would typically choose different fabrics or colours to achieve a certain effect and style, departing from an ideal projection in the technical sense.
The segmentation of the camera image relies on the costume reflecting the near infrared spectrum differently from the background.The costume must be reflective as we are dealing mostly with front lit stages (see below).We found this to be the case with most brightly coloured fabrics, particularly artificial ones.Overall, we found that the IR reflectance property requirements on the costume were quite easily met and are unlikely to cause problems.The costume selection is therefore mostly an artistic decision.
Lighting -front vs. back Running at 200 Hz, the digital camera has a rather short exposure time of 5ms.Additionally, the camera sensor is optimized to work in the visible spectrum and is therefore less sensitive to near-infrared light.Consequently, providing sufficient lighting is critical for a high quality camera image that is required for reliable segmentation.
Originally, the system was developed with an assumption that IR lighting would be provided exclusively by special IR illuminators as used for night-vision in CCTV installations.The initial series of lab tests were conducted with a pair of 75W illuminators.Under controlled lighting circumstances, there were two fundamental approaches to ensure high quality segmentation: • Only the background of the stage is lit (as in [13]), resulting in a dark silhouette of the performer against a bright background.With this setup, care has to be taken to avoid any IR light spill onto the performer, which is easy to control if the performer is reasonably far away from the background and a sufficient number of illuminators are available.
• The stage including the performer is lit from the front (similar to [14]).The performer typically shows up bright against a dark background, assuming that the stage back is not overly reflective and the performer is significantly closer to the illuminator, exploiting the light fall-off with distance.This method enables distinguishing between parts of the performer to some extent by varying the reflectivity of the costume.
The first method provided superior results when tested in the lab with relatively cramped space conditions.The main advantage lies in the total independence of the silhouette from the actual costume, resulting in sharp edges regardless of material or folds.Unfortunately, back-lighting is not practical for real theatre productions due to the interference from stage lights as elaborated in the next section.
Lighting the stage from the front has proven to be more practical.Although stage backgrounds tend to reflect IR, particularly projection screens, the performer shows up brighter than the background due to light fall-off.The segmentation breaks down only if the performer moves too close to the background or is wearing a non-reflective costume, both of which can be avoided.Segmentation quality can be further improved by placing the illuminators towards the side, avoiding direct spill onto the background.
Both methods above assume that the performer is either consistently darker or brighter than the background, even though the segmentation algorithm could detect changes in either direction from the previously learned image.However, such more generalised segmentation works best on colour images where not just brightness, but also changes in hue are detected.On greyscale images, however, it has proven to be significantly more reliable and less susceptible to misclassification caused by shadows to make the assumptions above, given that the lighting situation is known and controlled on a stage.

Lighting -stage lights
Lighting is a crucial and indispensable design element on stage.Incandescent stage lights, however, easily overpower any digital projection system, reducing its contrast up to the point where it almost disappears.Projectors with extremely high brightness exist, but they are often impractical due to cost considerations.In the traditional stage setting where projection is limited to the background the problem is addressed through separation: ensuring that no stage light spills onto the projection screen keeps the projection's contrast mostly intact even if the performer is brightly lit.
Projecting directly onto the performer renders this separation less practical.Clearly, for a system such as Living Canvas, lighting the performer must be kept to a minimum to retain contrast of the projection system.In our tests we had good success using side spots with coloured gels to set the overall lighting mood, but use the projector to superimpose high-contrast texture.Preserving saturated colours, however, would require a significantly stronger projector than the 3000 ANSI Lumens projector we had available.Figure 3 illustrates a range of lighting conditions.

Lighting -infrared
While the contrast-reducing effect of stage lights onto the projection is immediately visible, the incandescent lamps also generate a large amount of energy in the nearinfrared spectrum that is picked up by the camera system.Even modest stage lighting greatly overpowered the dedicated IR illuminators in our experiments.While we had not foreseen such a strong influence in the original system design, we found that this effect could be used to our advantage: Dedicated IR illuminators are either based on LED technology or incandescent bulbs with an IR-pass filter.The former type has a typical energy peak around 880nm or 940nm wavelengths, whereas the latter is much lower at 740nm -the visible light spectrum ends around 700nm.The camera sensor is significantly more sensitive at shorter wavelengths, closer to the visible spectrum; hence using the incandescent-type illuminators is more effective.
All our filter gels for stage lights let pass the near-infrared spectrum.Using a black filter gel on a stage lamp turns out to be the same fundamental principle as a specialised IR illuminator, except that the latter is optimised for long-term usage requiring an efficient cooling system and durable filter, whereas filter gels burn out after just few hours.For our application, replacing the gels after each show, is still orders of magnitude cheaper than buying the CCTV illuminators required for generating the equivalent amount of IR energy.
Using the stage lights for IR illumination reduces the cost and complexity.However, changing the lighting conditions during a performance means that the segmentation system has to cope with dynamic light conditions, whereas traditional backgroundsubtraction algorithms assume a static background.In our tests we positioned the lights such that there was minimal spill onto the background by placing them to the sides of the stage with a fairly narrow spot cone, only illuminating the performer.This setup worked reliably for a range of lamp intensities without requiring recalibration.Based on these experiments, we assume that the segmentation system can be used as is for a range of typical stage illumination situations.However, we will focus future development efforts towards developing a system that can cope with completely dynamic stage lighting by receiving control information from the digital light control systems (DMX) and adapting background statistics accordingly.

ARTISTIC CONSIDERATIONS
The Living Canvas initiative is a major project in progress.Some significant technological milestones have been met, providing confidence about the technical feasibility of the ultimate goal.The major challenge, however, lies in the artistic exploitation of the technology.The week-long exploratory workshop has provided some initial insight into the potential and limitations, but a lot of conceptual development remains to be done.It is crucial that the audience perceives the system as integral part of the narrative, not as a flashy add-on.Technology should be transparent and not attract any attention to itself.We consider the system successful if members of the audience remain fully immersed into the story, and only after the performance -if at all -start wondering how it actually worked.
Despite our enthusiasm for the potential of the technology, it is unlikely that we would employ the projection system for an extended amount of time during a performance.Instead, we see it as a critical intervention at selected moments within the narrative, where the virtual meets the real.Examples might be a dream sequence, a moment of insanity, or the meeting of a real human with a fantasy character.
Given the limitations of current hardware, it is unlikely that the system will approach perfection within the lifetime of the initiative.The artist is therefore forced to accept and work with the inherent errors, particularly the system delay.But this can be as much a limitation, prohibiting fast movements, as it can be an opportunity, such as a character trying to run away from its own "shadow".
Even though we formulated the idea of the virtual costume as the starting point of the initiative, it is not "reality" we are trying to achieve.We feel the system is much stronger at evoking the uncanny than trying to fool the audience.This steers the further development of the system away from absolute precision towards integration and control of effects.Precision is still required, as it is a pre-condition for tight artistic control over the effects.While we warmly welcome happy accidents during the development of the idea, there is no room for uncertainty and random errors in the final performance.

OUTLOOK
Development on the Living Canvas initiative is ongoing, with a current focus on fundraising.Cryptic plans to incorporate the technology into their 2010 production will go on tour later the same year.This will not only generate the necessary drive to mature the technology, but also provide a highly relevant case study for further academic reflection on this exciting new visual language.

Figure 2 .
Figure 2. Errors and accidents: Misclassification of the eyes leaving dark holes.Deliberately expanded silhouette resulting in "shadow" around the performer.Input freeze leaving a "ghost" on the background

Figure 3
Figure 3 Lighting conditions -Left: Dark environment is required to retain colours.Centre: High-contrast projection adding texture within strong blue lit environment.Right: Projector used as "lamp" with texture, overpowering red ambient light.