Seen Description: Visualising and crafting data in mixed realities

The immense potential of the convergence of digital technologies such as, 3D scanning and photogrammetry, 3D printing, motion capture, real time performance capture and mapping, camera tracking, interactive lighting and virtual production, is yet to be fully grasped, exploited and appreciated. The creative aspects of this technology convergence offer exciting new opportunities for the visualisation, mixing and crafting of materials from the physical and digital worlds; real world objects can be 3D scanned, manipulated digitally, printed out using the 3D printing process, and painted in the real world to be used as set dressing on a virtual production set to be recorded on digital video. The real and digital states of objects have become fluidly interchangeable offering unprecedented creative control to artists to shape the world at will. This paper highlights some of the possible applications – beyond the mainstream usage in film and games production – that the convergence of these technologies offer based on case studies of select student and staff projects undertaken at The National Centre for Computer Animation at Bournemouth University, UK. These applications include, but are not limited to, creative art, photography, digital heritage and preservation, and customised applications for rehabilitation of patients.


INTRODUCTION
The National Centre for Computer Animation (NCCA) at Bournemouth University, UK was established in 1989 and has been a pioneering institution in computer animation education. In 2012, the NCCA received the Queen's Anniversary Prize for contribution to world-leading excellence and pioneering development in computer animation. The key to the NCCA's success is its interdisciplinary emphasis across arts, science and technology (Comninos et al. 2010). As such, the definition of "creative" and the "technical" do not conform to arts and science respectively but are seen as interchangeable; programming can be creative and [texture or matte] painting can be technical, for instance. Beyond the creative and technical elements, a number of projects in the NCCA have also blurred the boundaries of the analogue and digital domains, seamlessly transposing and crafting material both the in the real and digital worlds. This paper explores a range of NCCA staff and student projects that have made this material transposition possible.

The Hunt
The Hunt ) was a computer animated film created by Adele Olsauskaite, Elisabeth Wetchy and Beate Nieuwoudt as part of their Final Major Project at the NCCA. The students wanted to create a painterly look in the surface textures of the environment and characters in keeping with the narrative treatment for the project. They spent a considerable amount of time testing ways of creating painterly looking textures using computer graphics (CG) but were not happy with the results. They finally ended up painting textures using real paint in the real world (see Figure 1), scanning these textures digitally (see Figure 2), and applying them to the 3D objects in the digital world (see Figure 3). The scanned textures could be manipulated before being applied to the objects and the overall look of the final rendered images could also be tweaked further during the lighting, look development, rendering, compositing and colour grading stages.
Olsauskaite  states that "From a practical standpoint there isn't much use in going through such process, but as an experiment it was quite fun.
[It] Also shows that there is space for multiple medias interacting together in unexpected ways. I know for me, never really having used digital tools for painting before my BA but having painted with oils and acrylics from an early age, there was a steep learning curve and a certain rejection to it, and while at the time of the project I wasn't soooo bad with digital painting, searching for ways to incorporate more familiar real world tools was comforting. That being said, I'm not sure how much of that process translated into the film after all of the filters were applied.". Nieuwoudt (Olsauskaite, Nieuwoudt & Wetchy 2022) adds: "I really enjoyed exploring textures in traditional media with Adele and found it helped set a headspace while we were trying to find ways of applying and conveying them in 3D. I know keeping the shapes readable was one of the biggest problems we faced and I'm certain it was the right decision to limit the number of moving camera shots (though keeping one or two was a good experience also to see how we could take them through the process). Treating the characters differently to the environments also worked well. They would've gotten lost if taken through the same process. Trying to portray painting of such a loose nature for a story with a very clear-cut vision was certainly a little beyond us but I'm still happy that we did the project even knowing that. Having our limited skills in mind definitely helped us plan it better than I feel we otherwise would have."

Abstract expressionistic 3D art
Given the Covid-19 lockdown, NCCA student Ashley Cornall took to experimenting with abstract expressionism in the context of the Light-in-Space movement (2021). Cornell created a 3D version of her room and created a painting on glass (see Figure 5) that she scanned (see Error! Reference source not found.) as used as gobo to project the light and colour onto the 3D scene (see Error! Reference source not found.). The rendered images were then printed on watercolour paper and painted upon further using watercolours. These paintings were then scanned back into digital space but during the scanning process, Cornell moved the paintings so that the final result produced a mechanically distorted and warped effect that resonated with the artists' experience of technology on her own life (see Figure  8).

Exeter cathedral colour reconstruction
Digital colour reconstruction of Exeter cathedral was undertaken by Khosravi (Khosravi et al. 2022) as part of the Vista-AR (2022) case study. A 3D scanned model of the Exeter cathedral was provided by Vista-Ar. Khosravi's task was to produce a plausible, evidence-informed colour reconstruction based on a previous study by Sinclair (1995) that provided residual colour samples from within the Exeter cathedral stonework. The textured model was then used within an augmented reality app to create an immersive cultural heritage experience.

Virtual production portrait shoot
Virtual production became hugely popular during the Covid-19 pandemic as travel restrictions came into force and studios tried to find ways of working in local, controlled environments. In summer 2021, an LED stage was created at Bournemouth University to explore virtual production best practices, challenges and constraints. A stills photography portrait shoot was undertaken as a practical experiment to explore craft best practice, plausibility of the resulting look in terms of digital and real-world light and camera match, and the challenges and benefits virtual production had to offer to photographers. The real life character was photographed against the LED screen with a range of computer generated (see Figure 12 and Figure  13), as well as photographed location backplates ( Figure 14).
The studio lighting and camera attributes could be adjusted to make the real-life model sit better with the background. Alternatively, the background image or virtual scene could also be manipulated in real time to create a different look. A second variation explored photographing the subject against a greenscreen using the LED wall a large volume light to front and side light the model (see Figure 15). The greenscreen could subsequently be replaced with the image used on the LED wall to create a more realistic light match. A third variation used the LED volume solely as a large lighting volume with the model without changing the background. This allowed the photographer comprehensive and instantaneous control over the subject lighting using the LED wall as a lighting 'canvas' to paint the colour and intensity of light to create the desired look (see Figure 16 and Figure 17). In order to create spatially plausible light bounce, a 3D scan of two characters sitting on a bench was created by the author by photographing the scene from various different positions. A 360 high dynamic range panoramic image was also created to generate a volume light map for the computergenerated scene. A computer-generated sphere object was then placed in the scene along with a volume light mapped using the high dynamic range panoramic image (see Figure 18-20). The spatial proximity of the sphere to the bench and characters resulted in spatially accurate reflections and light bounces (see Figure 21).

THE FUTURE OF VISUALISATION AND CRAFT IN MIXED REALITY
The advancements in digital technologies, such as 3D scanning, 3D printing, virtual production, virtual reality and augmented reality, have enabled artists unprecedented control over the interchangeability of material states across the analogue and digital domains. The artist now has the control to seamlessly switch between these domains based on where they feel the operation at hand could be better accomplished. The digital and analogue has thus become fluid. It would be very interesting to see the fusion of analogue and digital craft practices as these technologies become more affordable and commonplace.