Motion Capture Visualisation For Mixed Animation Techniques

In this paper we discuss a novel multidisciplinary method for computer animation, using motion capture ('mocap') as reference, combining techniques from 2D and 3D animation, and digital sculpting. Our method develops a process to create animation based on mocap data, without being restricted by standard practices that depend on existing rigged 3D models, allowing for visual expression and improvisation while taking advantage of naturalistic motion and interaction within a 3D environment. The standard mocap methodology for creating animation consists of retargeting (transferring) the recorded data from actors and performers to existing digital characters, providing them with movement. The motion is then polished and tweaked by animators, until the final result is achieved. The character’s animation is the result of the captured performance and the original character design, but they are not created at the same time, as they are put together later on. Something similar happens with 3D computer animation: in order for animators to articulate characters into poses that are interpolated by the computer, a model of a character has to be built first. Here, the animators fully control the performance, but the design of the character pre-dates this process, and can only be modified within certain constraints. Mocap is bound by physics and naturalistic movements, animation can be exaggerated, weight and force have to be conveyed, rather than imposed. Both differ aesthetically but none of the approaches takes full advantage of 2D animation methods, where drawings dictate shape, form and motion at the same time. The characters here can be designed for the movement they perform in particular shots. This opens possibilities for a more experimental approach, where abstraction can exist. Our method combines the different disciplines and allows form to be created for each key pose, using digital sculpting tools for development and mocap as reference. Then, poses can be interpolated so the method is still interactive, allowing for experimentation. Using drawing as the starting point from the mocap data allows for greater understanding of the poses by studying the human figure in motion. This creates new opportunities for designing the animation, regarding shapes, forms and movement. Good drawing is not copying the surface. It has to do with understanding and expression. We don’t want to learn to draw just to end up being imprisoned in showing off our knowledge of joints and muscles. We want to get the kind of reality that a camera can’t get. (...) But don’t confuse a drawing with a map! We’re animating masses, not lines. So, we have to understand how mass works in reality. In order to depart from reality, our work has to be based on reality (Williams 2001, p.34).


INTRODUCTION
No one knows for sure why a pencil tracing of a live action figure should look so stiff and unnatural on the screen, unless there simply is no reality in a copy. (Thomas & Johnston 1981, p.323) Disney animators were conscious of both of the potential and limitation of rotoscoping live action footage. References were shot on camera and studied in detail, informing and inspiring their creative spirits while coming up with new ideas for their animations (Thomas & Johnston 1981). Today, live action is used through motion capture ('mocap') technology to create digital animation. Using sensors on actors' bodies, motion is captured into computer data and digital characters can be moved without the necessary intervention of animators. But the key difference from the previous rotoscope technique, is that mocap can move characters even in real time. Although this technology considerably speeds up the painstaking process of animation, as Sito (2013) notes, it is important to acknowledge that it is not a painstaking process for the animators who actually enjoy doing it! Replacing the techniques of traditional 2D animation and computer generated (CG) 3D animation by mocap, may lead to what the author and animator Tom Sito calls the "uncanny hybrid". In this paradigm shift, often artists are hindered and discarded, and the technology gets repeatedly disproved and discredited by cheap results.
Achieving realistic animation is surely not dependent on technical processes, or specific art forms. A good example of this is Michael Dudok de Wit's The Red Turtle (2016) where, despite the simple 2D stylised designs of the characters, their movement is natural and often realistic. And this is achieved without rotoscoping any live action footage (Amidi 2017).
So, what methods can be used that take advantage of the best of both worlds? In which way can (or should) mocap be combined with animation? And what type of animation? Through experimentation, combining different techniques and approaches, while referencing real life, we should able to create bridges represented by new visual imagery, which hopefully can inspire and contribute to the development of this new artistic practice.

Standard methods
Standard mocap pipeline consists in recording an actor's motion and transferring the data to an existing 3D character. This animation can still be edited and polished, however while having a starting base for the performance can be truly helpful, mocap data can also be hard to manipulate. The process of retargeting the data from the actor to a character forces the animators to be concerned with matching the two together, rather than focusing on creating the actual performance.
3D animation, on the other hand, is created one frame at a time, manipulating the character in key poses that will be interpolated by the computer, resulting in in-between frames. Achieving good results depends on balancing both methods. A good creative opportunity appears when animation is thought in terms of shapes as well as motion. Not being constrained by a regular human physical form, but rather creating a typography or abstract shapes can inspire the creation of the illusion of life.

Alternative methods: digital sculpting
Our approach starts with building an armature connecting the original sensor markers onto a simplified figure, an armature. The markers themselves are also isolated into small spheres, providing information of the positive and negative spaces.
Visualising the data fosters the study of the figure in motion, from the timing of each action, to the spacing of each position, in a 3D virtual environment where lights and cameras are easily added to the scene. In this case, a simple walk was recorded at 120 frames per second (fps), converted into 12 fps so the number of poses could be legible. Here, the motion can be edited or cut, timings can be adjusted and multiple clips from different takes can be put together. At this point, no retargeting techniques have been used. This means that the data recorded isn't being applied to a different character 3D model, with different proportions.
The other large advantage of this departure from standard practice is that 3D geometry is used to build the points and the armature. Because of this, the polygonal geometry can be exported into a digital sculpting application, in this case ZBrush, and form can be constructed. There, the armature is inflated, subdivided into smaller polygons for greater resolution and sculpted. The spheres representing the markers provide information for the limits of the volume of the original actor. In the first iteration of this practice, cubes are controlled by locators, and locators carry the data from the original sensors. By creating cubes, we work directly with geometry that will be sculptable, contrary to locators. For this, faces of the cubes are connected, providing the structure for the armature represented above. Other structures are also possible within this process. A big limitation of this approach is that it is very difficult to overlay different poses, as they are spread out in 3D space. This becomes a problem when trying to sculpt similar poses in different times, or when creating interpolation of poses for in-betweens.
In a second iteration, a new armature is created and driven by a control rig, instead of the original locators. This rig consists in digital joints and deformers that can control and manipulate a 3D character. This rigging system is now driven by the locators, but it is possible to freeze its movement in space, for example in X and Z axis (left and right, back and forth) and allow only for the rotation of the limbs. This facilitates the overlapping of poses, speeding up the process of creating sculptures that relate to each other. At the end of the sculpting phase, all the different key poses are displayed only at their correct time, the motion in space is unlocked, the sculpted forms moved across the space.
Another advantage of this method is that the control rig created is built the same way it would be for creating standard 3D animation. So, all the poses can now be easily manipulated, and inbetweens will be automatically updated. Having a 3D model driven by a control rig, as is usually done in a 3D animation pipeline, differs by presenting the opportunity of creating form from specific poses.
Additionally, one can still control the display of both the digital sculptures and armature, that can be turned visible at their given time, or at a different one. This allows for illustrating motion and contributes to the final visual design of the animation.

Alternative methods: digital drawing
A balance between the original performance and the later animated performance questions both the aesthetics and techniques used. While mocap is a fast way of generating human motion in the computer, it does not replace 3D animation, nor is always the best approach for a project (Kitagawa, Windsor 2008).
On the other hand, why use animation at all and not just mocap? Artist and researcher Brigitta Hosea defines animation as an artificial construct that could not be created in real-time (Hosea 2012).
Although the animated character lacks physical presence, its very immateriality raises fascinating questions about the site of performance in animation, of notions of the animator as a performer and that, in the eyes of the viewer, an artificially constructed animated character is giving a performance (Hosea 2012, p.33).
Although the method of creating digital sculptures from mocap data is a step forward in defining the performance visually, in a pose-by-pose manner, the method lacks spontaneity in building the figure and designing shapes can be challenging. Adding digital drawing to the process allows for gestural design, giving the animator more space to interpret the original performance, and more flexibility designing shapes and motion simultaneously.
By being able to use drawing as a tool, the animator can express and define ideas in terms of character and composition (Kunz 2013).
Besides looking at drawing as a tool, one must also look at it as a graphic style or language. This means that apart from being used to develop ideas, drawing can be a personal form of expression, visible in the final format of the film (Kunz 2013, p.51).
So, how can drawing be combined with the previous method? Unfortunately, the 3D application Maya where the previous method was developed, has limited tools for drawing in three-dimensional space. To draw from any angle on the figure, requires for the drawing to happen on a twodimensional space existing between the camera and the 3D model. It would be possible to project the lines on the 3D model, but it would imply sculpting from a fixed perspective to follow the drawing. That would take away from the purpose of spontaneously creating shape and form from any angle.
In alternative, the mocap data can be sent to the 3D application Blender, which recently has been improving a drawing tool called Grease Pencil. This allows for drawing in a 3D space, taking advantages of being able to manipulate digital lines, using their vector characteristics, applying modifiers, using different camera perspectives to define the volume, and most importantly, converting them to 3D geometry. Once the lines are drawn, they can be moved in space to better represent the figure's volume. Lines can also be connected to create planes, facilitating the design of shapes, or can be converted to 3D geometry by being applied thickness. By combining the armature with the 3D lines and using them on a sculpting application, the process can be further developed. The armature provides the interior structure of the figure, the points from the sensors suggest the limits of the volume, the lines from drawings define shape and intent, characterising the performance from the animator's point of view, adding new poses with relative ease if necessary.

CONCLUSION
Autor Nancy Beiman, points out that not all scenes are possible to act out, in order to create references for animation (Beiman 2010). In this sense, being able use digital sculpting and drawing on mocap data shows to be a great improvement, if one wishes to build up visually on the performance. This could be to exaggerate something that it's there, or to add something that isn't, as weight, forces, or anything the mind wishes to imagine. Mocap offers detailed three-dimensional references, a great resource for studying the human figure in motion.
Good drawing is not copying the surface. It has to do with understanding and expression. We don't want to learn to draw just to end up being imprisoned in showing off our knowledge of joints and muscles. We want to get the kind of reality that a camera can't get. (...) But don't confuse a drawing with a map! We're animating masses, not lines. So, we have to understand how mass works in reality. In order to depart from reality, our work has to be based on reality (Williams 2001, p.34).
Animation director Richard Williams offers a good perspective on the topic, even if it isn't related to mocap per se. Artistic tools are just tools, in order to be properly used, nature and reality have to be studied first. If not, creating something new will most definitely fall short.
The method introduced in this demonstration is still in development. Exposing it to a wider range of motions will allow us to detect new improvements and advantages, as well as limitations over alternative techniques.
The author Paul Wells, analysing Chris Landreth's short-film Ryan (2004), refers that the more artistic and maverick projects are the ones that move the art form forward, away from the mainstream system. And in Jan Svankmajer's stop-motion work, Wells recognises that a more experimental and conceptual approach, not only results in novel visual developments, but more importantly, in the possibility of alternative narratives (Wells 2006).