Growth : Visualisation of predictive mathematical models using 3 D computer graphics and animation

‘Growth’ is a short film created as a result of a multidisciplinary collaboration between artists and scientists from the University of Dundee. The author worked with mathematicians to investigate how techniques and technologies used in 3D computer animation and visual effects industries might support the enhanced visualisation of predictive mathematical modelling of solid tumour growth. This paper will discuss the practical and artistic processes of this visualisation research, including the technical innovations required in undertaking this work – such as the creation of custom tools for ‘reading’ data, or the addition of stereoscopic output. By transforming numerical data into three-dimensional ‘objects’, artists can provide new ways of ‘seeing’ information and identifying patterns or results. Developing visualisation techniques can be used to improve the communication of cancer growth to patients (by increasing patient understanding and relieving their fears) through the use of new and innovative visual material.


INTRODUCTION
Cancer is one of the major causes of death in the world (particularly the developed world), with around 14 million people diagnosed and around 8 million people dying each year in 2012.The World Health Organisation predicts that current trends show annual cancer deaths will rise from 9 million in 2015 to 11.5 million in 2030 (World Health Organization, 2015).
As a result, research and development into improving the prevention, detection and treatment of cancer is a rapidly growing area.The aim of this project was to "develop quantitative, predictive mathematical models of solid tumour growth which can ultimately be used in planning patient-specific treatment protocols such as chemotherapy, surgery and radio therapy" (European Commission, 2015).
As part of this larger funded project, the mathematicians were also interested in visualising the data they were producing as a result of their predictive modelling.This resulted in a short collaboration during 2010 and 2011, between the Division of Mathematics and Duncan of Jordanstone College of Art & Design (DJCAD), both part of the University of Dundee.As a result of this collaboration, the author created a short film titled 'Growth'.This short film (and a complete 'making of') are both available online, and links can be found at the end of this document.This paper will discuss the process of visualisation undertaken by the author, in the role of an artist collaborating with scientists.
It begins with a brief look at the importance of collaboration, and the type of approach an artist may employ.The visualisation process will then be explored, using one particular mathematical dataset throughout.Finally, reflection on the work completed will be included, which will provide insight on visualisation-related problems or future work that could be undertaken.

BACKGROUND AND APPROACH
The idea of artists and scientists working together to achieve a common goal is not a new one.In fact, it is this collaboration that has not only challenged the way we perceive complex technical ideas, but also pushed existing concepts forwards.This section will introduce the idea of interdisciplinary working, and the approach that an artist may take to visualising scientific data.

2.1
Interdisciplinary teams Cox (2008)introduces us to the idea of the "interdisciplinary 'renaissance team'"which brings art, science and technology together, and directs research towards solving complex problems.
By embracing this cross-disciplinary approach, we are granted the best of both worldsthe technical knowledge and understanding of the scientist's mind, combined with the creative skills and abilities of today's artists.Add to this, modern technology (such as 3D computer graphics and animation or the use of stereoscopy), and we open the way to research which is truly innovative and inspiring, and as Cox (2008) confirms, allows these 'renaissance teams' to "make significant contributions to new visual research".Sandin et al. (2006) also discuss the idea of creating a 'renaissance team', re-emphasising its importance in bringing together the best artists and scientists and creating work that is "both beautiful and of great communication value for the scientists".
If we examine the 'renaissance team' and consider each of the contributions, the scientist's work often serves as the foundation for modern visualisation projects.
Scientific research is often quantitative and is responsible for generating a huge amount of information or data (often numerical) which is kept in its purest form.However, this collected data is often incomprehensible to anyone other a specialist in the scientific subject area.The main problem with this approach is that a scientist will typically be required to present information to a varied audience, many of whom may not fully understand the ideas that are being explained.
In contrast to this, Sandin et al. (2006) explain the importance of the artist in providing a collaborative solution, as artists are often more accustomed to organising and showing projects, using experience gained from creating and presenting at exhibitions.They can also bring a new type of creativity to any types of research, and are often known for their abilities in not only creating new media, but also finding "new ways of working with media" (Sandin et al., 2006).
The artist's work can be tailored to creating something new and unique, crossing multiple disciplines, and communicating ideas that previously could not be explained easily without this unique collaboration between art and science, which embraces the strengths of both fields.

Design Research
Design research describes a wide collection of investigative techniques, which can be used to improve design and research processes, gaining greater insight through practice and developing better outputs based on experimentation and iteration.
It is both reflective and cyclic in nature, where outputs can become inputs, and typically follows the stages shown in Figure 1, adapted from an original diagram by Maier (2010).

Figure 1: Typical iterative design research process
It is a methodological approach which encourages practice-based learning, or learning by doing, where "acts of making and reflection can occur along the entire length of the process" (Sanders and Stappers, 2014).This is of particular importance as it encourages critical evaluation during all stages of a design or research process, rather than just an analysis of a finished product.
The author implemented a design research approach throughout the practical visualisation work involved in crafting 'Growth', which created more opportunities for knowledge and learning to be extracted and then re-applied during the project, therefore improving both the process and the final outputs.

Stages of visualisation
In its simplest form, a typical visualisation workflow consists of three main stagesacquisition, processing, and visualisation.These can be seen in Figure 2.Each of these three stages will be discussed in more detail in the following sections.Due to the nature of the collaboration, the author's role during the acquisition and processing of data was minimal, with focus being almost entirely on the visualisation stages.

DATA ACQUISITION/PROCESSING
Mathematicians from the University of Dundee's Division of Mathematics provided all of the numerical data which would be used to create any visualisations.This data was output as part of their ongoing research on the M5CGS project.
The data was generated using predictive models, which aimed to show the rapid growth and spread of solid tumours.The models were built around intracellular dynamics (Figure 3) and combined several principles, including division, adhesion and repulsion.
Figure 3: Intracellular dynamics (Schlüter et al., 2014) The result of running these simulations were extremely large datasets, containing millions of lines of informationsuch as cell identifiers, positions (X, Y, Z), radii, cell states, and timestamps for changes to each attribute.Data files were huge, and often several gigabytes each in size (sometimes with multiple files to create a complete set).
After some initial testing, it was discovered that the best output format for visualisation purposes would be a text based format, using delimiter-separated values (typically separated by a 'tab' in this instance).This kept the data in a simple format, and would allow for the 3D computer graphics software to 'read' it in a straight-forward manner (shown in Figure 4).
As the mathematicians could output the data in a 'clean' and easy to use format, it required no further processing or preparation at this stage.This would involve the use of multiple industrystandard software packages.Autodesk Maya was chosen as the primary tool, which would be used for scripting, all of the 3D work, and rendering.Adobe Photoshop would be used to create any of the 2D images, and to also quickly test the 'look' of things.Adobe After Effects and Nuke (by The Foundry) were both used to composite all of the resulting 3D rendered images and apply final effects.
Each of these processes will be discussed in more detail in the following sub-sections, beginning with an example of how the mathematicians had already visualised the same dataset.

Existing mathematical visualisation
Upon starting the visualisation process, an example image was provided by the mathematicians.This showed how they had been visualising their data previously (Figure 5).
Seeing their current output was important as it helped the author understand some key issues to overcome; their data provided cell growth in threedimensions, but this was not used, the cells were too small to make good use of visual space, and the choice of colours made it difficult to see clearly what was happening.
It was however, a very practical improvement for the mathematicians, who until then had been looking at lines of numbers.It allowed them to make sense of their data, though as an artist, the author believed that this could be improved, particularly by viewing this in a three-dimensional way.

Figure 5: Example of mathematical cell visualisation
At this stage, it was also important to discuss the possible outcomes with the mathematiciansthe visualisation of their data would need to be suitable for their intended audience.
In the case of the author's short film 'Growth', it was intended for viewing only as part of the Masters Show exhibition, to demonstrate the visualisation techniques, and would not be shown to patients.

Scripting and Particles
Before any of the visual development in Maya could begin, the numerical data would have to be 'read' by Maya and translated into a threedimensional, animated representation.However, in its off-the-shelf state, Maya cannot understand text files.
By using Maya Embedded Language (or MEL), scripting allows for the manual creation of custombuilt innovative visualisation tools that can read the data and perform pre-defined actions based on what the tool reads.
In its earliest form, the script used in this project created a number of basic spheres (each representing one cell), which moved in X, Y and Z dimensions (fully three-dimensional) and changed radius.These values were automatically set based on the original mathematical data -ensuring the visualisation was an accurate representation of its numerical source.
The most recent version of this data-reading script (Figure 6) is much more complicated, and creates a series of 'locators' (trackable points in space) that are generated and animated automatically (based on the data).

Figure 6: Example of scripting for Maya using MEL
These locators are technically invisible, so the script also generates a group of particles that are paired with and 'follow' these locators, creating a visible body of cells.Before any visual development, they look like a series of plain spheres, shown in Figure 7.

Figure 7: Basic 3D representation of mathematical data
Originally, the script generated a series of polygonal spheres (this is easier to do), but due to the number of cells needed, it quickly became very expensive computationally, and therefore difficult to work with in Maya.The initial process of creating and animating a series of particles takes slightly longer, and requires more scripting, but is very 'lightweight' in comparison when finished.Using particles would also allow for the ability to 'mesh' together cells into a single organic object.
This final version of the data-reading script is also more technically advanced, as it checks if cells already exist or not, provides visual information on the data-reading process and has been optimised to run faster (approximately 3x faster than the original script).
Despite being faster, running these scripts still takes a significant amount of time to complete.For example, one data set has around 33 million lines of data -the software has to read each line individually and then generate or animate objects in 3D space, resulting in just 120 seconds of animation.

Visual development
After loading all of the data into the 3D software, it was time to begin developing the visual style of the objects.This went through a variety of iterations, many of which were completed in Photoshop for speed.Some examples of these can be seen in Figure 8.

Figure 8: Examples of 'Growth' look development
After several attempts, it was decided that keeping it simple would result in a clearer visualisation.It should be high contrast, and it should not 'scare' viewers by being overly realistic (avoiding the use of red where possible).Figure 9 shows a comparison between the Maya workspace view (top) and the final rendered 'look' (bottom).

Rendering and Compositing
So that the final visual style could be achieved, several different passes were rendered.Each pass contained different information, and when combined would form a complete image.
Rendering can be a time consuming process, and generates a large amount of images.As an example, the final version of 'Growth' was built using approximately 30,000 single frames (this does not include all of the additional testing that was also completed).For a single frame, there were six rendered passes; base, depth, dust matte, glow, matte, and x-ray (shown in Figure 10).These did not conform to the typical passes found in a 3D computer graphics workflow, as they were mainly used to separate elements and implement different lighting or shaders.
Once rendering had been completed, the shots were ready to be composited.'Growth' was composited using a combination of Nuke and After Effects.
The first step was to combine each of the various render layers, to recreate and fine-tune the specific visual style that had developed.
A shallow depth of field was also applied to the image sequence, using the depth layer that was rendered specifically for this purpose.By using an extremely shallow depth of field, the camera appears to be focusing on something very small (simulating macro photography).
Once all the layers had been composited, additional effects were added, to enhance the believability of the computer generated imagethese included motion blur, chromatic aberration and film grain.
Motion blur was applied to the cells, so that faster moving objects blur as they would when using a real camera.It was added to improve the animation of the cancer cells moving and expanding.
Chromatic aberration is a type of lens distortion where different colours of light separate (due to the way a lens disperses light).This creates a coloured 'fringe' around edges between light and dark areas of the image.
Film grain was added on top of the image and gives it a particular texture (similar to that of processed photographic film).By adding film grain, it makes the sequences look as if they were filmed using a real camera, and not computer generated.

Figure 11: Final version of frame 350 of 'Growth'
Figure 11 shows the finished version of frame 350, once all of the effects had been applied.The next step was to apply this to all of the shots so that a final composite could be rendered, and a finished movie could be created.

'Growth'
Upon completion of the film, a 'movie' style poster was created to advertise the exhibition that it would form part of (Figure 12).
The final film was exhibited at Duncan of Jordanstone College of Art & Design's annual Masters Show in 2011, where the author was awarded DJCAD's Interdisciplinary Prize for work that reached beyond art and design.
There was also interest from both the local news, and animation companies interested in the work being undertaken.
Feedback from exhibition visitors was positive, and part of its popularity was due to the uniqueness of the visualisation approach to the subject matter, which most people have an awareness of.

Anaglyph stereoscopy
As there was a small amount of time remaining before the MSc deadline, the author was able to experiment with anaglyph stereoscopy as an output format.This would allow viewers to put on 3D glasses and watch 'Growth' entirely in threedimensions, taking advantage of the 3D nature of the data that was being shown.
So that this could be produced, a black and white version of Growth was created, with a red and cyan split so create the stereoscopic effect.
An example of this is shown in Figure 13, with the top panels showing each 'eye' and the bottom panel showing the composited version ready for viewing.
This was well received during exhibition at the 2011 DJCAD Masters Show, but was also met with some interesting comments about how frightening it was to watch a 'real' tumour grow.

3D Printing
As part of the author's current ongoing PhD research, 3D printing has become of great relevance and importance.
As an experiment, the 3D model from 'Growth' has since been converted into a suitable file format and 3D printed, with the photographed result in Figure 14.

REFLECTION
Due to the nature of this collaboration, and it forming part of the authors MSc work, there was a finite amount of time available.It meant that the work was primarily focused on identifying and solving the visualisation problems, to allow for creating practical results.It meant however, that there would be no time for real-world evaluation, in a clinical context for example.
Instead, it was presented as a short film at an exhibition, which fulfilled the requirements for the MSc programme (the author was awarded an MSc with Distinction), and was met with positivity, though with some trepidation too, due to the sometimes difficult subject matter.
Working with mathematicians proved to be a unique opportunity for the author, despite a steep learning curve.It encouraged both sets of collaborators to develop a shared language, which enabled clearer communication.
It also highlighted a change in the artist's views, where once art and science seemed different in many ways, where actually the things that separate each field should be embraced as strengths that can compliment one another.
A key issue that was raised during the visualisation process was the difference between interpretation and representation.Due to the nature of the mathematical data, and the intended application as an aid in planning patient treatment, it was essential that during the use of the data, it remained 'true' and entirely accurate.
As soon as any part of a dataset is changed, it becomes an interpretation, and can be subject to criticismthough it is project dependant, some may suit a more interpretative approach to data visualisation, in an attempt to communicate better.
An important point to consider is that although these visualisation techniques have been applied to mathematical data, the technical skills and insight gained can be applied to a wide variety of projects.
Upon completing the MSc, the author undertook various different projects, which included the visualisation of asthma in young adults as a teaching device, the reconstruction of MRI data in three-dimensions, and monitoring the spread of pathogens in hospital ward spaces.
Most recently, these techniques have been used to visualise subsea survey dataexamining the bases of offshore wind turbines, and exploring deep sea shipwrecks.Visualisation can be wideranging, and helps the invisible become visible again.
Finally, it is crucial to note that without collaboration, this work, and any that came after it, would not have been possible.Through the combination of expertise in art, science, and technology, complex topics have become more accessible and research has benefitted.Cox (2008) makes an interesting final observation; "Supercomputers, graphics and creative human beings have the power to bring about visual enlightenment with regard to too much in this universe that was formerly abstruse mathematics."

ABOUT THE AUTHOR
The The author is now currently undertaking a fullyfunded PhD research project, also taking place in DJCAD.
It is funded by both the EPSRC and ADUS DeepOcean, and is based around investigating and developing the visualisation of subsea survey data.

Special thanks to
Between 2009 and 2014, a large-scale European Commission funded project was undertaken by the University of Dundee's Division of Mathematics, titled 'From Mutations to Metastases: Multiscale Mathematical Modelling of Cancer Growth and Spread' (or M5CGS).

Figure 4 :
Figure 4: Example dataset shown in Microsoft Excel

Figure 9 :Figure 10 :
Figure 9: Comparison between workspace view and render

Figure 14 :
Figure 14: 3D printed 'Growth' data This proved to a useful experiment, as being able to see and hold the data in a physical form was of great interest, though also quite terrifying in knowing what the object represented.As this was a quick test of the technical process, and happened several years after the original film, it did not form part of any main body of work, content of this paper formed part of the author's MSc research undertaken throughout 2010-11, based in Duncan of Jordanstone College of Art & Design, University of Dundee.
Dr John McGhee, who supervised this collaborative MSc project.Additional thanks to Prof. Mark Chaplain, Dr Gibin Powathil and Dr Daniela Schlüter who provided the mathematical data, and contributed to the development of the visualisations.The University of Dundee is also the first approved Cancer Research UK Centre in Scotland, with a multi-disciplinary team involving mathematicians, biologists, physicists and clinicians all located within the same centre.Also part of the University of Dundee, Duncan of Jordanstone College of Art & Design is one of the UK's leading research centres in art and design and ranked number one in Scotland for worldleading research.