Icon: Authentic 3d Cultural Heritage Models for the Creative Industries

Many UK museums are developing their expertise in the creation of 3D models of objects in their collection. To-date, these authentic, high-quality models have not been made available to the digital creative industries, who instead have relied on manually created models that cannot be relied on to be accurate. This paper introduces ICON, a cross-sector collaborative R&D project that is developing workflows and a technology platform to repurpose digitised models created for curatorial purposes so they can be used in film, TV, games and other creative applications.


INTRODUCTION
Many UK museums are developing their expertise in the creation of 3D models of objects in their collection.Traditionally museums, galleries and libraries have used 2D images to aid them in their collections management, conservation and research and public access to their collections.The opportunity of 3D imaging can make all of these areas of museum activity a much richer experience.UK museums have always been active in their support for the UK creative industries, notably through their picture libraries.
Images from UK collections can be seen on a daily basis in fine art publications, general media as well as on film and television.With the development of computer graphics in film and TV, computer games, and ubiquitous multimedia on the web, there is now an opportunity to market 3D models of cultural objects.High-quality digitised 3D models and textures are required for use in film and television post production, games development, architectural visualisation and, most recently, furnishing virtual business premises within VR worlds like Second Life.These models and textures are usually created from scratch by digital artists as required, but this is a costly and time-consuming process.The task of just researching the source designs takes a significant amount of effort before modelling can even begin.
In the ICON project, Evolutions Television, Smoke & Mirrors, System Simulation, the V&A and the University of Southampton's IT Innovation Centre are collaborating to develop a content exchange mechanism, through which 3D digitised design artefacts from museums will be made available for reuse by the digital media industries.ICON will allow for pre-digitised furniture, decorative objects, fashion, fabric designs and wallpaper patterns to be made available for the dressing of virtual sets and clothing avatars.Users of ICON content will benefit from easy access to pre-built high-quality authentic period and contemporary digital models.In return, we will enable a new revenue stream for museums that will allow them to resource further 3D digitisation work.
In this paper we will present the tools and techniques developed to achieve the vision of ICON.We begin with an overview of the ICON project and the overall system architecture, before describing in more detail some of the steps necessary to take 3D models originated for museum curatorial purposes and make them suitable for reuse by the digital creative industries.

THE ICON PROJECT
At the core of the ICON project is an online 3D model library through which 3D artists are able to browse, purchase and download 3D models of cultural heritage objects.Populating this library with authentic models depends on two strands of technical work: acquiring and repurposing the 3D models themselves; and ingesting metadata from the museum's catalogue describing the original artefact and converting this into a form that is meaningful to those outside the museum community.Although the authenticity of the models is an important selling point for ICON, so too is the quality of the metadata we are able to provide, as it is this that will allow the 3D artist to confidently select a model appropriate for their particular job.

ICON Architecture
The ICON requirements mirror the basic specifications of current digital asset management systems, and much of the terminology of digital asset management systems is used in this description.The basic outline of the ICON digital asset management system is shown in Figure 1.

Figure 1: ICON Architecture
The core components of the system are ingestion, repository, and delivery, supporting, in overview, the submission of models, the storage of models, and the delivery of models.Various shared services are used by each of these components.Shared services include components such as user management, transcoding, and ontology support.These services are used by more than one of the core components.The ICON system is presented at a common API, upon which ICON applications are built.The components and shared services in the ICON system are listed in sections below.

Ingestion
Model Ingestion is the process of bringing a new 3D model into ICON.Many operations need to be carried out in a "pipeline" of actions, including the creation of browsable renditions, automated model cleaning, and metadata mapping.The submitted inputs include: • the 3D model itself; • model surrogates; • information supplied by the user such as a description of the object; • additional metadata for example from the model creation software or from a museum collections management system; • user credentials; • environmental data such as the client's IP address.
Models are passed to the model mill for formatspecific processing.Any model surrogates are passed through a similar processing pipeline, producing renditions and processing metadata.Each metadata set -the combination of the metadata submitted and metadata extracted or derived from the models -is passed to the metadata mapper for translation into the common data model.This process uses various techniques including language and taxonomic mapping.
The outputs of model ingestion include: • An archive or master copy of the model.This will usually be the original model as submitted, but can instead be a normalised, high quality rendition preserving much of the original but using a standard format; • A set of renditions of the model; • A metadata record complying with the common data model.
Models and metadata are passed to the repository where they will be entered into the assessment store pending release to the published ICON system.

ICON Repository
The ICON repository is the heart of the ICON system.The repository is based on System Simulation's Asset Index+ digital asset management system, which is already used by a wide range of commercial 2D picture libraries and museums, including the V&A.It provides storage, discovery, and retrieval services to ICON applications, user interfaces, and other components.The repository maintains and manages a metadata store, and a number of model arenas.Data storage is managed through an abstract storage service.
The repository supports retrieval of models and metadata through an interface which takes in an identifier of the model and an indication of what is to be retrieved.Models are retrieved as if they are stored in or alongside the metadata record: they are not presented as an unstructured or anonymous set of models detached from the corresponding metadata.The system supports the concept of a preferred variant.This simplifies the use of this interface, allowing the caller to ask forfor example -a thumbnail image, without having to work out which variant or surrogate's thumbnail is the most appropriate.

Common API
ICON applications are built to a common API, which surfaces the functions of the ICON system.This ensures a clean separation of ICON applications from the core ICON system.It also facilitates and encourages ICON application development.Many ICON consumers will have technical abilities equal to the task of scripting and adapting ICON facilities to fit in with their workflows.This may increase ICON's takeup and allow consumers to increase the commercial advantage they can gain from integrating ICON into their business processes.The API can be realised in a variety of languages and protocols, although the focus is on SOAP in the initial API implementation.Technologies such as SWIG can be used to provide access into a range of languages without having to address each language individually.

ICON Applications
ICON applications make use of the ICON services and deliver ICON functionality to users directly or through other systems.They make use of the ICON common API to call on ICON services.There can be many ICON applications which can address a variety of functions and domains, including: • ICON contributors' interface, for users who submit models to the system; • ICON consumers' interface, for users who want to find models for potential purchase; • ICON administration interface for internal ICON use for monitoring system use, correcting metadata, approving models where necessary, addressing irregularities, etc.; • Consumer plugins for 3D applications to provide easy access to ICON models for consumers; • Manual model cleaning interface allowing people potentially to earn money providing model cleaning services to ICON.
In the following sections, we describe the different 3D scanning techniques trialled at the V&A, before detailing the metadata ingest, 3D model ingest, and 3D model browse and search functions of the ICON platform.

3D DIGITISATION
Rather than digitising objects afresh to be uploaded to the ICON library, the intention of the project is to enable reuse of models that have already been created for curatorial purposes (either as a preservation record, or for inclusion in virtual exhibitions).Nonetheless, 3D modelling experts from the project's post-production industry partners are working closely with the V&A Photographic Studio team to jointly refine their digitisation techniques and workflows.
The range of objects available to the ICON project within the V&A is extensive.The V&A Collection contains an example from almost every aspect of the built environment.As the National Museum of Art and Design the V&A has exemplary objects such as the national collections of British Sculpture, portrait miniatures, photography, metalwork, ceramics and many other designed and ornamental objects.It also has collections from China, India, Japan, Renaissance Italy, and the Middle East.All of these collections allow ICON to be generous in the range of material objects to test within the project.

Scanning Process
The A more traditional scanning technique was then tested with the Next Engine (NextEngine Inc 2011).This device is sold as a consumer item in the US as a product for hobbyists.It is a small self contained laser scanning device which is relatively inexpensive and easy to use.It has its own acquisition software which is automated.In our experience the colour reproduction is poor as the texture camera within the device is quite low level.
The resolution is also quite low.Highly reflective surfaces will not record well with this device and a lot of noise is created which degrades the resolution.Next Engine themselves suggest dusting objects with a fine white powder to reduce reflection of the object, something not possible with publicly owned cultural objects.The results we achieved were not of sufficient quality for scholarship but may be acceptable for visualisation for the general museum visitor, or as background in virtual productions.The highest quality scanner that the V&A Photographic Studio have recently used is that made by the Breuckmann company (Breuckmann GmbH 2011).This scanner uses structured light to create interference patterns onto the surface of the object from which the 3D points are created.There is still the difficulty of reflective and none structurally stable objects to face as with any other 3D scanning technique but the resolution of this scanner is considerably higher than any other device the V&A has used.The Breuckmann scanner has enabled the V&A Photographic Studio to develop a more mature workflow and analyse the requirements needed to create a volume of 3D image models.An awareness of the range of tasks, time taken to make models, storage issues and subjective quality are allowing a realistic programme of digitisation to be undertaken.Figure 2 shows a mesh resulting from the digitisation of a V&A artefact with the Breuckmann scanner.

Quality of meshes
Subjectively the Breuckmann scanner has by far the best meshes that the V&A Photographic Studio has made so far.Product specifications suggest that a resolution of 45µm can be achieved with this scanner and recent experience suggest that this is the case.For the scholar and conservator this level of accuracy is important.When they can rely on the accuracy of surface, and trust the metrics made by the scanning techniques, then they will use the image models for their own tasks.A concept then arises of the idea of a true surrogate object.A 2D image represents the appearance of an object at the time and conditions when the image was made.The 3D model can potentially take this further and give the viewer a richer and more accurate impression of all aspects of the object; it's mass, size, physicality, and surface texture.
For the user of the model the idea of quality is open to question.This will differ depending on the uses that the model is to be put.Low resolution models may be perfectly acceptable for visualisation for the general public where the higher may be necessary for the scholar.In general practice in 2D imaging providing the highest resolution is the norm as most effort is used in moving the object and image capture.File size is largely irrelevant in this process, so it is assumed that making files as big as possible for all potential purposes is best use of resources.There is no reason to believe that in the cultural heritage sector that this will be any different for 3D.

Future of Museum 3D Digitisation
The time that it takes to make 3D models of objects is of importance for any of the business activities in

METADATA INGEST
Once the museum object has been scanned, the next step in the ICON workflow is to ingest the 3D model produced by the scanning process along with its metadata into the ICON database.The ICON metadata ingest component is the part of the ICON ingestion module responsible for three main functions, exposed via the ICON contributors' interface: • Mapping external metadata schema used by various end user organisations (museums etc.) to the core ICON schema.This helps in creating consistent metadata for publishing and retrieval purposes.• Knowledge acquisition: in this phase users are able to ingest metadata pertaining to the 3D models into the ICON framework.Strategies such as thesaurus mapping, external repository imports and manual data entry are used to achieve this purpose.• Reasoning: during the reasoning phase new knowledge is inferred from existing information stored in the knowledge base.For example the style of an object could be automatically derived given the background knowledge such as the creation date, place of origin etc.

ICON Metadata Schema
The ICON Metadata Schema is designed to support four ICON system functions: automated processing of newly ingested models, discovery of models by potential customers, quality assessment (both as an automated part of the ingest process and by the user themselves), and commerce -the ultimate sale of the model to the customer and transfer of funds to the appropriate model provider account.Administrative metadata and metadata relating to individual model transactions are not included in this version of the schema but may be incorporated at a later date.Wherever possible, concepts have been used from existing 3rd-party schema.Notably, concepts that replicate information already held within museum collection management systems have been adopted from the Lightweight Information Describing Objects (LIDO) schema (Coburn et al. 2011).LIDO was developed by the CDWA Lite Advisory Committee and the Documentation Committee of the German Museums Association to support just this kind of aggregation of museum collections records.In reusing existing concepts our aims are to avoid expending unnecessary effort duplicating existing work, and, equally importantly, to ensure that the ICON platform can be interfaced with other museum information systems with a minimum of additional effort.Notably the Consortium has considered sourcing metadata from small museums' collections indirectly, from the Europeana database; our adoption of LIDO concepts will greatly simplify this process.

Schema mapping system
The schema mapping system allows users (typically administrators) setting up the ICON system to map external proprietary schemas which are used within the context of their organisation to the common ICON metadata schema.This serves three main purposes: • Standardisation: managing information conforming to various different schemas is a very difficult problem in a knowledge management context.By conforming to a single schema (ICON metadata schema) we are creating a standard set of metadata resource documents.All the parties using ICON system and users of the ICON schema will have a shared understanding of this knowledge representation format.The LIDO schema was taken as the base ICON schema for this purpose.
• Semantics and rules: By semantically annotating the information produced during the ingest phase we are enabling the provision of semantic searches and reasoning over the metadata produced.This is because the metadata produced as a result of the ingest phase is machine processable.• Data migration: Once the mapping between the two schemas (external and ICON) has been defined, it is relatively straight forward to migrate data over from external repositories (e.g. the V&A API) to ICON format.This way we do not lose the information which has already been recorded for the contents being described.
When the schema has been successfully imported, the user then uses the metadata mapper form to associate ICON schema fields with fields from their own schema, selected via dropdown menus.
Currently direct one-to-one field mapping is supported, and we have also implemented a simple text processing function to extract various dimension and dimension metric fields from the single text field used by the V&A.During development of the final ICON software components this capability will be enhanced to support mapping of multiple discrete concept fields to a single natural language descriptive text field in the contributing organisation's schema.As part of this development a flexible scripting interface will be provided allowing museums to define how they themselves wish their metadata to be parsed.

Metadata import system
The metadata import module provides facilities for knowledge fusion during metadata ingest.Web based forms are provided for users to fill in and populate as a result, the various concepts identified from the ICON metadata schema deemed to be useful for the search and retrieval of 3D models AJAX based intuitive interfaces are used here for suggestions during the form filling stage.This screen also has an import button which tries to auto fill the form by importing metadata which has already been recorded for a particular object.We have currently implemented this functionality using the V&A's public API and, once finalized, the relevant interface will be published so museums are able to craft their own ICON import modules.
The system knows exactly which fields to extract from the third party data because the mapping has been defined in the previous phase.However for organizations without a published schema or accessible API, the form can also be used to manually enter metadata as the model is uploaded.Note that technical model metadata is not included in the form as it will be extracted automatically by the ICON 3D processing components as the model is ingested.

3D MODEL INGEST
The ICON system needs to convert ingested models from one format to another.For example, a museum may upload a model in a format which is widely used in the museum sector but is not appropriate for delivery to ICON's consumers in the digital creative industries.In this case the model must be transcoded by the ICON system at some point before it is delivered to the consumer.Similarly, ICON user interfaces will use 2D thumbnails and small or low resolution versions of models for users who are browsing the collection.Processing of ingested models is performed using, the ICON model mill, an extensible processing engine we briefly introduced in Section 2.1.1.
The ICON model mill is a media processing engine which can perform a range of functions on 3D models in a variety of formats.The model mill is a framework for housing model transcoders and other format specific software.The ability to slot in support for new formats and new transcoders provides ICON with a sustainable technical basis for surviving technology and industry changes.The model mill's entry points are Identify, Extract, and Convert, discussed below.
The identify entry point works out what kind of model -or other object -this is.The identify function may analyse the content of the object itself, or use an external metadata element, such as the file name extension.The model type is returned as a MIME type.This information is used to make a first judgement on whether the model should be accepted for ingest.
The model mill's extract entry point extracts metadata from the model.The extract function uses format specific code to find out and return information about the model, aside from the essence of the model itself.Many digital asset formats include schemes for explicitly embedding metadata inside the object; the extract function will access this information.In addition, the extract function will intuit other metadata such as pixel or polygon counts, judgements of model quality, etc.
The ICON project will benefit from good assessments of model quality.These will allow better pricing and provide a better discovery service to consumers.Manual assessments of model quality will be expensive to resource, and so ICON will run an automatic quality assessment using inputs such as mesh complexity and texture resolution.This quality assessment will be housed in the model mill and will be accessed using the extract function.Quality assessments will select between a range of quality bands, with quality labels such tier1, tier2, etc. Metadata insertion is part of the convert function and can be run at delivery time to embed the appropriate set of up to date metadata into the delivered model.ICON will also explore use of automated remeshing, retopologising, and other automated conversions affecting model quality and applicability.These facilities will also be surfaced through the model mill's convert function.The convert function will also provide protection and tracking facilities, if and when they are developed for the domain of 3D digital models.This includes the 3D equivalents of watermarking and hidden steganographic codes, digital onscreen graphics, digital rights management (DRM), and other copy protection schemes.

Mesh conversion
Mesh data can be submitted via the ICON web portal in any of the supported formats.During ingestion the transcoding module automatically converts the mesh and stores the results on the ICON file server.This provides flexibility to the enduser in choosing a download mesh format based on their specific requirements.The transcoding module also applies edge-collapse simplification methods to produce lower polygon count versions for use in real-time applications such as games and simulations.The transcoding module is currently based on the SVG Library created by the Visual Computing group of the ISTI -an institute of the Italian National Research Council (CNR).The transcoding module is able to run concurrently allowing multiple meshes to be converted simultaneously thereby simplifying the ingestion process.

Auto thumbnail generation
Upon Ingestion, meshes are automatically passed to Autodesk Maya® (Autodesk Inc 2011) and rendered as static images and 360 degree turntable animations for use on the ICON web portal as thumbnail previews.The 360 degree animated thumbnails (see Figure 3) are advantageous due to the difficulty in automatically producing clear and aesthetic static images given the wide variety of possible input meshes.

MODEL BROWSE AND SEARCH
Ingesting 3D models and their corresponding metadata is only half the story.We then need to make them available to potential customers in the digital creative industries.Our market analysis has shown that one of the leading reasons postproduction houses purchase pre-created 3D models is simply to save time -they are used when there is a pressing deadline to complete a job and get it out to the client.Therefore we are developing a web-based front end to the ICON model library, the objective of which is to allow customers to identify quickly and with confidence appropriate models for their job.Appropriate means not only that the model is of adequate technical quality, but also that the modelled object is right for the scene into which it will be inserted -for example a 3D artist looking for background objects to composite into a Victorian interior for a period drama wants to find objects right for that period, without then having to carry out their own research to make sure that chair they just downloaded isn't really an Edwardian design!
The ICON Consumers' Interface consists of a search and browse web interface for 3D models for the end users of the ICON system.Although a keyword based search interface was already available as part of the default Asset Index+ installation on which ICON is based, it was determined through user trials that a browse and smart search system was essential.An early prototype interface, shown in Figure 4, was developed to test potential features with end-users, but plans are currently being drawn for a new consumer UI.

Figure 4: Prototype model consumer web UI
In the new consumer web UI, a hybrid search approach (using a combination of semantics and folksonomy based search) will be employed.Users will be able to browse through object collections and refine their search criteria with the help of knowledge filters.An example of a sample query might be "give me all the vases in China between 1800 to 1900 which are red in colour".The actions performed by the users on the search site will automatically be transformed into queries to the database.
Upon the selection of a 3D model users are presented with various renditions (high, low and medium resolution images) automatically generated for it.The mesh transcoding component is also currently capable of producing an animated GIF image of the mode automatically.The user is able to add any models that catch their interest to ICON: Authentic 3D Cultural Heritage Models for the Creative Industries Richard Beales,Ajay Chakravarthy,Sam Kuhn,Steve Luther,Michael Selway,Mike Stapleton & James Stevenson 123 their 'lightbox', a personal user area that acts as a staging post of any models they might subsequently wish to purchase.A standard ecommerce component handles the purchase and payment of the model after which the user can download the model.

FUTURE WORK
We have identified four areas where further work is required.Firstly we are still learning which 3D digitisation techniques are most appropriate for different types of object, and refining the associated digitisation workflows.As we gain experience we will publish guidelines to assist other museums and cultural heritage institutions in the process of digitising their own collections.The automated 3D model mesh refinement function requires further development, and this work will be guided by dialogue with other post-production houses to establish acceptable mesh quality for different applications.The metadata ingest and search functions are to be improved adding two significant capabilities: • automatically inference of 'commercial' annotations, for example to indicate the decorative style of an object; terms such as 'baroque' or 'modernist' are typically omitted from the original object's museum record as they are not considered academically precise.This process is currently undertaken manually be museum picture libraries to drive 2D image sales, but is time-consuming and laborious.• automatic compilation of themed model collections and recommendation of groups of models that could authentically be placed together in the same virtual scene.
Finally, our intention is to create illustrative videos using models from the V&A collection to demonstrate to 3D artists the creative potential of authentic cultural heritage models.As a teaser, a single-frame test render created by 3D artists at Smoke & Mirrors is reproduced below as Figure 5.

CONCLUSIONS
In this paper we have presented ICON, a platform for repurposing digitised 3D models of cultural heritage objects, and their accompanying metadata, so that they can be reused by the digital creative industries.We have described the overall platform architecture and detailed some of the technical steps required, including the various 3D digitisation techniques trialled at the V&A.If you are interested in following progress of the project we invite you to join our 'Friends of ICON' mailing list: please contact icon-friends@ssl.co.uk.

Figure 2 :
Figure 2: Digitised V&A artefact, created using the Breuckmann scanner the museum.Value for money and visibility of the collection are of great importance to museums in the current fiscal climate.Hence whether the 3D image can add value to the work of the museum is still open to question.Analysing the use of the 3D model for scholarship, conservation and public access is the next task of the studio in making this type of digital content.All of the technical changes that occur in imaging create a level of uncertainty in these groups in museums.They are naturally conservative and are slow to respond to new technology.It is the business of the image makers to educate these groups in the opportunities that this new technology offers them.When these new forms of imaging are established as routine and new ways of seeing are created then the commercial opportunities being developed in ICON will become possible.

Figure 3 :
Figure 3: Still-frames from an automatically generated model thumbnail

Figure 5 :
Figure 5: Test render using an ICON model The model mill is called to determine the format of the model, and to produce suitable renditions of the model.Renditions are automatically generated versions of the original model, including non-3D preview versions and high quality renditions in other data formats.Ingestion behaviour, including the list of renditions to be generated, is defined by a set of configuration parameters called ingestion rules.
The model mill is also called to extract metadata from the incoming model.Extracted data includes technical metadata embedded in the model by hardware or software involved in the model's creation.It also includes any metadata embedded in the model by the (human) creators of the image.
ICON is a collaborative R&D project supported by the UK Technology Strategy Board.The project partners are Evolutions Television, IT Innovation Centre, Smoke & Mirrors, System Simulation and the Victoria & Albert Museum.