Creative Reverse Engineering: From remote sensuality to haptic metrology

Our on-going experimentation in close-range Photogrammetry has overcome common challenges to generate accurate, high-resolution 3D models using a single DSLR camera and innovative approaches and computer-coded devices. More recent investigations across disciplines sought to bridge the gap between traditional creative skills and modes of visualisation, and what can be made possible through digital means. Working as artists and designers alongside museums and historians, we have developed new photogrammetry equipment and approaches to help overcome the complex fluid geometry, shadowy occlusions and delicate lamina edges of challenging monochromatic garments. The resulting 3D models have allowed the rapid extraction of faithful seams and surfaces direct from the processed ‘mesh’ and into CAD modelling environments for the production of new designs, patterns and production tooling. Current collaborations seek to apply these developments to the capture, visualisation and reverse engineering of iconic garments and museum artefacts, alongside the 'remote sensing' of traditional pattern cutters whose eyes, minds and hands are the equivalent of the digital approaches explored in our paper. The second phase sought to unpick some of the familiar structures of object-and asset creation in and for Virtual and Augmented Reality. Experimenting with equipment, methods and processing techniques allowed us to explore the potential of 3D visualisations and readily available tools within Virtual and Augmented Reality for garment design and other forms of object-based creative ideation, sketching and prototyping. In the next stages of our on-going research we explored a more haptic form of CAD-enabled modelling and metrology, aiming to bring these approaches within the reach of a wider range of creative users: enabling artists, designers and makers to move more seamlessly between digital tools and virtual environments, and actual things in real time and space.


INTRODUCTION
On-going artistic and technological research and experimentation in digital capture for processing via computer vision has led to developments in the realm of close-range photogrammetry (Dines 2017a-d), in particular where the camera meets its most challenging subjects -and objects.Early successes with small objects, precious materials and highly detailed surfaces including pile fabrics and metal-wire embroidery (Dines 2017c,d) have helped to generate incredibly accurate, high resolution 3D models using a single DSLR camera, alongside inventive methods and innovative devices to overcome traditional restrictions to photogrammetric capture.
More recent investigations have collaborated across disciplines and contexts, exploring the potential of photogrammetry to facilitate the creative reverse engineering of historic and contemporary garments through non-destructive means: both as a feed for design inspiration and iteration; and analysis as part of design history, pedagogy and communication.
The workflows for capturing and processing photogrammetric models with a static camera, creating relative motion between it and an object using a turntable, are well known and documented.Refinements to this process were explored in more detail in an earlier phase of research into hardware configurations; capture and processing methods; and assistive devices in the form of 3D computer coded markers and targets (Fig. 1, Dines 2017d).This work has been further developed for the specific capture of garments and their subsequent reverse engineering through Computer Aided Design (CAD).These results have fed into a new phase of investigations into the potential of VR and AR for a more haptic and embodied relation to garment design, making and production through hybrid physical and digital means.
This paper explores and documents these related experimental processes and proposes the next steps in exploring the cross-over between physical and digital methods of ideation, sketching, designing and making that are at the heart of contemporary creative production.
The next stages, further developments, 'live trials' and collaborative projects with artists, designers, fashion historians and museum collections will be presented at the Electronic Visualisation in the Arts conference in July 2019.

REMOTE SENSUALITY
Digitising physical objects and materials is increasingly part of creative practices of all kinds, not just in the realms of entertainment or more interactive digital art forms.Here digital asset creation via CGI can produce very high levels of realistic surface rendering on forms that can be animated and articulated, lit and interacted with.These methods have also found numerous uses in the creation, visualisation and communication of fashion design and retailing, in particular for new designs not yet produced as physical samples.These can be visualised in highly convincing motion through digital pipelines involving Fashion CAD with digital 'materials' and 'physics' (CLO3D), 'light stage' scanning of real fabrics and materials (Change of Paradigm -Innovate UK 2016) and CGI Animation (Delamore 2015).
In the realms of fashion history and cultural heritage, the physical garment is the repository not just of its form, material and design properties, but also of the traces and history of wear that make each artefact unique and help to evidence its provenance.Museum conservation may seek to reverse the ravages of time in some instances, but in others the traces of use and event help to embody the ultimate significance -and life storyof a particular artefact.
The disciplines of Material Culture (Buchli 2002), Anthropology and more specifically Digital Cultural Heritage have long been 'digitising' their objects of study -and entire collections -via various forms of visible-and structured light 'scanning'.These types of 'no-touch' remote sensing use digital devices (cameras, scanners, etc.) to capture visual information, then rely on computer vision to reconstruct 3D form and surface appearance from the resulting image data.
It is amongst and between these various disciplines that 'structure from motion' (OpenCV 2016) -the principle underlying all photogrammetry -comes into its own.It is a form of remote sensing that can provide a photorealistic digital version of an original object without the intervention of its reconstructive algorithms making itself explicitly known in the final model -although they are still part of the micro 'decision-making' of digital reconstruction that lies beneath the surface.Perhaps more than m/any other remote sensing approaches, photogrammetry is reliant on good capture and processing techniques to produce an accurate, faithful and convincing 3D digital representation of any object.

Digitising garments
As artists whose practices are reliant on these technologies, we have been working collaboratively with designers, photographers, curators and historians to explore the potential of photogrammetry to accurately capture different types of garments, going on to use their digital doubles in various contexts, workflows, pipelines and experiments that break down the divide between the digital and material in practical, creative and conceptual ways.
The following Table 1 compares example results from within various environments and with access to different types of equipment.These are divided by type according to: (i) The relative motion between camera and object and (ii) The number of cameras available.
Resource demand, knowledge requirements and technical complexity also increase from left to right, along with more controlled and modified lighting; a larger number of cameras set on more complex supports; and with escalating methods of synchronisation and data handling involved.Our colleague Liam Leslie's (2018) experimental captures under conservation conditions initially only allowed 'stepping around' garments in ambient light within museum facilities (Table 1a).This provided far from ideal conditions for the evenly lit and shadow-less images required for effective photogrammetric reconstruction, resulting in fascinating dense point clouds of garments that were rarely seen outside of the conservation or study room, but less technically effective 3D models.Uneven surfaces, holes and interpolated sections resulted from the uneven lighting and relatively low image quality, but gave an excellent indication of the measures and methods that would be required for more accurate photogrammetric captures of these garments as part of on-going collaborations with the current research project Exploding Fashion (Evans & O'Neill 2018).
Earlier single camera, static object (step around) captures in photographic studio environments (Table 1b) had allowed for greater control of lighting with diffused overhead daylight and good internal reflectance from white walls providing conditions akin to the 'overcast sky' often cited as ideal for outdoor photogrammetry for archaeology and palaeontology (Mallison & Wings 2014).
Building on discoveries, methods and equipment for the capture of highly detailed objects in challenging lighting conditions (Dines 2017b) and recycling some of the infrastructure, we scaled these approaches up to create a capture 'cove' able to accommodate large objects in 'all around' lighting conditions (Fig. 2).Diffused and dimmable LED continuous lighting panels were reflected from walls built of white polystyrene 8'x 4' sheets to create a 'gas field' of even and very diffuse lighting from all sides.This was used to capture monochrome garments for Hussein Chalayan (Table 1c) using methods described in more detail below.
The final test context was within a professional body-capture rig for visual effects in the film industry (Table 1d); a capital, technical, knowledge and processing-power resource not generally available to most artists, designers, curators, or even major museums.This rig featured 115 cameras on 16 towers arranged around the central object/subject position; cross-polarised strobe (flash) lighting synchronised to fire simultaneously with all of the camera triggers; and integrated data downloading and file handing pipelines for the subsequent processing of the images into 3D models.

Digital material
Known for his innovative work with computer aided design and additive manufacture, Hussein Chalayan needed to 'digitise' a unique jacket that he had made directly in fabric using more traditional approaches.To produce a pattern would have involved unpicking the seams and flattening out the fabric, mirroring the challenge of reverse engineering garments in historic or theatrical costume collections.
Dark, monochrome, relatively featureless and with fine edges, the jacket did not lend itself readily to many methods of '3D scanning', falling back on photogrammetry for its flexibility and potential to adapt to a range of situations, scales and object types.Only two cameras were used for the capture, mounted on a vertical support at opposing angles to the object's imaginary 'Z axis' and each able to frame the whole of the garment.This gave sufficient overlap and coverage of object geometry to be able to reconstruct a reasonable model.Very diffuse lighting (Fig. 2) minimised shadows and glare, illuminating the garment from all angles.A series of two and three-dimensional coded and non-coded markers (Dines 2016, Biddulph 2018, Autodesk ReMake) helped overcome the challenges of a dark, monochrome garment where pixel colour did not change significantly over the surface of the object as it rotated in front of the cameras.These markers also helped the processing software to reconstruct fine edges; avoid confusion due to symmetry; and to rapidly and perfectly align all of the images from both 'rings' in a single processing 'chunk' or set of images (Fig. 4a).

CREATIVE REVERSE ENGINEERING
Fashion designers and pattern cutters have long generated the shape of new designs through calico 'toiles': life-sized fabric prototypes of garments constructed over 'dress' forms.Once evaluated and the final seam positions decided, these are deconstructed to help draw up flat patterns from which a garment is cut and sewn into its final material.This 'pattern-cutting' process is skilled and time consuming, requiring the translation of complex seam lines and 'curves' from a threedimensional form into accurate outlines, measurements and interconnections mapped onto flat pattern surfaces.
Chalayan had produced a half-life-sized calico 'toile' (Ashdown & Vuruskan 2017) as a 'maquette' for a new concept design.The capture process for the black jacket was repeated (Fig. 5a), but with the pale calico a closer match for the white background, requiring a less aggressive mask tolerance and some 'magic wand' selection and editing of mask bleeds across too-close tonal boundaries.The need to capture two monochrome objects, one very dark and one very light, in rapid succession was facilitated by our 3D coded and non-coded markers.Changes in contrast between the objects and background were managed via the masking process within Photoscan Pro.The resulting model (Fig. 5b) was decimated from 4.5 million to 1 million faces and exported as a set of obj/mtl/jpg files before importing into Rhinocerous CAD modelling software.
The photorealistically rendered model 'texture' proved helpful in preserving the 'look and feel' of the toile for the original designer, and in supporting subsequent collaborative digital design decisions.The initial half-sized model was rescaled to life size and the known dimensions of the 3D coded targets and coloured markers allowed dimensions to be readily checked using Rhinocerous' 'analyse' tools.
CAD specialist Peter Hill at the London College of Fashion worked closely with Hussein Chalayan to develop digital garment designs from the black jacket model, and to reverse engineer the toile to produce NURBS surfaces from the tailored fabric geometry.Seams were extracted from their 3D representations in the mesh model, and complex surfaces lofted from multiple sections of the scanned form.These were then converted into 3D printable shells and flat pattern sections for the production of the final garment in a combination of rigid, semi-rigid and flexible materials, via additive and subtractive computer aided manufacturing processes including Selective Laser Sintering in nylon powder (Fig. 7).

VISUALISATION & IDEATION
The ability to readily capture bodies, garments and artefacts to create accurate mesh models is in high demand for fashion design where CAD workflows are well established.At present the capture facilities available to individual designers and in education are relatively limited in scope.Outputs from specialist scanners tend to produce smoothed-out, mesh models best suited to NURBS-based CAD design workflows and avatarbased GCI visualisation (CLO3D).The detailed meshes of photogrammetric models are also able to provide accurate, real-scale shape data from which CAD points, curves, surfaces and volumes can be 'drawn' or extracted.Their photorealistic textures and material 'look and feel' facilitate more experiential design visualisation, supporting the flow between physical and digital, traditional and experimental methods.Our next phase of experimentation brought these mesh models into virtual-and augmented reality as a life size virtual 'canvas' for design ideation, sketching and prototyping.

Experiential visualisation
Previous experiments with games engines as onscreen 3D viewers for large photogrammetric models had been able to handle textured meshes with millions of faces (Dines 2017d), automatically loading into the scene in perfectly aligned chunks.Experimenting with 3D models of dress forms in real-time in a more interactive VR environment demanded lower polycounts to maintain refresh rates and overall performance.High-resolution 'turntable' and 'step-around' photogrammetry models of dress forms were re-topologised in open source Instant Meshes software and decimated to 40,000 triangular faces with a single texture file of 2048 pixels.These were able to load rapidly into VR to give photorealistic representations of traditional dress forms at life size.The ability to scale and locate imported meshes accurately in relation to real space and actual objects was also tested using a re-topologised toile.
Precisely overlaying the virtual garment onto an actual object in real space provided an underlying haptic form and size reference for drawing and further 'design sketch' iterations with VR tools.Initial trials with various third-party VR drawing and CAD software and plug-ins were conducted on HTC Vive and Vive Pro hardware via SteamVR runtime, including Google Tilt Brush, WIP RhinoVR and the VR Sketch extension for Sketch Up (Fig. 8).An overlay of the actual space captured via the HTC Vive Pro headset's onboard stereoscopic cameras allowed the user to 'see through' the headset via a video feed.The ability to visually locate and then interact with real objects accurately in near-real time allowed the user to 'trace' existing objects whilst wearing the headset, using the VR controllers to 'sketch-capture' real things.We also used both the Vive and Vive Pro headsets as chest-mounted-POV 'cameras', allowing the user to see the actual object with the naked eye, and the overlaid 3D models and traced marks via a mirrored feed to an external screen.This further emphasised the desirability and usefulness of a head up display with a wide field of view, or more locally mounted 'view finding' screen.
Experiments with AR technology required more decimated versions of our 'VR-ready' photogrammetric models, converted into FBX format for Microsoft HoloLens 1. Users were able to visualise 3D models of toile designs in real space at life size, and to 'project' them onto dress forms or actual bodies, but only within the limited field of view of the display, requiring viewing 'at a distance' to see the whole body/garment (Fig 9 ).

HAPTIC METROLOGY
Tracing real objects in VR and AR confirmed that the 'outside in' Infra-Red spatial tracking via SteamVR on HTV Vive hardware was understandably far more accurate and refined than the 'inside out' optical tracking of the HoloLens.We now needed to establish and physically reconstruct the position of the virtual tracking point in VR controllers to trial the touch measurement, or 'haptic metrology' of actual objects at real scale via VR CAD tools.
A slightly different virtual 'tool tip' position is defined by and for each VR application.The location of this tracking point within the physical boundary of the controller is not usually an issue, as establishing position, parameters and datums through contact with physical objects is not required beyond 'room set up'.However, for our Haptic Metrology experiments the location of the 'tool tip' inside the ring of IR receivers on HTC Vive controllers in SteamVR runtime resulted in a significant offset when 'tracing' real objects in VR.
An evaluation version of Valve's 'Knuckles' controllers had IR receivers in a ring around the outside of the user's hand, reducing IR occlusion and improving the tracking of hand pose.The virtual tool tip lay outside the 'body' of the device allowing us to locate the position of the tool tip of one controller in relation to the other in VR, then measure and replicate its real-world position using a physical object, providing a haptic equivalent to a virtual 'brush tip' through mechanical means (Fig. 10a).

Figure 10: (a) Adapted Knuckles controller 'follower' tool (b) Sketching on virtual toile at life size in VR
The adapted controllers accurately captured the tracing gestures made over the dress form using the VR headset as a virtual 'camera', head-or chest mounted on the user or hand-held by an observer.This streamed to a secondary screen whilst 'VR Sketch' recorded tool paths at real scale in CAD space.This extension for SketchUp was also able to load texture-less STLs of our photogrammetric models, and its virtual 'pencil' tool allowed very accurate drawing and sketching over the virtual model to create distinct CAD entities (Fig. 10b).

Augmented design
Using the HoloLens we attempted a rather 'close range' use of Fologram's (2018) augmented reality applications in Rhino and Grasshopper to explore a more haptic form of sketching and design ideation in AR.These apps are more usually put to work in architectural and sculptural visualisation, design interaction and fabrication at a far larger scale and distance.Here projections of CAD models can be overlaid onto real objects and scenarios without the limited field of view of the HUD causing major issues.
Our 'close range' users were not able to view the whole of the virtual object they were attempting to draw 'on', restricting the flow of sketching and ability to review what had been laid down.
Rescaling 3D models in HoloLens/Fologram was possible to increase what was virtually 'in frame' but this undermined the benefits of being able to see-and-touch things in real space and at life scale in AR.
Tracking the 'tap and hold' hand gesture in allows multiple curves to be previewed in the AR environment before being 'baked' into Rhino as mesh 'pipes' using Fologram's 'Mixed Reality Drawing Tool'.This was able to track relatively large-scale hand gestures smoothly when made in 'free space', but was less effective within 'arm's reach' of the real or virtual form.
Users were able to make sketch gestures in response to real and virtual objects, but the tracking limitations of the HoloLens hand gesture granted little ability to control absolute position in relation to an imported mesh or previous gesture.Having a real object in the field of view also interfered with the optical hand tracking where the linen surface of the dress form was close in colour and tone to the skin of the user's hand.We were not able to use the clickable 'spline curve' function in Fologram due to hardware issues with the separate device supplied with the HoloLens which did recognise other Bluetooth input devices, but only at 'menu' level.

Coded markers
Fologram's Grasshopper definition for tracking ArUco cubes allows for accurate and effective tracing of these computer vision markers using a cube corner as a reference point.The whole cube becomes a 'brush' or tool tip so that the resulting CAD element has a sectional width equivalent to the 3D diagonal, with the cube itself preventing the user from seeing any of the corners in contact with a real object.Fologram's more precise 'Marker Tracing' definition allows for an offset 'wand' emerging from the ArUco cube.The track of the tip of this wand could be previewed in the HoloLens itself and the accompanying Rhino viewport, although the frequent 'cross' position markers made detailed gestures less readable on the PC.Clicking to activate a conditional 'SendMeshes' component sends a relatively narrow 'pipe' to Rhino, 'baking' the path of the wand tip into the 3D CAD space.
We constructed a prototype, more ergonomic and controllable marker tool from a 60mm ArUco cube and a 5mm knitting needle (Fig. 11a).The physical tool shaft allowed a more familiar interaction akin to holding a pencil/pen/brush, and improved balance and weight distribution.Passing the needle through diagonally opposite vertices avoided the shaft penetrating and obstructing any of the ArUco marker faces and preventing them from being recognised, which would hinder the optical tracking of the cube pose by the headset.The position of the ArUco cube on the needle/tool shaft also allowed better visibility of the tool tip by the user, and less visual obstruction of the object being designed or drawn 'onto' in AR.Fologram projects a 'wand' from the corner of a designated 'A' face in relation to an origin at the centre of the cube, building on OpenCV ArUco Marker Detection of an imaginary Z axis from the centre of any marker.Our new sketching tool was modelled in Rhino to establish the vector transformation required to move this wand tip into line with the physical tip of the prototype tool.A number of custom CAD 'pen' vectors were input into the Grasshopper definition and physically tested with their adapted AruCo cube tool equivalents.
The resulting optical tracking relied on a steady hand and no vibration of the cube whilst tracing over the surface of the real dress form.A minimal offset of the cube from the pen tip was trialled, but this led to the occlusion of the tool tip by the ArUco marker for the human eye, rather than the HoloLens camera.Hand postures were also an issue, with the user's drawing grip sometimes too similar to the 'hand tap' gesture, sending conflicting signals to the CAD input.
The limited field of view of the HoloLens HUD made it hard to optically track the cube and see the whole of the object as 'head gaze' line of sight with the ArUco markers had to be maintained at all times.Successful interim trials were conducted with a user and an observer splitting the devices between them, with the observer stepping back to use the HoloLens as an AR head gaze 'camera'.This allowed for a wider field of view for the HUD to maintain optical tracking of the ArUco 'pen' tool, and a more 'natural' visual and haptic engagement of the user with the real dress form (Fig. 11a).The resulting tracing was accurate in terms of shape, scale and spatial location, albeit with a low-poly 'toothpaste' extrusion as an interim output in the CAD space (Fig. 11b).

NEXT STEPS
The ability to deploy trackable haptic tools in Rhino/Grasshopper for Fologram indicates the potential to define tool tip positions for spatially tracked controllers in VR games engines, and to generate parametric drawing 'tool paths' at real scale in VR for use in computer aided design.We are continuing to explore the new WIP RhinoVR in terms of its more refined CAD capabilities, in tandem with the advantages of enhanced spatial tracking in a prototype haptic VR 'capture rig'.

On-going
experiments with open source performance capture software Brekel OpenVR Recorder and HTC Vive Pro body position 'trackers' showed potential for the generation of live 'step-in' garment-avatars from photogrammetric captures of toiles, historic or contemporary garments and their rigging and animation in VR.A more 'actual' version of CLO3D's avatar-based garment design pipeline might be possible, allowing the integration of traditional craft and born-digital approaches, and the visualisation and augmented 'try on' of early designs and prototypes, toiles, and samples in motion.
In addition to increased efficiency, greater interoperability and 'flow' between traditional craft processes in fashion design, pattern cutting and garment construction and 'born digital' workflows and pipelines, the ability to dissolve the many barriers between them through forms of creative reverse engineering and haptic metrology could also support creativity and introduce 'degrees of freedom' not previously accessible in either approach.We are continuing to explore these potentials in collaboration with colleagues and students across disciplines at University of the Arts London, and in various museums, archives and study collections.

Figure 1 :
Figure 1: 3D computer coded target 'decoy' assisting in the capture of a 40mm, symmetrical, shiny metal object.

Figure 3 :
Figure 3: (a) 3D marker prototypes (Biddulph 2018) (b) reconstructed in textured model.Automatic 'replacement' masking in Agisoft's Photoscan Pro removed the white background and any static infrastructure features.Relatively aggressive mask tolerances (Mallison & Wings 2014) ensured no 'halo' of high contrast pixels or white points around the black object in the Dense Point Cloud.The resulting model (Fig. 4b) required no 'clean up', having smooth surfaces and good separation of the jacket from the paler dress form.

Figure 8 :
Figure 8: Sketching on a virtual toile/actual dress form.

Figure 9 :
Figure 9: Combining virtual / actual dress forms in AR.

Figure 11 :
Figure 11: (a) Close-range AR HUD observer (b) ArUco pen tool tracing seam lines on a real dress form in CAD.

Table 1 :
Photogrammetric models produced by methods of increasing technical complexity and resource demand.Screenshot images of textured 3D models all produced using Agisoft's 'Photoscan Pro' software.