Innovative Strategies for 3D Visualisationusing Photogrammetry and 3D Scanningfor Mobile phones.)

3D model generation through Photogrammetry is a modern overlay of digital information representing real world objects in a virtual world. The immediate scope of this study aims at generating 3D models using imagery and overcoming the challenge of acquiring accurate 3D meshes. This research aims to achieve optimised ways to document raw 3D representations of real life objects and then converting them into retopologised, textured usable data through mobile phones. Augmented Reality (AR) is a projected combination of real and virtual objects. A lot of work is done to create market dependant AR applications so customers can view products before purchasing them. The need is to develop a product independent photogrammetry to AR pipeline which is freely available to create independent 3D Augmented models. Although for the particulars of this research paper, the aim would be to compare and analyse different open source SDK’s and libraries for developing optimised 3D Mesh using Photogrammetry/3D Scanning which will contribute as a main skeleton to the 3D-AR pipeline. Natural disasters, global political crisis, terrorist attacks and other catastrophes have led researchers worldwide to capture monuments using photogrammetry and laser scans. Some of these objects of “global importance” are processed by companies including CyArk (Cyber Archives) and UNESCO’s World Heritage Centre, who work against time to preserve these historical monuments, before they are damaged or in some cases completely destroyed. The need is to question the significance of preserving objects and monuments which might be of value locally to a city or town. What is done to preserve those objects? This research would develop pipelines for collecting and processing 3D data so the local communities could contribute towards restoring endangered sites and objects using their smartphones and making these objects available to be viewed in location based AR. There exist some companies which charge relatively large amounts of money for local scanning projects. This research would contribute as a non-profitable project which could be later used in school curriculums, visitor attractions and historical preservation organisations all over the globe at no cost. The scope isn’t limited to furniture, museums or marketing, but could be used for personal digital archiving as well. This research will capture and process virtual objects using Mobile Phones comparing methodologies used in Computer Vision design from data conversion on Mobile phones to 3D generation, texturing and retopologising. The outcomes of this research will be used as input for generating AR which is application independent of any industry or product. 3D-AR


INTRODUCTION
This research aims to provide preservation of real objects in 3D. An object based cost effective preservation and a 3D reconstruction pipeline using Mobile Phones. Single pipeline would enable customisation of data from taking photographs from mobile phones, 3D model generation (pictures->point cloud->mesh), 3D model texturing, patching model, retopologising, location based interactive AR simulation and digital archiving. This research represents the unique pipeline which would allow user to build their own 3D models in AR with one mobile app. This paper underlines the current stage of the research which is the 3D model generation referred to as Photogrammetry workflow. A key aspect of this research is that it's not just theoretical, but a major portion of this study involves the practical implementation of this pipeline. The first stage of this research has been implemented for this study. For the scope of this paper, C++ computer vision libraries are used including Open Multi View Geometry (OpenMVG), Structure from Motion (SFM) sequential pipeline and Multi View Stereo (MVE) for implementing the first stage of the pipeline. The rest would follow in later parts of the research to implement this pipeline.
The photogrammetry technique generates a 3D Model from multiple photographs of any real world objects. (Daneshmand, Helmi et al. 2018). It is more suitable to generate 3D models using photogrammetry which are otherwise hard or time consuming to create using 3D software like Maya,3Ds Max etc. There are a number of photogrammetry software pieces which allow one to generate models from photographs for free but some are quite expensive and also require extra hardware to scan the object (Lansard 2018) . Even after using these software pieces a manual preprocessing on the images and post processing like mesh clean-up is done by skilled 3D artists. This research aims at reducing these costs and providing a customised application to generate models using pictures taken from a mobile phone.
Photogrammetry is widely used in Game arts to generate models using low budgets. Companies like NVIDIA have coordinated with Reality Capture (Paid photogrammetry software) to give a wide pipeline for mesh generation using photographs. (Bishop, Cowan et al. 2017). Although after signing a NDA, one can use Reality Capture SDK using a paid licence of any of their products. Unity 3D has developed their own photogrammetry workflows but only using Photoscan (Paid photogrammetry software) as the main skeleton to the flow.

BACKGROUND
Laser scanning and photogrammetry are used for generating 3D models at high costs in terms of the equipment required. They also require extremely powerful computers for processing the models (Staff 2017). Even after getting the 3D scan, a lot of work needs to be done on the scanned object model such as unpatched areas in the model, high topology of the mesh. All these factors influence the reusability of the mesh. This project, in principal, aims at creating a free of cost mobile app which would allow users to generate their own retopologised, patched textured models and place their representation in an interactive AR. No additional hardware will be required, just a mobile phone with the app (in process) would let users to develop their own 3D content in an interactive AR simulation.
A large and growing body of literature has investigated the development of 3D scanning and Augmented Reality applications for PC as well as mobile phones (Nobrega, Jacob et al. 2017). Previous studies have demonstrated significant advantages of using these processes for education, retail, cultural heritage preservation and more but they are mostly product dependant. For instance a model created for educational purposes might not be reusable for heritage preservation. A 3D AR model created for a furniture simulation might not be a usable asset for a game. The research to date has not been able to establish a product independent single pipeline which will account for these aspects of 3D scanning, AR and all that comes in between. This study involves generating a pipeline with customised 3D assets (textured, patched, retopologised) from scratch which is collaborative amongst multiple industries and its interactive AR simulation for live demonstrations and archiving.

PIPELINE DESIGN
Prior to commencing this study, assuring research was done to establish a single pipeline which would encompass tasks from generating a 3D model all the way through representing AR visualisation.

Figure 1: 3D-AR Reconstruction Pipeline for Mobile phone Application
This 3D-AR reconstruction pipeline (Figure 1) has 4 main stages which are described as following. The first stage of this research (Photogrammetry workflow) has been implemented and the results are discussed further in the paper.

Photogrammetry Workflow
The first stage is to take pictures and videos from a mobile phone. The process of photogrammetry involves generating digital assets using overlapping photographs from different angles. Once enough photos are captured, image processing algorithms are applied to these photos to identify and define common objects and features. On the basis of these features, extractions are made and a 3D Point cloud is generated. The 3D point cloud is then further solidified to generate a mesh. Using vertex colours a textured 3D Model is generated which acts as an input to the next stage.

Retopologising
In the second stage, the 3D Model from the Photogrammetry workflow is used as an input to the amplify imposter. This phase allows the post processing of a 3D model to make it a complete low topology patched model. Since it is quite common to have broken or high topology meshes in the results. The amplify imposter measures overlapping angles and assigns overlapping images depicting a low topology mesh when in actuality it is just a common overlapping image combination to match the actual object.

In App 3D View
The third stage is the display of 3D retopologised and textured meshes. This Mobile application module is to show the user the digital results of the 3D mesh generated using the pictures captured. The user can view in 3D the digital object with details of the model and texture.

AR Kit
The fourth stage is to let the user utilise this 3D mesh and view it in AR. The images of the object may be taken from one place but will provide an AR view to match any environment and place. Also a GPS location will be associated with the 3D object for anyone to know where the actual object was located. (Hsiao, Rashvand 2015) 4. METHODOLOGY

Test Data
The first phase of the pipeline Photogrammetry workflow was implemented using the Pierre Moulon image data set of Sceaux Castle. Various algorithms have taken Sceaux castle data as a benchmark to test the quality and accuracy of the mesh. The Sceaux castle data set and the resultant mesh quality helps to validate the accuracy of the solution presented in this research.

Input
The first step of the Photogrammetry workflow is to take multiple overlapping images of the same focused object. The first development iteration of images used in this research are not taken from a mobile phone but a known data set as mentioned above. Multiple overlapping images provide good vertex points for image processing algorithms.

Process
Step 2 of the Photogrammetry workflow is image processing consisting of 4 major operations as mentioned below.

• Image concatenation
This process joins the image with the neighbouring image and pairing them until the last is paired with the first one. The resultant image set is a combination of paired images. In this example image 1 is paired with image 2. Image 2 is paired with image 3 and so on until the last image is paired back with image 1.

• Image feature matching
This process draws lines on areas of common interest in the concatenated pairs. Thus every paired image has several lines drawn from one image to the other in the pair. Lines are drawn on the marked pairs of the previous extracted features that is lines on pair (1,2) and (2,3).

• Computing point cloud
This process performs sequential construction on the components paired by extracted points. Once the construction is done in 3D point space it uses the point information to colour the vertices and camera poses to calculate the angle position to map the images for construction.

• Computing mesh
Converts the image format to MVE file format. Computes and constructs dense depth maps for the point cloud. Reconstructs dense depth maps and reconstructs a surface mesh from the depth maps.

Output
Step 3 of the Photogrammetry workflow is the successful reconstruction of a colourised 3D mesh.

RESULTS
The first set of experiments examined the impact of using core Computer Vision libraries. The combination of OpenMVG, SFM and MVE has resulted in some usable 3D meshes. As shown in Figure 11and Figure 12 , results obtained from the introductory procedures show no significant change in the quality of the mesh. Only trace amounts were obtained from these results, the point cloud evidently speaks of the results and so does the 3D mesh. Where as shown in Figure 13 , striking results were established on the same data set. An interesting observation emerged from the results not just in terms of the quality of the mesh which is very visible but also there was a successive increase in time and processing of the final mesh. This indicates strongly that, from the processing of these meshes on a PC, and by creating a middle plugin for these libraries, the entire mesh processing can be done using Mobile phones.

STAGE OF RESEARCH
Current implementation includes successful generation of a 3D mesh using Computer Vision libraries. The first step of the pipeline Photogrammetry workflow was tested using the processing and memory of a PC. The successful time saving approach gained by this research proved evidently and now these procedures can input as test scenario for the development of the mobile app pipeline which would directly take captured images and process a coloured textured mesh completing Mobile Photogrammetry workflow.

DISCUSSION AND CONCLUSION
Present study was focused on designing the unique 3D-AR Reconstruction pipeline. This study set out with the aim of assessing current trends and development in both the field of Photogrammetry and AR. As mentioned earlier, this research intends to provide practical implementation of the suggested pipeline along with theoretical study. The research results gained by PC Photogrammetry workflow through this paper clearly demonstrates the effectiveness of transferring the techniques to Mobile phone Photogrammetry workflow marking the success of first stage of the pipeline. Future work includes practical execution of the remaining stages of the pipeline for a mobile phone. Along with this, work needs to be done for the integrity, accuracy, processing and time cost evaluation of the generated 3D mesh and its interactive visitation in AR. There exists a lot of photogrammetry and AR apps on the market, but the unique pipeline, described in this paper, will allow the users to create and publish their own content in 3D based AR, free of cost with no additional hardware. The initial objective which still withholds the crux of this pipeline is to use the power of mobile phones to create 3D content which would be as feasible as taking photographs from a smartphone.
In future development, successful implementation of the pipeline would result into potential application of reusing these 3D-AR assets for instance in heritage preservation, assets for mobile games, retail sector, food industry, educational sector, independent 3D-AR archiving and more. Relevant studies show a drastic development in this domain but unfortunately all the development is either product or domain dependant. This research tends to provide this technology for everyone to be able to use this for personal or professional use without having to go through costly ways of demonstrating their products.
The next stage of this research is to ensure the quality of the mesh including retopology, patching the model and interactive simulation for location based AR. So people can also view the location of the model with the model in the digital archives.