How We Harnessed the Potential of UAV Imagery in the Classroom through VR and Other Technologies

Moving from a project workflow of 3d modelling to print, 3d print, milling, and model building, the Knowlton School has found emerging technologies that can be integrated into the curriculum and student experience. UAV scans can bring accurate site plans and models into the creative process while virtual and augmented reality can provide a more immersive way to interact with the project. This paper provides the steps to identify what technologies fit your environment and how to move towards adoption among faculty and students.


INTRODUCTION
Technology is a core component of curriculum in today's design schools. There are content creation programs such as Rhinoceros, Revit, and ArcGIS (as well as Maya, Sketchup, InDesign, Illustrator, and many more) used to generate digital projects. Then there are ways of producing your final project, such as 3d printers or plotters and fab labs. Design schools make things in the digital realm to be viewed and explored in the physical space. At first glance, all these technologies stand on their own in a specific area as a tool for creating or exploring content. What is not always clear are the threads that connect all of these technologies for means of creating potential paths for Augmented Reality (AR) and Virtual Reality (VR) into the classroom. While each institution is unique, this paper will discuss the tools we used to create a flexible and sustainable resource that students are able to utilise when approaching their research questions in unique ways.
Let's set the stage. The Digital Library, as part of the Knowlton School at The Ohio State University, supports the programs of Architecture, Landscape Architecture, and City and Regional Planning (CRP). While an archive, we partner with the College of Engineering IT team to explore emerging technologies that could be used to support the educational mission of all three programs in some fashion. This fulfils my mission of not only supporting academics, but creating new and interesting projects that can be fed into the digital archive.
As of Fall semester 2017, the workflow for all three academic sections was pretty simple (Figure 1). The content was pulled from various sources, using a combination of Rhinoceros, InDesign, Illustrator, and ArcGIS to create their project, and then the final project was either printed, 3d printed, milled, or hand-built (often all four). The process was simple but limiting and stale. That is when the first project came across my desk in the form of a question. "Can we use drones to capture sites for projects?" There were too many regulations and requirements involved for IT to take on the project. However, it is my job to attend to these types of requests and explore the ways we can grow to build resources around a big picture.
Before taking on any new project, I ask five questions to gauge how to proceed. These questions are: 1. Is the technology mature enough? 2. Is the technology useful enough? 3. Is the technology flexible enough? 4. Is the technology sustainable enough? 5. Is the technology cheap enough? While these questions help guide the conversation, the term "enough" is very fluid and changes often. I approach this with an understanding that there is no right or wrong answer, or even a better or worse option. It is simply a matter of determining the better option for that particular moment. What do I mean? During the process of researching and writing this paper there have been some huge advances in the field of AR/VR. Namely, the Microsoft HoloLens 2 was announced at Mobile World Congress 2019 and Oculus continued to tease additional possibilities for their stand-alone unit, the Oculus Quest (Bohn 2019). Both of these technologies have potential to redraw the lines for my project between AR and VR, but this is something that was unknown three months ago. Which brings me back to the five questions. Lean into the term "enough" and find what fits best for you and your institution now with an eye on the future.

SPECIFIC CHOICES
In the case of Unmanned Aerial Vehicles (UAV) for the classroom, it was clear that a DJI Mavic Pro and Pix4D software for processing would allow us to create highly accurate 2d maps and 3d models of sites that students could then use as the foundation for their projects (Pix4d 2019). CRP started using the UAV surveys to inspect transportation corridors on a large scale to discover poverty disparities between neighbourhood's surrounding campus. Architecture students could use the 3d site model (exported from Pix4D as an OBJ file) to plan their play equipment that they were building for Columbus Metro Parks as part of a Spring studio. It even went beyond RGB imagery and now we are exploring microclimates for Landscape Architecture by mounting a Sentera NDVI camera to one of our DJI Mavic Pro as well as a custom-mounted Flir Duo R on the other. One year into the UAV experiment and the data is being used in a number of different ways. Now what?
Looking at the initial workflow, UAV imagery went before Rhino and was all about data input ( Figure  2). Our next exploration involves fixing our stale output problem. Two important items led to our focus on providing additional options for output. First, the New Media Consortium's (now part of EDUCAUSE) Horizon Report in both 2016 and 2018 mentioned Virtual Reality and Augmented Reality as a technology that will impact education in the future (Johnson et al. 2016, Becker et al. 2018. Second, an internal survey of our students found that the average Architecture and Landscape Architecture student spends between $300-400 per studio per semester in printing and supplies. Some students reported spending more than $1000. This cost is only going up and students are desperate to find ways to showcase their work in a novel way that will not cause them financial hardship. For us, embracing VR makes sense.
However, VR is complicated. Going back to the Horizon Report for Higher Education, the 2016 edition places AR/VR into the two to three years time-to-adoption category while it disappeared from the list in 2017 altogether. It then re-emerged in 2018 in the four to five year time-to-adoption category. The reason for this instability is that while it continues to show promise, the technological barriers have taken considerably longer to overcome than anticipated. Most significantly, you still have to be tethered with a long cable to a highend graphics card in order to run your $500 VR headset in your dedicated space of at least 3 square meters for optimal performance. These limiting factors have created a threshold for adoption that keep the consumer away and have developers struggling to create useful titles.
But change is on the horizon. With prices of both VR units and graphics cards declining, as well as the inclusion of inside-out technology that drives Windows Mixed Reality and improves freedom of moving spaces, we are finally at a place where we at the Knowlton School can move forward with our experiment. The plan is now to find the hardware and identify the individuals who can take the technology, create an experience, and spark the imaginations of faculty and students with a demonstration of possibilities.

FIRST STEPS
The startup costs for VR were very inexpensive as we were able to repurpose some of our rendering computers that already had the NVIDIA GeForce GTX 1080 Ti graphics card. We only had to purchase the Lenovo Explorer headset that runs Windows Mixed Reality (WMR) and download Enscape and Unity through free educational licenses. While WMR is less proven than Oculus and HTC, it's Inside-Out technology was a better fit for our needs of having to relocate the VR device into different spaces throughout the creative process from concept to final review. With WMR being directly embedded into the OS of each Windows 10 computer and supported by multiple hardware manufacturers, a $250 investment was a great way to start.
More important than hardware are partners and ambassadors. This is where networking, listening, and looking play a key role. One of my graduate assistants, Lawrence Servis, was working on a project in studio that was perfect for a demonstration. He and his partner, Samuel Tibbs, were designing a Virtual Reality Resort for Chile. Samuel, who is very passionate about VR and the future of E-gaming, was quick to volunteer for the project. I provided them access to a high-end computer, a VR headset, and the assurance that they could get whatever software they needed, and we were off. The idea was to take what was already created, and find ways to push the experience to VR in a way that others could easily replicate in the future. As the buzz for this project grew, interest and support continued to build as the firm that Samuel worked for, SHP, also provided testing equipment and expertise as we moved towards demo day. HP also provided their version of the WMR headset and their VR Backpack for us to test. Neither of these pieces of equipment were useful but it allowed us to try new things and make appropriate decisions.

HELLO WORLD
The final step was to plan the demonstration event where faculty and students would have the opportunity to see and experience the possibilities of VR. Given the importance of this demonstration, it was key to have a clear message about why this technology is worth exploring. It is necessary to have a concise pitch that answers not only what you hope to accomplish but also why it is important and what the benefits are. Provide the big picture with attention to the small steps that make it possible. Our pitch was as follows: "Virtual Reality will allow students to experience their project in an immersive way with few extra steps and only a small monetary investment on behalf of the school. While perfect for the exploration and creation of a project, it can also be used for visualization during reviews to reduce the reliance on costly and wasteful physical products like plots and 3d prints." Based on the feedback for our pitch, the response to our presentation and the energy of participants at the event, the technology was received with excitement. It also hit my success milestone one semester early as it was already integrated into the final review of a student project the very first semester. In addition to buzz at the Knowlton School, our VR initiative also has received notice and support at the college level. The College of Engineering is now asking how this technology could become a resource accessible to all students. We do not have all the bugs worked out yet, but our small investment has paid dividends.

OUTCOMES
Four months later we have found a way to integrate VR into student projects as well as a review, albeit on a small scale. Regardless, the proof of concept is valuable while finding sustainable ways to integrate the technology. We have created workflows that have added content to the creative process via UAV imaging and Pix4D, as well as found ways of using different types of screens for different purposes (Figure 2). It has also provided the perfect opportunity to stop and research direction. Technology moves fast in the Spring around the Consumer Electronics Show and other trade shows. Reflection is important when exploring emerging technologies. As previously mentioned, Spring 2019 had some big announcements in the realm of AR/VR. While some of the news, like the previously mentioned Microsoft HoloLens 2 and Oculus Quest are moving the industry forward, some news wasn't as positive for our project.
With Microsoft pouring all of their resources into AR, they were signalling a withdrawal from WMR. Headlines such as "Has Microsoft given up on Windows Mixed Reality?" and "Microsoft has dropped Windows Mixed Reality devices from its affiliate program-are they dead already?" cast doubt on VR and wide application (Hanson 2019, Caldwell 2019. In an article with Jamie Feltman of UploadVR, Greg Sullivan, director of communications for Microsoft's Mixed Reality arm, said "interestingly, the immersive VR headsets were pitched as consumer devices and I think did not meet, in general, the high expectations that were set for them there. But what you're seeing now is more and more commercial applications for those types of experiences" (Feltman 2019). While it may appear we placed our investment in the wrong brand, it did highlight two important things. First, it is applications such as ours that VR most benefits. And, AR is becoming more approachable for most people in an interactive setting as seen in Microsoft's HoloLens 2 release presentation at Mobile World Congress 2019 (Microsoft HoloLens 2 release 2019).
Adding to our discussion inside Knowlton School, a student-led and faculty-supported interactive exhibition took place in February 2019 as part of the "Forested Exhibition". Students created a virtual forest inside our building using Unity and Vuforia. In doing so, they completed the finals steps of our possible workflow, integrating projects created in Rhino with AR projection in the physical space. While separate from our project of moving from UAV to Virtual, it gave me ideas of creating possible paths as well as a final project to showcase the whole workflow. What were they?

FINAL SHOWCASE
The final project and workflow is designed to showcase how every piece of technology in our building can be used together to create various alternatives to how projects were created even just a few years ago. The project we are focusing on now is key because it is a unique Columbus Metro Park property that is still not open to the public and has an air of mystery surrounding it because it is an old stone quarry. It is also a difficult site because of its sheer cliff walls, forests, limestone ponds, and 350-acre size. If we can do this project well, it represents the hardest type of workflow we would encounter on both ends.
Over the course of multiple trips, we made 20-30 flights gathering different images from different weather conditions and angles to create a usable and interesting 3d model. Those flights also served as training flights for new student pilots, which further supported the growing mission of UAVs use inside the classroom. The 3d model was then taken into Rhino and artefacts were removed to make it ready for the next step of the workflow. This is where the potential for our programs really begins to shine. We took the 3d model and had it carved by the 5-axis mill into a 2'x2' piece of Styrofoam to create a site model for the "class" project. This created an AR target that could be used by all "students" in the class during a review. Then using that same Rhino file, various Digital Library student employees set to work creating content for both VR and AR. For VR they could work directly inside Rhino creating a building or site plan and launching Enscape periodically to see how the space works to scale in VR. They could also take the Rhino file (via OBJ file type) into Unity and export a standalone application file of the site for VR to allow users to virtually walk through the 350 acres. Or they could create looping video experiences for one of our TV carts for studio reviews. Lastly, they could take the Rhino file into Unity and export a photo layer into Vuforia. Once loaded into Vuforia, a phone or tablet could be pointed at the milled Styrofoam quarry and the project would appear layered digitally on top of the scale model. Better yet, an entire studio could point their own devices at the targeted model to view each project en masse. This would allow it to become a practical and useful part of studio reviews.
While many of the lessons are still being learned and this project is still awaiting its big reveal, we have taken our out-dated workflow and created a sustainable and flexible workflow (Figure 2) that can be easily adopted in our programs and beyond. Total monetary investment to get the basic experience was $3,450 ($1000 for a DJI Mavic Pro, $1900 for Pix4D instructor license, $300 for a VR graphics card, and $250 for a WMR headset) but its value added to the classroom far surpasses that amount. It is hard to predict where this technology is going in the next year or two but as the technology matures, we will look back and see how UAV and VR/AR revolutionised the classroom experience at Knowlton School in the years to come.