All The Stars We Cannot See: A deep look into real-time satellite traffic

All the Stars We Cannot See (2020–2022) by Megan Smith & Yujie Gao, is an interactive immersive installation which geographically places the user in locations around the world so that they can look up into a virtual sky and see in real-time which satellites are flying overhead. With the satellite identification application, N2YO, tracking more than 23,000 satellite objects combined with the rapid increase of SpaceX launches, the artists are working to render visible an experience which will allow people to see the impact of the satellites in the sky, and to understand the presence of technology surveillance globally. This project is in development with UBC Visualisation Emergent Media Studio, with a proposed public launch on their state-of-the-art multi-screen immersive system in Spring 2021. The project will be adapted for a sensory enhanced VR exhibition and for an Augmented Reality application.


INTRODUCTION
This paper presents 'All the Stars We Cannot See', a new interactive art installation developed by Megan Smith and Yujie Gao.The practice-based research project began in 2020 with an investigation of satellite traffic orbiting Earth and it has developed into an immersive visualisation built in TouchDesigner, which pulls live data from the N2YO.comsatellite tracking server.The project tackles several big topics, first to deliver an immersed experience for the audiencewhere the physical presence of space traffic is rendered visual beyond the scope of a research paper, second to expose the extent of space traffic circumventing the globe, and third to enable people to have an informed opinion about the increased presence of space objects through sharing the accessible art work in exhibitions, web apps, journals and across social media.
A further goal, which ties with Dr. Smith's ongoing research is to continue to articulate the value of accessible tools as art-making devices, to question technologies and industry processes, and to tell stories about popular culture (Smith & Desnoyers-Stewart 2019), in this casemaking the somewhat invisible presence of satellite traffic visible.This paper will detail the process of building the visualisation, present other art works working on analogous topics, and touch on the work within the parameters of global space politics.
Technically, this paper will introduce satellite systems of orbit with the purpose to relay satellite flight patterns, density and presence in the space surrounding Earth.Considering that these satellites are constantly orbiting on a designated path, flying invisibly in the sky, serving many purposes, collecting and transferring data across networks, there is a colossal amount of data to be mined into a visualisation that demonstrates the power of this veiled space.Blending this data with an augmented and immersive experience will build a story with an ethical responsibility to reveal invisible critical issues and make what is invisible more tangible.

TECHNOLOGICAL CONTEXT FOR MAKING
Over the last decade digital media, virtual and augmented reality tools such as the VIVE and the Oculus (VR) headsets, gaming computers, tablets, iPads, Chromebooks etc., and DIY -MAKER toolkits, open-source software, combined with physical computing hardware have become more affordable and accessible (Smith, Desnoyers-Stewart & Kratzig 2019).This has affected the path of creative practice.With an ever-expanding presence of video uploaded to YouTube daily, Instagram and Snapchat extension into society and an activated culture of dancing TikTok influencers affecting the flow of consumerism and often seen as a source for education, it is not surprising to see more art projects built on these platforms, permeating devices, and using the IoT structures to build new art, new tools, and new stories.Amidst this frenzy of produce and consume, Smith and Gao pose the questionis this information fever distracting us from larger global stories, and what is one story that ties the fervour to a global experience?They settled on: "How many satellites are out there, anyway?" Working with their research question they discovered a number of accessible databases tracking Real-Time satellites that are regularly updated to keep up with new launches.NY2O and Heavens Above databases became useful tools for understanding the presence of satellite traffic.Of note: the authors/artists are not space scientists.The work they have developed is made through pulling readily available data into visualisations that can be understood in both qualitative and quantitative ways.
Their visual discovery, academically presented here for the first time, demonstrates a powerful message about the area surrounding the planet.As society is engrossed by media, the artists believe that through the visualisation of quantitative data in more accessible, dynamic, and interactive formats opportunities for learning present themselves.In a sense, their goal is to make this data accessible and to share it across multiple visual platforms as a method to articulate quantitative research to new audiences, and in doing so to build work that is as visually seductive as the popular content crossing social media networks in order to capture imaginations with science and art.

Orbital mechanics in the project
Satellites are space-based objects that move in an orbit.This research specifically visualises humanmade satellites that are orbiting Earth.These satellites follow engineered paths that are locked into orbit through Earth's gravitational pull.The North American Aerospace Defense Command (NORAD) generates an identification number for each known human-made space object launched into orbit.There are currently over 43,235 objects assigned identification, of which 22,453 are actively being tracked (N2YO 03/2021).Many items that are no longer tracked are considered lost in space or defunct.
Of the 43,235 orbiting objects the artists were able to drill down into the data to distinguish categories within.They can identify the satellites by their NORAD ID or other factors such as year of launch, country of launch or size of the object.These identifiers enable the artists to set constraints within their visualisation tool to ultimately tell stories.For instance, all satellites launched within the 1980s could be assigned the colour red, the 1990's blue, etc.
Locating the satellites as they orbit the Earth occurs through applying the mathematical calculation of three-dimensional space called the Polar Coordinate System where the position of a point is specified by three numbers: the radial distance of that point from a fixed origin, its polar angle, and the azimuth angle (N2YO).By combining the polar coordinate system with the desired constraints, the artists are able to plot various satellites into a two-dimensional visualisation.To track the satellites over time, the artists pull the satellite footprintlatitude, and longitude that are formatted in decimal degrees every 15-30 seconds, depending on the output they are trying to achieve, in real-time from N2YO into their program.This process builds up a series of data points that they plot on a digital globe which over time trace the routes of their objects.By refreshing the visualisation, the routes of the satellites are visible.The pinpoints move slowly and dedicated observation is required to see the change over time.The artists have also written into the program a trail that connects the data points.While this process may appear to be tracking the objects as they circumvent the globe it is in fact drawing a very fine and time controlled connect-the-dots path between the object locations.

TouchDesigner in the 'All the Stars' tool
Working with TouchDesigner, an accessible 'nodebased visual programming language for real-time interactive multimedia content, developed by Derivative' (TouchDesigner 2021) the artists built a toolkit which enables them to look specifically at satellite traffic through various parameters.They can transform real-time numerical data to imagery and pictorial composition, by constraining datasets to parse position, colour, time of appearance, or duration of tracking arrays.Working specifically with the Galileo set of satellites the artists have pulled real-time data with the Satellite ID, name, International Designator, launch date, latitude, longitude, and altitude.This gives the artists significant data to begin the process of visualising in TouchDesigner.The working process of designing this artwork within the TouchDesigner framework looks in part like the screen captures in Figure 2. In this situation, the Galileo satellites are being accessed in real-time to trace their orbital path every 30 seconds for a duration of 7 days.A trail was enabled to connect the points to build a timestamped drawing of the orbital paths.be drawn to illustrate the objects that have been lost, though they cannot be actively tracked.This second illustration would be comparable in density with today's NORAD numbers, at approximately half the total objects launched into space are now lost or defunct.As a method to illustrate the defunct and lost objects the artists could use the year that the satellites ceased to function and their last known position.While some companies are working to design their orbiting objects to deteriorate over time, or to plummet to earth where they can be disposed of or recycled (Huang et al. 2020), the artists can only speculate at the amount of space debris that has resulted from the thousands of professional and amateur launches of objects into space.

Ways of seeing space
'The Blue Marble' image produced in 1972 by astronauts of Apollo 17, and the subsequent satellite images created by NASA over the last five decades have been functional visualisation tools used to captivate the attention and imaginations of several generations of people.John Berger's 'Ways of Seeing' essays, also produced in 1972, were important tools in articulating the value of perspective and how art of its time helps to reveal and document cultural nuances and political change at the time of creation.Taking possibly Berger's most discussed topic of the gaze into context when reflecting on the impact of looking into space, or looking back from spacethese actions are generally associated with heightened experiences of sense of space and self-awareness; where the gaze imbues a topical focus, and generates a concept of knowing, which can be interpreted as understanding.However, when looking into the sky only renders clouds and blue sky, or a starry night with some occasional satellite movement, it can be difficult for humans to grasp the full spectrum of space traffic as the naked eye only detects a small fraction of the dynamic situation.In this sense, the gaze falls short without human intervention to increase scope.This shortfall of the human eye leads to reliance on information from other sources which can ultimately mean that information about space is produced by those organisations and people who have access to satellite data, can read it, or manage satellite images stores and curate their content.In essence, institutions like NASA, who broadcast their visual data and satellite imagery curate the space content that is shared.'All the Stars We Cannot See' arrives on the art scene with these established tools as its guiding posts, and an established human understanding of the basics of perspective of Earth from Space.Nevertheless, this work does not break any communication ground in terms of identifying the mysteries of space, but it does tap into the otherworldliness that is generated with space-based visuals, like 'The Blue Marble'.Where 'The Blue Marble' enables humans to identify with the size of Earth, its difference between land and water, it's cloud and ice cover, in a similar way 'All the Stars We Cannot See' opens the eyes and psyche to the density of satellites orbiting Earth.The work also opens up the mysteries of satellite density, and orbital paths of various satellites by rendering visualisations of these paths.Coming back to the discussion of accessible tools, the work draws on established software and spreadsheets of information to compile the quantitative data into a project that can be understood through qualitative means, by setting the eye or the gaze on the visualised data.

Experiencing satellite transmission
Artists working with satellite transmission have been doing so since the 1960's with perhaps 'Our World', produced in 1967 by the BBC, and managed in Canada by Marshall McLuhan broadcasting at Expo 67, being one of the leading moments in curated content across this medium.The rules of the initiative, which synched communication across 14 countries, and 5 continents, included a real-time broadcast.The Beatles wrote and performed 'All You Need is Love' for this event.Over 10,000 technicians were employed to work on the project that utilised 3 different satellites to transmit the program.(CBC Archives, 2021) While the complexity of the undertaking may have overstated the combination of projects presented, this did open up exposure of the medium to artists.Kit Galloway and Sherrie Rabinowitz developed 'Hole in Space' in 1980, a live-feed happening between New York City and Los Angeles.Nam June Paik developed 'Good Morning, Mr. Orwell' satellite-based artwork in 1984, and fast forward to 2016 and Dan Goods has developed 'Orbit' a sculptural installation which acoustically transmits real-time satellite sound.'All the Stars' We Cannot See' draws on these works to situate its importance in relaying space traffic to the world.Like most satellite artwork, it also relies on military research as inspiration for making.Somewhat like Rebecca Allen applied the military's Movie Map technique of photographing space in all directions approximately every ten feet, usually done from the rooftop of a car.She worked this technique into the 'Aspen Movie Map ' (1978) which garnered the attention of the public and sparked their imaginations.These acts ultimately led to the technique being applied by Google in the making of their Google Street View and their Maps archive.
Among the various outputs planned for 'All the Stars We Cannot See' the artists are working to make the installation feel immersive and larger than life through a partnership with the Visualisation Emergent Media Studio (VEMS) at the University of British Columbia.The work will be installed on the VEMS room scale rounded screen wall at the UBC Okanagan campus.As this work will be the first major project produced at VEMS and the content is being controlled for press release in Spring 2021, no images were ready for print at the time of writing.A visual articulation of the project involves the relaying of the real-time feed across the screens.The floor of the studio will be covered in grass and picnic blankets will be scattered to build a sense of 'starry night sky watching'.The installation room will be dark and the experience when watching the real-time feed will be of satellites slowly inching across the screen with locational feed updates coming every 30 seconds and a stardust trail leaving digital residue to mark the passage of time.The longer a person spends in the artwork the more satellite traffic they will witness.They will also perhaps begin to decode the colour details of what they are seeing.The artists have not yet settled on their colour coding by year or by country.Design for virtual reality and augmented reality is also underway and will follow closely behind the VEMS launch.In the first instance the project will be ported into Unity and launched for VR by summer 2021.The artists are currently seeking funding to develop the project as an AR app in Fall 2021.
A series of two-dimensional digital images, and small time-lapse animated gifs will be produced for social media (Instagram SnapChat, TikTok) as educational tokens, fulfilling a major goal of the projectto produce this data in dynamic and seductive ways which capture the attention of social media users.A high-quality archival ink print series will also be produced for exhibition ready presentation of the project in combination with the immersive VEMS installation.All of these digital tokens will reference particular sets of satellites as they orbit the Earth on specific days.The Galileo network will become a set, as will a Starlink satellite series.These latter satellites currently form the most populated active satellite category.Their unique linear formation which links 60 objects at a time form lines in the sky like no other satellite networks.This makes them highly visible as a different form within the visualisation.Satellites classified as Military will be logged, as well as Amateur Radio.Countries as categories and years will also be developed as images.At this early stage of creation, the artists are not limiting themselves to too many classification constraints as through the process of visualising this data they are working to expose the extent of space traffic and its complexity circumventing the globe.

CONCLUSION
This project through its active steps to visualise space traffic should help to broaden knowledge of satellite density in space, and make visible what is essentially a veiled space.In particular, by releasing images like figure 5. publicly the artists anticipate that for some the reaction to this image will be similar to that of seeing 'The Blue Marble' for the first time; a mix of shock and wonder.Their aim to circulate the research through multiple interaction spaces ranging from fully immersive room-size visualisation to small animated gifs across TikTok will push the research out into many demographics.The artists further anticipate that the project will stimulate conversation on the pollution of space and on surveillance, and that this project could serve as a visual aid to trigger a reaction on those topics.The artists hope that through their publishing of this research through popular and engaging means they can relay the expanding situation of the occupation of space, made visible now to the naked eye.

Figure 2 :
Figure 2: Data flow in TouchDesigner, numeric data transformation to image data.Using the N2YO live database the artists pull satellites via a REST API into various categories.The diagram below details how the raw JSON data is parsed through their visualisation process.Categorisation based on position, and launch dates are used to visualise the Galileo satellites in the example.The colours identify the years of launch.

Figure 3 :
Figure 3: Data parsing and visualisation graph for TouchDesigner system.
Figure 4. Details the path of these satellites orbiting Earth over multiple days.

Table 1 :
Example of Galileo satellites' raw data parsing to table.