The Limits of Algorithmic Perception: technological Umwelt

The algorithmic processes underlying digital media entail that elements of the unseen are integrated into processes of visualization. What is visible in a digital image, for instance, is merely the tangible output of procedures which are obscured from view. Borrowing from ecological theory, using Jacob von Uexkuell’s concept, the Umwelt, this paper explores synergies between notions of biological perception, cybernetic sensing and Harun Farocki’s concept of the operative image. Working from this conceptual framework, a technological notion of Umwelt is developed and applied in analysis of related computer science research and relevant artistic projects. The transdisciplinary approach applied in this investigation aims to frame artistic experimentation with algorithmic mediation of perception in reference to the technoscientific context these practices emerge from.


INTRODUCTION
The ecological concept of "Umwelt" describes how the parameters of a given perceptual framework link together an agent's ability to take in and to act upon perceptual information, which has new significance when applied to technologically mediated forms of perception.Applying this concept to the development of a technological notion of Umwelt, this investigation considers how the parameters of human perception relate to those of technologies which complement, augment and automate visual processes.Starting with an introduction to Jacob von Uexküll's theory of Umwelt, this concept is then compared with related ideas from cybernetics and then used to analyze relevant examples of research and art projects.This paper lays out a conceptual framework through which to understand cybernetic perceptual relations and the experimental methodologies employed to investigate the boundaries between the visible and the invisible.

UMWELT
In a seminal text of ecological theory, A Stroll Through the Worlds of Animals and Men: A Picture Book of Invisible Worlds, Jacob von Uexküll built up the concept of "Umwelt", which he defined as the perceptual world of an organism.The Umwelt has a two-fold nature informed by the perceptual apparatus of the organism in tandem with the organism's situation as an agent within its environment.The organism's engagement with its environment hinges on sense, on the one hand, and agency, on the other.Each carrier of significance is thus bound in a circular relationship to the perceivable world via an organ of perception and the organism's contingent ability to act upon the sensory information it takes in.
The Umwelt of the tick has been explored at length, its relatively limited sensory abilities making it a useful subject for the study of sensory perception.A tick has few sensory organs, but those it has are finely tuned to particular information about its surroundings.Its carriers of significance are relevant sensory phenomena which act as biosemiotic triggers to perform biological functions ranging from finding food to reproducing: the odor of the butyric acid contained in the sweat of all mammals; the temperature of thirty-seven degrees corresponding to that of the blood of mammals; the typology of skin characteristic of mammals, generally having hair and being supplied with blood vessels; (Agamben 2004) Each perceptual cue (merkmal) is conjoined with the tick's perception of the world through its sensory organs and its corresponding agency to act upon that impetus (wirkmal) (von Uexküll 2009).In the absence of perceptual signifiers, the tick ceases to exhibit behaviour linked to respective stimulus.Phenomena which lie outside of the parameters of the tick's sensory organs go unnoticed and, therefore, un-acted-upon.
The whole rich world around the tick shrinks and changes into a scanty framework consisting, in essence, of three receptor cues and three effector cues -her Umwelt.

(von Uexküll 2009)
Figure 1: Effect Circle (von Uexküll 1926) Establishing the limitations of a given perceptual framework offers an entryway to better understanding its inner workings.In order to render perceptual blind-spots visible, it is necessary to determine the boundaries between what can be established as perceptible from that which is imperceptible.Taking behaviour as an indicator of perception, von Uexküll made connections between the actions of an organism and what it is able to perceive.For example, in experimentation with tick perception, it was established that a tick will readily attempt to consume any liquid which is 37 degrees Celsius.The tick's attempts to drink water as though it were blood reveal that the tick lacks a sense to discern the difference.In such imperceptibility made tangible, it is made manifest that the perceptual apparatus of an organism is directly linked to its agency within the perceptual world it inhabits.The outward limits of an organism's perceptual abilities define for it the boundaries between the perceptible and imperceptible, and, consequently, the limits and conditions of its activity.

TECHNOLOGICAL PERCEPTION
The concept of Umwelt offers a framework for understanding the contingency between sensory input and action, which is applicable beyond biological forms of perception alone.In order to adequately describe the human Umwelt, it is necessary to account for the mediation of human perception which occurs through the use of technology.Forms of sensory enhancement, telepresence, digital representation and automated sensing function symbiotically with biological perceptual apparatus.The mediation of sensory perception through technology couples together mechanical hardware with biological sensory organs, creating cybernetic relations between body, machine and the perceptible world.Like the Umwelt of the tick, anthropocentric media have been tailored precisely to the human perceptual framework to the extent that the parameters of such technologies are effaced.The dimensions of the screen, the viewfinder, the VR headset correspond as closely as possible to the human field of view, resolution and depth of field.Film and video aim toward simulating a seamless flow of moving images by maintaining a high enough frame rate that the human eye fails to catch the change of frame.The colours which compose our images and screens, too, are determined by those wavelengths of light to which eyes are attuned: red, green and blue.Optimal viewing distance and angle are also inbuilt in the design of visual technologies.Even our algorithms have acquired a human preference for eyes, faces, cats and dogs (Nguyen et al. 2015) reflecting the tendency to circulate images representing these themes more frequently than other kinds of images.
In this sense, Umwelt, the world as it is experienced by an organism, comes to be a technological Umwelt, the world as it is encountered through technology.The technological Umwelt is the perceptual world of an agent as determined by its perceptual apparatus.Expanding "organism" to "agent" in this definition acknowledges the abilities, agency, and influence of non-living perceptual actors, as found in instances of technological perception.Additionally, "perceptual apparatus", willfully equates biological perceptual organs with the technical implements used to extend or to substitute them.This may be applied to the perception of biological organisms as mediated by technical apparatuses, but it may also be extended to purely technical interpretations of visual information, as the performance of automated sensory tasks presents the capacity for machinic agents to constitute relationships between their interpretation and action upon perceptual phenomena.A digital camera set to automatic settings, for instance, does not perceive, as such, but it takes in, interprets and responds to the visual information about the environment in which it is situated.A robot, indeed, has an Umwelt (Emmeche 2001).The technological Umwelt is a coordination between external stimulus taken in, internalized and translated into action.The relation between automated sensory input and action could already be seen in early cybernetic experimentation.Grey Walter's tortoises (Grey Walter 1948), made in 1948 and 1949, demonstrated a simple form of automated visual navigation.The robotic tortoises were programmed to move toward areas with high light intensity, orienting their movement based on information from integrated light sensors.Much like the tick, the primitive cybernetic "animals" were driven by a relationship coordinating sensory input with action.Picking up increased light in one area in comparison to another directed the robots where to move.
Figure 2: Machina Speculatrix (Grey Walter 1948) Von Uexküll, himself, mused about the internal world of the invertebrates he studied: We no longer regard animals as mere machines, but as subjects whose essential activity consists of perceiving and acting.(von Uexküll 2009)

ERROR, ILLEGIBILITY AND INVISIBILITY
The agency displayed by entities which autonomously interpret and act upon sensory information from their environment make it difficult to consider machines as being isolated from biological processes.Extending a non-anthropocentric view to the design of visual media entails a reimagination of what it means to see.As a thought experiment, were one to construct television screens for bovines, the screen would need to be panoramic, as their eyes are set on either side of their skull.A screen designed for humans would fall directly into the weak spot of cow vision.Cinema for bees would necessitate an alternative colour spectrum from that of humans, as ultraviolet wavelengths are important signifiers for bees.Looking to non-living perceptual actors, turning the digital camera on its own output gives a taste of what designing media for computer vision might entail.Watching television through a video camera reveals banding from the screen refreshing, moiré patterns, bleeding pixels and other feedback errors.Glitches emerge from turning the camera onto the screen, as machines are not the intended audience for their own productions.Paradoxically, increased reliance upon automated sensing also creates openings of imperceptibility within those very systems.The seamlessness of human-centred media is a finely tuned illusion, which quickly breaks down when approached by way of an alternate framework.In this way, the errors which arise in technical perceptual systems allow the establishment of the limits of the technological Umwelt.Similar to the vision tests employed to examine human vision, the extent of a machine's vision is established by the boundaries of what it does and does not see.
Digital images have been likened to databases (Hoelzl/Marie 2015), as they exist as sets of alphanumeric code, instructions for the performance of operations which lead to the production of an image.The optical content of digital images exists parallel to their algorithmic basis.But what is said to be "behind", "under" or "unseen" in a digital image is difficult to discern.Different file types may appear roughly equivalent when displayed on-screen.What is visible in a digital image gives little clue as to the mathematical processes which determine visual content, except in situations of error indicative of misinterpretation.The relation between an image and its source code is a cypher, except to those conversant enough to read it.By looking at the technological ecosystem in terms of Umwelt, and by identifying the parameters entailed by the heterogeneous visual systems which make up such a hybrid ecosystem, it is possible to identify the spheres of imperceptibility which emerge from visual technologies.
The pervasive automation of perceptual tasks in some cases renders humans a secondary audience for visual information.From drone navigation to mass-surveillance to more commonplace applications such as the use of machine learning to read addresses on envelopes and to thereby sort mail, algorithms are increasingly present in everyday visual tasks.The terms operative image (Farocki 2004), or operational image have been used to describe this turn toward machinic images.An operative image is the visual information used by machines in the performance of a task.Operative images are uncompromisingly non-anthropocentric in that they may or may not be visible to humans.With the expansion of operative images, human visual perception may be minimized or taken out of the equation entirely.The unseeing eye of the machine need not necessarily communicate with "meat-eyes" (Paglen 2014).In the operative image, performing an automated visual task is prioritized at the cost of obscuring the method.Such a massvisualization by way of a machinic aesthetics places humans at the disadvantage of being immersed in algorithmic imagery which has moved out of our visual range.Finding ourselves in such a world of invisible processes is akin to taking a stroll in the limited Umwelt of the tick.

PRACTICAL EXAMPLES
Researchers, hackers, artists and other practitioners have experimented with methodologies for learning about the limits of algorithmic perception.

Related Technical Research
Adversarial image research, for example, applies an error-based approach to learn about the parameters of automated image analysis.Adversarial inputs are designed to cause errors in machine learning algorithms and to thereby unravel the processes behind a given error, but also within the system as a whole.Due to the nature of machine learning, automated visual tasks are obscured from view, developing solutions mathematically, rather than based on human-centred visual appearance.The difference between an image which will be misread by the computer and an ordinary image is often difficult for humans to see-if it is visible at all.In this sense, adversarial images create a two-fold invisibility: a human-readable surface image which is unreadable to the computer and an underlying algorithmic "image", which is invisible to humans (for examples of relevant adversarial approaches see: Nguyen et al. 2015;Athalye et al. 2017;Su et al. 2017;Bock et al. 2017).

Relevant Artistic Practices
Adam Harvey has been working with algorithmic biometrics in his project CV Dazzle, in which he created a look-book of suggested styling tips for evading detection by security cameras.The strategy employed involves makeup and hairstyles which break up the features of the face in order to thwart facial recognition.Much like biological carriers of significance, the features of a face in reference to each other act as technological carriers of significance to facial recognition systems.By reconfiguring, scrambling and disrupting the carriers of significance which comprise a face to a camera, it is rendered illegible.In that sense, a person can become invisible, simply by being unreadable to a machine learning algorithm.The security camera's "eye", so to speak, is fine-tuned to seek out faces, but if it doesn't find any, it lies in wait.In a related exploration of the outer limits of what constitutes a legible face to a computer, the author has produced a series of digital images entitled Unface (see Figure 4).Using Photoshop's contentaware fill function, images of faces were selectively erased and automatically filled in by the computer, resulting in blurry repetitions of visual elements from remaining sections of the image, and the erasure of the "meaningful" parts of the face.The images are produced through a process bringing together human and computer image interpretation, whereby the human agent collaborates with the computer's rendition of how missing sections of an image might be completed.In a speculative sense, the images function as representations of what the computer takes to be a readable image.Hito Steyerl's How Not to be Seen: A Fucking Didactic Educational .MOV File plays with technical visibility and invisibility from a different angle, noting that that which is unable to be captured by a camera's resolution is made invisible.She works with this theme from several angles, asserting that it is not only the technical side of image calibration but also social, economic and political structures and processes which determine visibility.Ghostly bodies cloaked in green and others with pixels for faces drift in and out of frame, and the artist smears herself with Chroma key paint, camouflaging herself into the surface of the video.Becoming a picture is described in the work as one method for achieving invisibility from surveillance, becoming imperceptible by adopting the very modalities through which visual information is captured and processed.

CONCLUSION
A common thread among the examples covered here is a consideration of how machines process perceptual information and the inherent boundaries embedded within those systems.In each project explored here, there is an awareness that while these perceptual systems may be modelled upon biological perception, they depart rather drastically from the modalities through which we perceive.The tendency of these works to adopt the methods of visual technologies themselves as a way of subverting them also manages to make the borders between visible and invisible more tangible by revealing a contrast in what constitutes a legible image.The disappearance of images into operative processes and the prioritisation of procedure over visibility indicates that machinic interpretation is the intended audience of these aesthetic productions.Adopting technologically-centred systems of signification and interpretation also entails the production of new spheres of imperceptibility.Relating these aspects back to the idea of a technological understanding of Umwelt, it can be seen that technological processes are thoroughly integrated into the modes of perception employed in the practices explored in this paper.Technology, in this sense, plays a role not only in mediating human perception, but in turn also plays a receptive role toward which aesthetic productions are aimed.
Approaching visual technologies by way of a technological notion of Umwelt, this paper lays out a conceptual framework through which to understand cybernetic perceptual relations.While the world may be saturated with surveillance and automated visual processes are increasingly common, hypervisibility lends itself to the production of its own spheres of invisibility.The mediation of perceptual phenomena which occurs through technology alters the sphere of the perceptible, expanding perceptual capacities in some cases, at the same time as introducing new limits (Ihde 1990).Much like the relations between the perceptual apparatus of various species, differences between biological and technological perceptual apparatuses manifest discord between respective systems' signifiers.Far from exclusively expanding the realm of the visible, the advancement of algorithmic visibility brings with it new aspects of blindness, invisibility and the unseen.