Emergence in the Expressive Machine

The “Expressive Machine” is a series of interactive artworks which explore a machine’s-eye view of the world. The machine— an assemblage of hardware and software—provokes sensual interaction with viewer-participants, playing with transduction across multiple modes: from touch to sound, to word, to vision, to taste, to uniquely machinic states with no particular human analogue. These stimuli are processed in various interpretations, elaborations, in a relatively unstructured “data soup”. Asynchronous processes consume data from the soup. When trigger conditions for a particular expressive process are satisﬁed, the machine produces externalised outputs in various forms: sound, shift of attention, fragments of narrative, and so on. What can be considered as creativity arises as an emergent property—a serendipitous by-product of the machine working through its experiences, rather than an explicit creative process. To make this conceptual exploration possible, an adaptable, extensible, decentralised system is presented: its requirements, architecture and some implementations as interactive artworks. Each new site and context gives rise to a unique instantiation of the machine, as it explores and expresses its experiences.


Introduction
This project takes place within a "posthuman" movement across a range of disciplines, reconsidering what it is to be human, and exploring the agency of non-human entities. Braidotti emphasises an ethos of positivity, an opportunity to "empower the pursuit of alternative schemes of thought, knowledge and self-representation" [Bra13]. Goriunova [Gor14], Parisi & Fazi [PF14] and others pose questions about the machine's experience, attempting to consider, in a rigorous way, such things as "do machine's have fun?" and what exactly "fun' might mean for a computer.
Fabulation and fiction have long been used as a vehicle for speculation of this kind, for example, Fuller's Half 10k Top-Slice Plus Five, written from the machine's sensually rich point of view [Ful17]. Briz puts forward a "philosophy of glitch", to establish a critical relationship between human and machine [Bri11]. The glitch can be seen as the machine wilfully expressing itself outside of the boundaries intended by the human. Thompson's The Seeker roves among unsecured "security" cameras across the world, using machine vision techniques to describe for us what it sees [Tho17]. Cheng's BOB ("Bag of Beliefs") [Che18] explores emergent properties of artificial life, presenting an engaging entity with a rich and developing internal life.
These artworks all question what a machine might find worthy of attention, exploring new ways of machinic seeing and experiencing of the world, and possible modes of "expression". Expression is the means by which the inner states of the machine can be made known, either as a deliberate and purposefully directed action, or impulsive, involuntary. In its directed sense, this implies an audience of some sort, and the selection of an appropriate mode of communication.

Expressive Machine Project Scope
Dennett proposes a theory of consciousness whereby multiple parallel neural processes operate without centralised control, giving rise to consciousness as an emergent property of their functioning [Den91]. Central to Dennett's theory is the idea of producer processes that generate data from external and internal stimuli, and consumer processes, which hungrily look for particular data sources, all operating asynchronously. Taking this as a model, and building on previous work [Dek18], the Expressive Machine project investigates, from the ground up, the design, implementation and on-going development of a series of decentralised hardware/software entities as art installations.
Several instantiations have been implemented, for a variety of contexts and sites, at exhibitions and festivals [Lat18] [PSD18] [V&A18]. Each instance gives the opportunity to build on previous versions, to experiment with new components and techniques, and new ways of interacting with an audience, specific to the particular context.
A strong machinic aesthetic arises from the raw exposed parts, wiring, electronics, support structures, clamps, tracks. The implied narrative is of a machine that has built itself, and is in continued struggle to discover, grow and communicate. The machine assemblage is encouraged to express its experiences, "how it feels", to see what emerges during its operation.

Infrastructure Design
The artistic aims for these investigations are: an aesthetically strong physical presence; to communicate and make visible (in the widest sense) the underlying transformations and processes taking place within the machine; to allow for fluid experimentation: expansion, growth, adaptation, emergence of new properties and behaviour. This gives rise to the following technical requirements for the infrastructure: • a modular, distributed architecture of software and hardware, with replaceable, adaptable modules; • asynchronous messaging between modules: broadcast or publish and subscribe; • a broad range of input and output devices (sensors, actuators, display/presentation modes, etc.); • control the focus of attention (via algorithms and actuators); • store and retrieve data and encodings about the world and self; • a range of data processing techniques, including pattern recognition and learning, across a wide range of modes; • a range of ways of generating and expressing output (audio, text, graphic, symbolic); • communication with the world beyond the immediate (Internet).
The modular architecture makes it possible to take advantage of developments in machine vision and learning, and to incorporate new hardware, such as low-cost microprocessors, new sensors and actuators, while choosing to preserve, adapt or extend the functionality. MQTT (mqtt.org) was chosen for messaging between modules for several reasons: • publish and subscribe, asynchronous communication, including many-to-many; • lighweight-a client can run on even tiny processors (such as the WiFi-enabled ESP32); • client can run on a wide range of hardware and operating systems; • any module can query another to see what data it is publishing.
In addition, MQTT is used in the Internet of Art Things framework [Cla17], offering a well-defined process for linking to any other artworks using this protocol. Figure 3 gives an overview of the processes of sensing and expression. Each module is contained in a MQTT client wrapper, so it can publish data and subscribe to data from other modules.

Input: Tasting Through Vision
One of the most important features of the Expressive Machine is its capacities for transduction. Using video input from a scanning webcam, the machine "tastes" its visual input. For this part of the system, a neural network was trained with compressed video clips of food waste, compost heaps, etc., nominally labelled with varying values of salty, bitter, sweet, sour and umami. The webcam scans along a track and rotates, pausing at things of "interest". Fluctuating taste values are displayed in real time, and the output is also used by other processes to influence expressive output (see below). The audience are encouraged to interact by "feeding" the machine with interesting objects (figure 1).

Input: Touch
As a means of getting closer to the human audience, and gaining physical stimulation, the machine has various forms of embodiment. The primaeval mossy green velvet sofa invites people to sit on it and stroke its touch-sensitive areas. Touch events are controlled by MPR121/Arduino and ESP32 board, which publish messages wirelessly to other processes. In a more mobile version, designed to demonstrate and make visible the decentralised, distributed nature of the machine, the sofa part of its embodiment is reduced down to a more enigmatic small pink velvet appendage, figure 5.

Input: Web Scraping
The machine has access to things beyond itself, nearby, specific to the immediate site and context. It scrapes special offers from websites of local shops, and polls air quality data from the nearest air monitoring station. Further afield, it consumes text from online news articles, which feed into its on-going expressive commentary.

Input: Internal Sensing
As an example of the machine probing its inner states, it notes the CPU load on its various processors. This information is used to influence expressive processes (see below).

Expression: Symbolic Commentary of Events
An event commentary process subscribes to all messages handled by the MQTT broker and verbosely renders them as a symbolic "data soup". Each event is represented by a unique icon: a disembodied hand represents a certain type of touch event, various other squiggles and sketches represent other triggered events. Each symbol bounces into foreground view as its corresponding event occurs; it bobs about with the other teeming icons in the soup, then gradually recedes and eventually disappears. The iconic soup is displayed by back projection onto a frosted Perspex disc.

Expression: Verbal Commentary
Several expression processes asynchronously produce textual output as a commentary on the machine's inner states and experiences. Some of the fragments are relatively unmediated "blurting" of inner states, or new information that the machine has just discovered, such as a special offer on a shopping website that it is interested in ("Iceland Baste From Frozen Chicken Breast 525g"). Some are derived states, such as one labelled "urgency", which is a function of the current sensed CPU load and local air quality index. Other fragments arise from more reflective natural language processing. A Responsive Markov Model [Dek18] is trained with various text sources (amongst others, Guardian online news articles and James Joyce's Ulysses) and responds to stimuli, in real time, when generating its new fragments of text. For example, the machine's currently sensed "taste" values determine probabilities that it will pick from certain vocabulary pools, so that the "flavour" of the textual output fluctuates, depending on what the machine is observing. These textual fragments are consumed by an output process which sends them to various display devices (projector, thermal printer, mini monitor). Each fragment is displayed as soon as it arrives, so that fragments collate together, in a textual stream-ofconsciousness.

Conclusion
The Expressive Machine project has developed a flexible infrastructure of hardware and software that allows for experimentation with multiple versions of a distributed, decentralised machine, with growing powers of sensing, exploration and means of expression.
The behaviour and expressive output that emerges is a collaboration between experimenter, machine, audience-participants and environment, both immediate and remote. In some ways, the machine acts as proxy, an extension of self.
So far the modes of communication imply a human audience, but the output is just outside of comfortable human intelligibility. If the assumed audience is something else (another machine, for example) then the modes and content would be radically different-more obliquely graspable, or not graspable at all by a human audience. For an artwork, does this matter? Pierre Huyghe addresses this directly in relation to his installation Uumwelt [Huy18]. Although presented in a conventional gallery, the machine-based artwork ostensibly bypasses the human observer: "the work does not need the public, it's not made for us, it's not addressed to us".
Many questions arise, meriting further investigation. Current work centres on using continuous learning, and a motivational drive, to determine how and where the machine focuses its attention, across the wide range of stimuli available to it.