Mini-XEPA: An Installation of Collaborating Intelligent Light and Sound Sculptures

Mini-XEPA is an installation of about 16 systems that anticipates a future where machines form their own societies. The project is a second generation following the original XEPA project of a few years prior. These systems exhibit apparent artistic creativity by applying modest artificial intelligence to aesthetic challenges. The small mini-XEPA intelligent sculptures create animated light sequences and musical sound. No scripts, coordination information, or commands are used. Each mini-XEPA “watches” the others and modifies its own aesthetic behaviour to create a collaborative improvisational performance.


INTRODUCTION
Mini-XEPA is an installation of (typically) 16 small systems that interact with each other intelligently. These systems provide not only an opportunity to present installations of light, colour, and sound, but they also provide a platform for experiments with various modules and strategies for computational aesthetic evaluation (Galanter 2012).
Mini-XEPA builds on the earlier light-only XEPA project by adding sound, and by significantly upgrading the software and underlying paradigms. As a platform for experiments in machine-tomachine interaction, Mini-XEPA will continue to be a work in progress. In addition, the Mini-XEPA software will be retrofitted for use on the earlier XEPA platform as an upgrade.

XEPA hardware
A short description of the earlier XEPA work will help to set the stage for a discussion of mini-XEPA (Galanter 2014). As an installation XEPA is reconfigurable and variable. Individual XEPA light sculptures are constructed using four to eight onemeter-long light tubes. XEPAs can be wall-mounted, free-standing, or suspended sculptures. Various installations may have differing numbers of XEPAs of different designs. Each tube is a milky white diffuser with 16 RGB LED lighting elements inside acting as 16 pixels. Each pixel is individually addressable as a 24-bit colour using the lighting industry DMX control protocol. Each XEPA uses three inexpensive processors. An Arduino Mega 2560 is used for high-level observation and decision-making. An Arduino  Leonardo is used as a light server for real-time DMX communications used to control the LED tube animation. In addition each XEPA has a BeagleBoard-xM available for eventual use as a sound server and/or music synthesizer. The BeagleBoard will come into use once the Mini-XEPA software is retrofitted. Figure 2: The three XEPA processors and data flow.

XEPA software and behaviour
Each XEPA sends XBee data radio packets at the start of a phrase describing what it is doing. This includes properties such as base hue, colour harmony scheme, animation pattern, tempo, and so on. There is no score or conductor. No XEPA "tells" another what to do. This use of data radio is the same, in principle, as if the XEPAs simply watch each other and only modify their own behaviour. This is meant to suggest that not unlike human societies, individuals in future machine societies will interoperate by robust high-level observation rather than brittle low-level protocols.
At the start of a given performance each XEPA begins in a randomized state. Using a quasigenetic approach, each XEPA then explores a huge combinatorial media space of colours, patterns, sequences, and tempos. Each XEPA has been "taught" fuzzy themes invented by the artist at a high semantic level with suggestive phrases such as 'house on fire' or 'spring life.' The artist implements these themes by creating weight tables. For example, a palette of blues and whites would be given a large weight for the theme 'arctic zone', while a palette of reds and yellows would be given a low weight.
These themes then serve as aesthetic attractors in the media space. Each XEPA "looks around" and determines how close each XEPA is to an attractor. Once the XEPA closest to an attractor is identified, it uses a genetic crossover function to change its own state. It copies and adopts some small part of the state of that closest XEPA. The XEPA closest to an attractor mutates, in an attempt to improve.
After about two minutes the performance ends. And after a short pause in the dark the next performance begins. While there will be some similarities across performances in the end-state, the performance is more about the process of convergence. Surprise can be generated, for example, when a given XEPA happens to pass close to an attractor on its way to the currently favoured attractor. It may unexpectedly become the "closest XEPA" and change the direction of overall performance.

MINI-XEPA
Mini-XEPA is, in a sense, XEPA version 2.0. As noted earlier the mini-XEPA software will be retrofittable to the original XEPA hardware. The high-level observation and decision-making software remains Arduino compatible. Both also use a Linux-based processor running the same Supercollider-based sound server. The notable difference requiring device specific code is the LED light hardware. Because of their physical differences, XEPA and mini-XEPA requires light sequences specific to their topology. But the requests sent to the light server will be the same.
Like XEPA, mini-XEPA uses three processors. But where an Arduino Leonardo and Mega 2560 was used in the XEPA, mini-XEPA uses Teensy 3.6 boards which offer higher performance, more memory, and a smaller physical footprint. And where XEPA uses a BeagleBoard-xM for sound synthesis, mini-XEPA uses a Raspberry-Pi.
A mini-XEPA is much smaller than a XEPA. This allows mini-XEPAs to be installed in greater numbers. It also eases the logistics of shipping to and from galleries. As installed there are typically 10 to 15 mini-XEPAs rather than 3 to 6 XEPAs. This allows for greater complexity in interaction.
Each mini-XEPA uses a 7 inch 32 by 32 matrix of RGB LEDs. The LED matrix is turned 45 degrees so it appears to be a diamond shape rather than a square. This simple artistic strategy causes the LED matrix to look less like a computer display, and more like a light object. Immediately below the LED matrix is a horizontally mounted cylindrical powered stereo speaker of fairly high quality and similar size.

Mini-XEPA software and computational aesthetic evaluation
As described above, XEPA uses a very basic form of computational aesthetic evaluation. The smallest of distances from various aesthetic attractors serves as that measure. Being able to use a number of aesthetic attractors corresponds well with the notion that even within the constrained aesthetics of the XEPA light sculptures, high quality experiences can be fast or slow, mostly red or mostly blue, frantic flashes or slow undulating pulses, and so on. What is much more determinative is the ensemble effect of a set of properties, and the way they conspire to create a coherent mood or tone.
However, filling out a dense table of weighted possibilities across all themes and properties leads to a combinatorial explosion as more themes and properties are added. And while focused methods for computational aesthetic evaluation may be, at this time, few and far between, this method of weight tables cannot put them to use.
Mini-XEPA uses a neutral framework into which arbitrary evaluation methods for a specific property, or set of properties, can be plugged in. This is done using a finite set of intermediate tags. A theme such as 'house on fire' might be tagged as 'hot', 'red', 'fast', 'monochrome', and so on. Each tag has a corresponding evaluation function based on one or more properties. At any time a given mini-XEPA will have a score for each tag. These scores can be thought of as fuzzy membership measures. The distance a given mini-XEPA is from a given attractor is calculated based on how well their tags match. This distance score can be buttressed by factoring in aesthetic evaluations that are "universal" and would apply to any theme. For example, at any time a mini-XEPA will be using a colour palette, and rules of traditional colour harmony can be used to make an aesthetic evaluation. Colour harmony evaluation is described in greater detail later in this article.
By establishing a finite vocabulary of tags, and for each tag a method for evaluating fuzzy membership based on actual properties, the creation of themes is greatly simplified. The combinatorial explosion found in XEPA is avoided, and the artist can invent themes using high-level semantics. And as new methods of computational aesthetic evaluation are invented, they can be plugged in and exploited by introducing corresponding tags.
A special note regarding artificial neural networks is worth mentioning at this point. Artificial neural networks, and especially deep-learning networks, are topics of intense current research. Some of this activity has included applications in the arts.
For example a technique called "inceptionism" has been developed by researchers at Google (Mordvintsev et al. 2015). By exposing a deep network to a very large corpus of images, it can be trained and then stimulated with a novel image. Through a process of iterative feedback, a kind of connectionist image processing synthesizes textures within the image. These textures may include hints of objects such as fish, birds, and dogs. The resulting images are often compelling with a psychedelic/hallucinatory or surreal style.
Similar systems can be used to apply learned styles to source images. What is less clear is how these systems can be used to create images de novo, or evaluate existing images as to quality. In any case, one can imagine that as these methods are developed they can be "plugged into" a future version of mini-XEPA.

Heuristics for aesthetic evaluation
Because the primary goal of mini-XEPA is artistic, computational aesthetic evaluation heuristics that are speculative or not scientifically verified are fair game. But that's not to say this project has no scientific utility. If nothing else, development work on mini-XEPA can contribute to hypothesis formation. It can also serve as an artistic "reality check" for scientific theories about aesthetic perception.
The plug-in evaluation methods used by mini-XEPA can be rule-of-thumb heuristics, or models based on real-world data. And these methods can often be used not only for evaluation, but for generation as well. Generation begins with a random generator of examples for some property, and then this stream of examples is filtered using the evaluator shaping the output.
An example of this is the creation and evaluation of chord progressions used by mini-XEPA. The company Hooktheory.com has created a large database of chord progressions transcribed from well-known pop music songs. Hooktheory.com uses this database to create interactive music education software. In cooperation with that company I've used a 450 song version of the database for use in mini-XEPA.
One of the oldest techniques in machine music composition is an analysis/synthesis method using Markov chains (Hiller and Isaacson 1958). As a simplified example, musical scores are analysed as sequences of notes. The score is processed with a sliding window of 4 notes. The result is that for every sequence of 3 notes a list of possible fourth notes is compiled along with their relative probability. One can then take that Markov analysis, and apply a Monte Carlo method to create sequences of events.
For mini-XEPA the Hooktheory database was used to create a Markov analysis at the level of chord changes. The resulting table of chord sequences and probabilities can be used to analyse the probability of an existent chord progression, or it can be used to generate a chord progression based on this real-world model.
It should be noted that a high-probability chord sequence is not considered aesthetically superior to a low-probability chord sequence. As a matter of taste, for some listeners the reverse might be true. Instead high-probability chord sequences are considered 'simple' and low-probability sequences are considered 'difficult'.
What is important is whether a given chord sequence is appropriate for a given theme. Such a measure can be buttressed with additional analysis such as the relative density of major, minor, augmented, and non-tonal chords. For a theme such as 'little schoolhouse' a 'simple-music' tag might apply. For a theme such as 'traffic jam' a 'difficult-music' tag might apply.

AESTHETIC EVALUATION OF COLOUR
For both XEPA and mini-XEPA a very important property is the set of colours used. Because direct rather than reflected light is what is seen, a simplified pragmatic colour model can be used that ignores issues of value. This discussion focuses on hue, keeping in mind that saturation in this constrained context would refer the to the amount of white content in the colour. For the purposes of this article 'colour palette' refers to a set of hues. The term 'colour scheme' refers to a vector that matches the colour palette in size, and indicates the relative proportion of each hue in use.
An artistic goal of both XEPA and mini-XEPA is the exploitation of what might be called "painterly colour." In previous publications I've noted the deleterious effect the default use of an RGB colour wheel has had on generative art. (Galanter 2013) That is reviewed here to set the stage for a colour palette aesthetic evaluation discussion.

Implementing painterly colour
In earlier light sculptures I've noted a dominance of cool colours, a relative lack of warn colours, and a difficulty in achieving a strong yellow without a hint of green or orange. This is also true when using a design-oriented computer language like Processing. I believe this is due to the default use of an RGB colour wheel for colour mixing. While it is true that electronic displays use red, green, and blue elements, that does not mean that an RGB colour wheel is the most appropriate one for artistic colour mixing or the creation of colour harmony.
As taught in basic colour theory for design, and illustrated in Figure 3, apparent colour mixes differently for light and pigment. An RGB system best predicts how (additive) coloured light will mix. For example, yellow light can be made by mixing green and red light. An RYB system such as the Itten colour wheel favoured by painters best predicts how pigments will mix. For example, yellow pigment is considered to be a primary colour that cannot be made by mixing two other colours. The problem is that built-in software functions typically reflect a linear mapping of the RGB colour wheel. Most notably this results in the warm colours being compressed into about 1/3 of the RGB colour wheel arc length. But on an RYB colour wheel the arc length is much more evenly split between warm and cool colours.

Figure 4: Random RGB colours (left) versus Random RYB colours (right)
Novice generative artists often use randomization to create surprise. So when hues are selected using a typical hue and random function, as shown in Figure 4, the overall effect is skewed towards greens, blues, and violets. A more balanced effect is achieved when a RYB colour wheel is randomly sampled. But such functionality is not typically builtin.
To achieve a more painterly look in terms of hue the strategy is to work with an RYB colour wheel for high-level colour mixing, the creation and evaluation of colour palettes, and so on. These hues are then converted to their RGB equivalents in the background at a low-level for rendering on screen, paper, or in LED lights.
In the course of mapping RYB hues into RGB representations I noted that there was a problem where cool colours transitioned to warn, and where warm colours transitioned to cool. The yellow-green seemed too green, and the red-magenta seemed too red. To improve those transitions I moved yellowgreen half again closer to yellow, and red-magenta half again closer to magenta. This is illustrated in Figure 5. I named this colour wheel RYB-Plus.

Figure 5: The RYB system (left) versus the RYB-Plus system (right)
It was previously noted that the hue to be displayed, typically described with a number from 0 to 360 for a position on the colour wheel, must be mapped into RGB values for display in a way transparent to the artist. Figure 6 illustrates this for the typical RGB colour wheel supported by most software hue functions. It is a simple linear interpolation with the hue varying from 0 to 360 on the X-axis, and the output RGB values varying from 0 to 255 on the Y-axis. Figure 6: Hue-mapping based on an RGB colour-mixing wheel rendered to RGB for device display Figure 7 illustrates the mapping of RYB-Plus hue to RGB values for display. Note the bend in the red and blue lines where the correction is made for the transition points from cool to warm and warm to cool.

Figure 7: Hue-mapping based on an RYB-Plus colourmixing wheel rendered to RGB for device display
As a practical matter LED displays produce hues that are a bit off-target relative to the eye. In software, the RYB-Plus to RGB routine is designed to be manually adjusted for a given type of display. There are tables for the RGB values for each of the 12 hues in the RYB-Plus colour wheel. These are tweaked to match the 12-hue colour target by eye. When called the conversion routine then interpolates using those 12 sets of RGB values. This approach has been successful in, for example, matching colours across different types of LED.

Implementing colour palette evaluation
Traditional colour theory for design includes the notion of colour harmony. Colours that are "in harmony" are thought to be those that are in some sense evenly spaced on the RBY (i.e. Itten) colour wheel. This is a notion of aesthetic evaluation that has stood the test of time, but remains scientifically speculative.
In implementing aesthetic evaluation of colour palettes for mini-XEPA this notion has been extended beyond the traditional categories. This can at best be described as an artistic decision that seems to be verified by my own subjective experience for colour palettes made up of 2 to 6 hues.
The general approach begins with the realization that for a palette of a given number of colours, there are a number of ways the colours can be organized for analysis. For example, a palette of three colours might be viewed as a potential equilateral triangle forming a triadic palette, or it could be viewed as a potential isosceles triangle forming a split-complementary palette. To address this problem a given colour palette is analysed as a candidate for all of the relevant colour palettes, and then awarded the best score.
For each potential palette configuration an error score is calculated by essentially comparing the actual hue values to the theoretically perfect values. First the hues are sorted, and then the arc lengths are calculated by subtracting hue N from hue N+1. Then each arc length is subtracted from the corresponding perfect arc length and squared. The sum of the errors squared is then divided by the number of hues in the palette. Dividing the sum of the squared errors helps make evaluation scores for palettes with different numbers of hues comparable. Since what is calculated is essentially an error score, the lower the number the more desirable the colour palette.
For example, for evaluation as a triadic colour palette, each of the three arc lengths is subtracted from 120, squared, summed, and divided by three.
There are three types of colour palettes considered: regular palettes, irregular palettes, and serial palettes. Regular palettes are perhaps the best-known palettes in traditional colour theory for design. They include hues configured as triadic, tetradic, and complementary palettes. As with all these forms of colour harmony, group rotation of the colours has no impact on the evaluation, and only the relative distances between the colours matters. Irregular palettes still exhibit symmetry, but are based on configurations that have two arc lengths in play. This is an extension of the traditional tetradic colour harmony scheme.
Finally, serial palettes are those where the hues are evenly spaced other than one longer arc length. This is an extension of the traditional analogous colour harmony scheme. Once the lowest (best) scores for possible regular, irregular, and serial palettes are found, the lowest of the three is put forward as the raw aesthetic evaluation score.
A penalty is then calculated for hues that are "too close" to each other. Any arc length less than 30 degrees is subtracted from 30, squared, and summed. That amount is then added to the prior raw score. This penalises pairs of hues that may appear to be ambiguous or overlapping.
When several random palettes are evaluated the resulting raw score plus penalty can span several orders of magnitude. To bring the span of scores into a manageable range one is added, and then the base 10 log is taken. This results in a log-score from 0 up to 5 or so.
Experimenting with thousands of trials using random palettes my subjective judgement was that the threshold for "acceptable" palettes was quite similar regardless of the number of hues in the palette. For two, three, and four hues a score of 2.5 or less was acceptable. For five hues a score of 2.7 or less was acceptable. And for six hues a score of 2.9 or less was acceptable. To adjust for this, and to allow evaluation scores to be directly compared regardless of the number of hues in the palette, the log-score for five hue colour palettes is multiplied by 2.5/2.7, and the log-score for six hue colour palettes is multiplied by 2.5/2.9.
Keeping in mind that no particular scientific claim is being made here, it's my artistic and subjective experience that this method of computational aesthetic evaluation is highly correlated with my own aesthetic judgement. The threshold of 2.5 is almost always correct in determining acceptability, and given two palettes the better of the two almost never has the higher (poorer) score.
It would be interesting to run a controlled userstudy to verify that this method of evaluation holds for the general population.

Implementing a colour palette generator
As was mentioned earlier, an aesthetic evaluation method can be used to create a corresponding generator by using it to filter a source of random properties. This approach was used with the evaluation method noted above to create a colour palette generator for mini-XEPA.
The generator itself is rather simple. Given the number of colours needed in the palette, the generator makes a small number of random palettes, evaluates them, and returns the one with the best (lowest) score.
It was noted that in order to return an acceptable palette scoring 2.5 or less, testing different numbers of random palettes was required for different palette sizes. This varied from about 2.566 trials for 3-colour palettes to 8.49 trials for 6 colour palettes.
These numbers were arrived at by running simulations generating millions of palettes, and tweaking the number of trials for each palette size until the generator returned an acceptable palette half of the time. By using a Monte Carlo method to determine the number trials in a given call, on the average the generating function runs the needed number of fractional trials. Strictly speaking mini-XEPA would not need an optimised colour palette generator. Completely random colour palettes could be used initially, and then the quasi-genetic nature of the group performance could be expected to improve the colour palettes over time. But from an artistic point of view having somewhat acceptable colour palettes from the start of each performance is more aesthetically pleasing than not.