A Pedagogy of Creative Thinking based on Sonification of Visual Structures and Visualization of Aural Structures

This paper describes two related pedagogies for design education – one for teaching visual design and the other for teaching sound composition – in which the symbols and organizational principles of aural experience are transformed to the visual domain and vice versa. The purpose of the pedagogies is to develop in the student the ability to visualize and auralize from direct observation as an alternative to copying and modifying existing designs. The paper begins with a discussion of visual and aural perception and their relationship to thinking and imagination. The author then describes the use of semiotic transformation in teaching courses in sound composition and visual design. 1 Educating the Visual and Aural Designer Teachers in the visual and aural arts often misunderstand the principles of education required for applied designers in contrast to those used to train fine artists. Although fine artists often base their work on self expression, playful novelty, or conscious copying from an established tradition, the applied artist is usually in the position of designing a structure which communicates information to meet the needs of an external client. Thus training for the designer, whether a designer of sound or image, must include learning to think of visual and aural structures as solutions to design problems. Recently this author has come to realize that an effective way to teach this kind of thinking is to ask students of sound design to analyse works of visual art then translate their structures and symbols into aural equivalents and, conversely, to ask visual design students to analyse sound compositions and use their structures as organizing principles for visual designs. Mladen Milicevic [15] has called this process “semiotic transformation.” 1.1 The Function of the Designer There is a great tendency to confuse the role of the designer with that of either the fine artist or the stylist. The term “designer” is often applied to the stylist. Most properly thought of, the information designer is a problem solver or information “architect” who takes ideas or information developed by a client and puts it into a form which will communicate effectively. A graphic designer takes text and images which communicate information about a business, for example, and creates a publication which presents the information clearly and interestingly to the reader. The stylist, though often confused with the designer, usually takes a design already produced by an original designer and re-shapes or decorates it to make it more novel or individualistic. Thus a creator of “designer” jeans does not really design a piece of clothing which functions more effectively than the original Levi Strauss product of a century ago, but rather creates a more novel variation of the original design. Such a “designer” is most properly called a “stylist.” The fine artist may develop his or her work in a variety of ways. Some artists consciously copy the tradition of a certain genre of art. Others treat art as a form of free self-expression and create works meaningful to themselves, but which sometimes are not meaningful to others without explanation. Others may try and challenge the viewers’ or listeners’ expectations or perceptions. 1.2 Visual and Aural Imagination Clearly the designer of a visual or aural structure must understand the information to be communicated then develop a concept for an effective presentation of that information. Ideally this should be done by understanding the nature of perception in the viewer/listener and developing an idea about how the information to be communicated could be designed for successful perception. A Pedagogy of Creative Thinking Based On Sonification of Visual Structures and Visualization of Aural Structures ICAD'98 2 Often art education and design practice fail in meeting this ideal. Too often design is copying of a look, a sound, indeed a “style,” and applying it to a problem which may or may not be suited to the solution. One only has to look at the many copy-cat confusing web sites to see this phenomenon, or listen to much popular music. Yet design progress requires original conception, not ornamentation or augmentation. 1.3 The Integration of the Senses There is an inherent integration of the human senses as noted by Sergei Eisenstein [6], Erich M. von Hornbostel [20], Oliver Sacks [18] and others. The experience of one sense can, at a minimum, remind one of a sensation of another sense. For example, when most people hear someone talking on the radio, they develop a mental picture of the appearance of that person. William Gaver [7] has noted that in everyday perception of sound in the environment people tend to visualize a sound by associating it with the source they believe produced it. In the most extreme form of sense integration, synesthesia, a person exposed to stimulation of one sense actually experiences physical sensations of another sense. Richard Cytowic [4] has documented cases in which people actually see colours when hearing music and feel shapes when tasting food. But true synesthesia is a neurological condition. The present project is an attempt to make use of the inherent “everyday” integration of the senses (in Gaver’s sense) in order to allow sense experience of one form to promote creative thinking in another. 2 Thinking Visually and Aurally The idea of an artist studying more than one art form in order to derive first principles is not new. The painter Wassily Kandinsky wrote “One art must learn how another uses its method, so that its own means may than be used according to the same fundamental principles, but in its own medium. ” He added that “The natural result of this work is a comparison of the elements of one art with those of another. Music is found to be the best teacher.” [10] Though perception and creative thinking are certainly related, there is a vast difference between the value placed on visual experience and on aural perception. 2.1 Vision as the Basis of Thought Scholars studying the anthropology of communication, especially Edmund Carpenter [3], Marshall McLuhan [14], Walter Ong [16] and David Berlo [2] have noted that literate cultures tend to emphasize sight over the other senses. Western societies, especially, have tended to privilege the sense of sight since the widespread adoption of literacy made possible by the printing press. In such cultures people equate seeing with thinking and use visual terms as a kind of code to describe understanding of thought: “I see your point,” “Can you make yourself more clear?” “Your logic is fuzzy,” “I get the picture,” “ Seeing is believing, etc. Cultures which do not equate seeing with thinking would have difficulty decoding such statements in a meaningful way since they do not share the same code. 2.2 Sound as the Basis of Thought Cultures which do not have a strong tradition of literacy develop an aural culture with its own particular characteristics and code. In these cultures terms related to thinking often get expressed in aural terms. For example, “I hear you” is often used to mean “I understand you.” Though people from most literate cultures would rather go deaf than blind, people from oral cultures feel deafness to be a greater hardship than blindness. Non-literate cultures tend to believe something heard over something seen. Even Plato expressed skepticism and distrust of the then-new medium of writing, believing it to be inferior to word-of-mouth communication. [17] Speech in non-literate cultures tends to make use of several aural and visual elements of expression: the meaning of the words chosen (“diction”), the vocal inflections of the person saying the words, and the body movements, positions and gestures of the speaker. Speakers in non-literate cultures, including children in all cultures who have not yet learned to read, tend to use many inflections and gestures. But as people become educated in literate cultures they are often taught to “modulate” their vocal inflections, stand still as they talk, and not use gestures. Thus speech becomes reduced to the single element which can be coded by writing or printing: the meaning of the words themselves. [3, 14,16] A Pedagogy of Creative Thinking Based On Sonification of Visual Structures and Visualization of Aural Structures ICAD'98 3 2.4 Resonance and Thought David Berlo [2] and Tony Schwartz [19] remind us that meanings do not reside in words, images, or sounds, but in the cultural experience of each reader-viewer-listener. Thus perceptions bring to mind meanings which may not be the same for each recipient of the same act of communication. The complexity of meaning is apt to be greater than the symbol used for communication. Schwartz writes “A listener or viewer brings far more information to the communication event than a communicator can put into his program, commercial or message.” [19] Maribeth Back [1] has taken a similar approach to sound design, letting a well chosen sound evoke a deeper cultural meaning. Miroslav Malik, [12] a Czech-Canadian designer of multimedia exhibits for worlds fairs and other multicultural festivals, often used a “mosaic” design approach in which many different images are presented at once drawn from the cultural backgrounds of different cultures. Viewers from each culture are then able to decode at least part of the message. The ability for a sound, word or image to evoke a complex thought allows for very powerful communication with a minimum of means or a narrow channel. But such associations can also lead to “stereotyping” in which a symbol always brings to mind a single instance of what could otherwise be a multidimensional thought. These symbols become icons that are not analysed for the information they contain but which simply redirect the perceiver to his/her own stored meaning. Indeed, some people try to find symbolic associations in certain kinds of messages where none are intended. A common example is the pre


The Function of the Designer
There is a great tendency to confuse the role of the designer with that of either the fine artist or the stylist.The term "designer" is often applied to the stylist.Most properly thought of, the information designer is a problem solver or information "architect" who takes ideas or information developed by a client and puts it into a form which will communicate effectively.A graphic designer takes text and images which communicate information about a business, for example, and creates a publication which presents the information clearly and interestingly to the reader.
The stylist, though often confused with the designer, usually takes a design already produced by an original designer and re-shapes or decorates it to make it more novel or individualistic.Thus a creator of "designer" jeans does not really design a piece of clothing which functions more effectively than the original Levi Strauss product of a century ago, but rather creates a more novel variation of the original design.Such a "designer" is most properly called a "stylist." The fine artist may develop his or her work in a variety of ways.Some artists consciously copy the tradition of a certain genre of art.Others treat art as a form of free self-expression and create works meaningful to themselves, but which sometimes are not meaningful to others without explanation.Others may try and challenge the viewers' or listeners' expectations or perceptions.

Visual and Aural Imagination
Clearly the designer of a visual or aural structure must understand the information to be communicated then develop a concept for an effective presentation of that information.Ideally this should be done by understanding the nature of perception in the viewer/listener and developing an idea about how the information to be communicated could be designed for successful perception.

ICAD'98 2
Often art education and design practice fail in meeting this ideal.Too often design is copying of a look, a sound, indeed a "style," and applying it to a problem which may or may not be suited to the solution.One only has to look at the many copy-cat confusing web sites to see this phenomenon, or listen to much popular music.Yet design progress requires original conception, not ornamentation or augmentation.

The Integration of the Senses
There is an inherent integration of the human senses as noted by Sergei Eisenstein [6], Erich M. von Hornbostel [20], Oliver Sacks [18] and others.The experience of one sense can, at a minimum, remind one of a sensation of another sense.For example, when most people hear someone talking on the radio, they develop a mental picture of the appearance of that person.William Gaver [7] has noted that in everyday perception of sound in the environment people tend to visualize a sound by associating it with the source they believe produced it.
In the most extreme form of sense integration, synesthesia, a person exposed to stimulation of one sense actually experiences physical sensations of another sense.Richard Cytowic [4] has documented cases in which people actually see colours when hearing music and feel shapes when tasting food.But true synesthesia is a neurological condition.The present project is an attempt to make use of the inherent "everyday" integration of the senses (in Gaver's sense) in order to allow sense experience of one form to promote creative thinking in another.

Thinking Visually and Aurally
The idea of an artist studying more than one art form in order to derive first principles is not new.The painter Wassily Kandinsky wrote "One art must learn how another uses its method, so that its own means may than be used according to the same fundamental principles, but in its own medium."He added that "The natural result of this work is a comparison of the elements of one art with those of another.Music is found to be the best teacher."[10] Though perception and creative thinking are certainly related, there is a vast difference between the value placed on visual experience and on aural perception.

Vision as the Basis of Thought
Scholars studying the anthropology of communication, especially Edmund Carpenter [3], Marshall McLuhan [14], Walter Ong [16] and David Berlo [2] have noted that literate cultures tend to emphasize sight over the other senses.Western societies, especially, have tended to privilege the sense of sight since the widespread adoption of literacy made possible by the printing press.In such cultures people equate seeing with thinking and use visual terms as a kind of code to describe understanding of thought: "I see your point," "Can you make yourself more clear?""Your logic is fuzzy," "I get the picture," "Seeing is believing, etc. Cultures which do not equate seeing with thinking would have difficulty decoding such statements in a meaningful way since they do not share the same code.

Sound as the Basis of Thought
Cultures which do not have a strong tradition of literacy develop an aural culture with its own particular characteristics and code.In these cultures terms related to thinking often get expressed in aural terms.For example, "I hear you" is often used to mean "I understand you."Though people from most literate cultures would rather go deaf than blind, people from oral cultures feel deafness to be a greater hardship than blindness.Non-literate cultures tend to believe something heard over something seen.Even Plato expressed skepticism and distrust of the then-new medium of writing, believing it to be inferior to word-of-mouth communication.[17] Speech in non-literate cultures tends to make use of several aural and visual elements of expression: the meaning of the words chosen ("diction"), the vocal inflections of the person saying the words, and the body movements, positions and gestures of the speaker.Speakers in non-literate cultures, including children in all cultures who have not yet learned to read, tend to use many inflections and gestures.But as people become educated in literate cultures they are often taught to "modulate" their vocal inflections, stand still as they talk, and not use gestures.Thus speech becomes reduced to the single element which can be coded by writing or printing: the meaning of the words themselves.[3,14,16] A Pedagogy of Creative Thinking Based On Sonification of Visual Structures and Visualization of Aural Structures ICAD'98 3

Resonance and Thought
David Berlo [2] and Tony Schwartz [19] remind us that meanings do not reside in words, images, or sounds, but in the cultural experience of each reader-viewer-listener.Thus perceptions bring to mind meanings which may not be the same for each recipient of the same act of communication.The complexity of meaning is apt to be greater than the symbol used for communication.Schwartz writes "A listener or viewer brings far more information to the communication event than a communicator can put into his program, commercial or message."[19] Maribeth Back [1] has taken a similar approach to sound design, letting a well chosen sound evoke a deeper cultural meaning.Miroslav Malik, [12] a Czech-Canadian designer of multimedia exhibits for worlds fairs and other multicultural festivals, often used a "mosaic" design approach in which many different images are presented at once drawn from the cultural backgrounds of different cultures.Viewers from each culture are then able to decode at least part of the message.
The ability for a sound, word or image to evoke a complex thought allows for very powerful communication with a minimum of means or a narrow channel.But such associations can also lead to "stereotyping" in which a symbol always brings to mind a single instance of what could otherwise be a multidimensional thought.These symbols become icons that are not analysed for the information they contain but which simply redirect the perceiver to his/her own stored meaning.Indeed, some people try to find symbolic associations in certain kinds of messages where none are intended.A common example is the preoccupation naive art viewers often have trying to figure out what a non-representational work of art is "supposed to be."

3
Context and Imagination

The Role of Context in Fostering Creativity
Designer Steve McCallion relates an assignment a professor of his gave in architecture school: "to create an innovative office tower.He insisted that we not reference outside resources -no books, no photographs. . .The result?A depressingly familiar display of foam core and hot glue."[13] Given a lack of context, designers often resort to the stereotypical thinking mentioned above.They tend to recall a stored impression and make it an "original" design by changing small details.They become stylists.By greatly changing the context of a design problem it is much more likely that original designs, not "styles," will result.Thus if one asks a class simply to design a "car" most will create a stylistic variation of the typical car.If however, one stipulates that the car has to be able to cross water and also drive up sand dunes, then the discontinuity of the new context will more likely produce innovative thinking.Indeed, the architect John M. Johansen has produced some highly original experimental designs -the kind McCallion may have been seeking from his students -by setting for himself some imaginative, but recontextualizing, problems: a structure which hangs between twin towers of New York's World Trade Center and is supported by them; a theatre which uses magnetic levitation to move people and objects; buildings which "adapt" themselves to changing environmental conditions similar to adaptation in biological forms, etc. [9]

Discontinuity as an Agent of Contextual Change
Eric Hoffer [8], Peter Drucker [5] and others have discussed how discontinuity leads to innovation.It is often reported that immigrants to the U.S. have a better-than-average record of success as inventors, business entrepreneurs, and artists.The change of social context leads these people to creative thinking more easily than someone brought up within the culture.It is dealing with the discontinuity of change that seems to produce creative thinking.Discontinuities can be caused by various forces.Many young people in the 1960s became disillusioned with certain social structures and political policies in the U.S. and sought innovative changes.Writers and painters often travel to consciously experience discontinuities which might awaken them to new ideas and images.
In this present project I have produced discontinuity as a stimulant to creative thinking by asking students to "re-frame" their visual perceptions in aural terms and to re-frame their aural perceptions in visual terms.This process of semiotic transformation involves the conversion of the symbols and structure of one sensory medium to those of another.It is an attempt to re-integrate the senses so that auditory experience will play a role in visualization and visual experience will play a role in aural thinking.

Semiotic Transformation as a Stimulant to Design Thinking
There is a tradition of using art of one form as "inspiration" for another.Some teachers of fine art play music in the classroom and ask students to "paint what they feel."A similar approach is used by some writing teachers.
In music there is the tradition of "program music" which is intended to bring to mind a specific experience.Thus in Beethoven's Sixth Symphony we hear passages intended to remind us of a storm, of villagers playing, etc. Lest we miss the intent Beethoven thoughtfully annotated the score.The pedagogy being presented here is not related to any of those strategies.Its purpose is not to evoke shared cultural memory of certain images or experiences, nor is it intended to focus the student on his/her own "feelings."It is analytical and structural in nature.The student is not asked to "interpret" a sound or image, but to understand the nature of its elements and how they are related, then create an artwork in another medium which uses elements of a "similar" nature arranged in a "similar" structure.What sounds are similar to certain pictorial elements, and what pictorial representations would be good "translations" of sound elements, are decisions the student designer must make.The result then is not self expression (the domain of the fine artist) but analytical problem solving (the domain of the true designer).

Sound Composition Based on Sonification of Visual Experience
In teaching a course in sound composition and musique concrète a few years ago I faced a problem similar to that of Steve McCallion's professor (see 3.1 above).When I asked students to make sound compositions from "found sounds" recorded by them using a portable DAT recorder, I tended to either get compositions that were more or less random collections of sounds or imitations of non-concrète music.Without a structural framework the students floundered.Since most of these students were media studies majors with no background in music composition, they could not rely on their understanding of the design of music to design these sound compositions.But each student had taken a foundations course in two dimensional visual design.As an experiment I took a abstract painting from the wall in my office and asked the sound design students each to create a sound composition equivalent to the visual work of art.I explained that the students were to study the visual art, break it down into separate visual elements, then find sounds equivalent to each element.Finally, they were to study the arrangement, or composition, of the elements on the canvas and arrange their sound elements so they seemed equivalent in time and sonic space to the arrangement in visual space of the visual elements.
This process of semiotic transformation greatly improved the structures of the student sound compositions.In the following semesters we repeated the assignment successfully using visual works by a number of artists including Ann Wilson, Elsworth Kelly, Mark Rothko, Cy Twombley, and Jackson Pollack.Most recently I expanded the technique by showing two works of visual art and asking students to create a composition which begins with the form of the first visual design and gradually develops into a piece having the form of the second visual composition.In all cases the visual designs presented are non-representational in order to avoid the effects of stereotyped experience discussed above.
The results can be summarized as follows: 1.
By limiting the composition to elements re-mapped from the visual model, students used fewer different sounds and made better use of repetition and variation.

2.
Greater structural unity resulted from their "seeing" in the visual model how to think of the sound composition as a structure having multiple overlapping elements, not just a string of linear events.

3.
In having to do the semiotic transformations, the students improved their analytical skills by learning how to break down visual and aural structures into individual elements.

4.
By re-framing their visual experience aurally, the process of composition became focused in the brain, with the tools of the sound studio a means of execution rather than a means of composition.

Visual Design Based on Visualization of Sound Experience
During a recent sabbatical leave, I tested the possibility of reversing the process described above: to use sound composition as a pedagogical aid in teaching visual design.When presenting a student with a visual design problem one often finds that student solutions tend to copy other designs the student has seen.Indeed, some teachers actually show worked-out solutions as models.But this tends, in my view, to promote an emphasis on "style" rather than on true design.I once joked that it would be better to teach visual design by radio, but then realized that using sound might provide a way to train students to come up with original visualizations that are not copied from already seen "styles."Basing visual art on sound is not entirely a new concept in teaching art and design.In addition to the "listen and paint what you feel" exercises noted above, some major artists have experimented with the visualization of music.Paul Klee, in his Notebooks [11], describes exercises in basing visual art on musical rhythm and structure.He includes an analysis of a three part passage by Bach, but most of his analysis seems to have been made from the musical score rather than from the sound itself.
For my teaching experiment, I designed electro acoustic sound clips produced by computer sound synthesis.As with the visual art the sounds were non-representational. Again, too, the task was analytical: to find elements in sound that can be represented by equivalent visual structures.
Students were trained to make the transformations a step at a time.Small, short sounds were produced and students asked to produce visualizations of them.Then longer sound passages were presented and students were given time to create drawing which represented both individual sound elements and the relationship of these elements to each other.In the most advanced stage four minute compositions were played and students were asked to create color paper cut-out collages representing the sound piece.
Several results were noted: 1.
There was sufficient formal consistency between both the elements and overall compositions to indicate that the semiotic transformation from the sound events were genuine.An example can be seen in the three drawings below, each a different student representation of two plucked string notes, one louder than the other:  The students developed a remarkable ability to articulate their reasons for making visual choices based on their sound perceptions, thus supporting the analytical purpose of this pedagogy.4.
Many of the student visual compositions were interesting and original visual designs completely pleasing on a purely visual level.5.
Many of the students felt the experience helped them to understand the design process better.

Conclusion
Semiotic transformation can be a useful tool of visual design education and visual composition can be used as tool of teaching sound and music composition.With modern computer tools it is easier for a designer to work in both visual and sound domains.Thus a multi-sensory pedagogy as described in this paper may be an appropriate direction for design education in the 21 st century.

Figure 1 :
Figure 1: Student sketches by Bill Miller, Frank Corridori and Sophie Phan

A Pedagogy of Creative Thinking Based On Sonification of Visual Structures and Visualization of Aural Structures
ICAD'98 6