Developing a physical colour input device for studying digital sketching in design

This paper presents a development process of a sketching system prototype, with a primary focus on a component that allows the user to input colour values directly from the environment. The device can be either held in hand, or the colour materials can be fed to it. The practical aim of the research is enabling a richer variety of interaction of colour in sketching type activity, and to provide insight into the design tools developed from a design perspective. The paper describes the design of the system and the user tests, conducted with four designers, who tested different approaches towards the prototype.


INTRODUCTION
Sketching is an important part of the dialogue between the designer and the designed object.Sketches are used to develop and record ideas and present them to others.The multitude of functions that sketching has and their implementation on the computer poses challenges for designing digital sketching tools.This paper describes a study about the development of a system for sketching, the central component of which is a device that inputs colour values from the physical environment.
The prototype required the design of electronics, two software programs and the casing.To evaluate the success of the approach a series of user tests were conducted.In each test, the device was used in three different ways, to probe for different possibilities.
The study is used to examine design tools and their development from the designers' point of view.Building a sketch tool serves as an entry point into researching design tools.The research has also the practical aim of enabling richer possibilities for colour in design software.Ultimately the prototype would become a non-intrusive portable system for drawing sketches.The paper describes the device design and the user test outcomes.

BACKGROUND FOR SKETCHING
Design authors have mostly discussed sketching in relation to the paper medium, probably still the most common form of sketching.Sketching is a conversation with the drawing, and should be seen as one possible medium for exercising reflection-inaction, a dialogue with oneself.(Schön, 1983).Bryan Lawson (2004) categorizes design drawings according to purpose rather than appearance.For example, experiential drawings are abstracted records of appearances, and proposition drawings are a kind of thinking drawing where moves are mode.Ching, a long time author of popular books on design and design drawing, similarly described sketches as "speculative drawings", putting weight on their private nature.(Ching, 1997) In addition, Suwa and Twersky (1996) have pointed out that professionals use the sketch to consider non-visual functional relations not explicitly seen in the sketch.
To summarize, the appearance of the sketch (plan, section, perspective et c.) may not always indicate the purpose of the sketch.Ideas are started on paper and the sketch begins to suggest things that were not part of initial, explicit intent of the designer.Lines on a sketch can represent appearances, but also immaterial things like propagation of light and sound or routes of movement of people.The objective of the sketch may not even have a visible counterpart in the sketch.
'Sketching' then describes an attitude rather than a particular medium.This paper concentrates on drawing and use of colour.Sketching is here understood to be a loosely organized activity, which has more significance in the early stages of design, where ideas and appearances need to be generated and evaluated quickly.

Sketching as a digital design tool
A basis for development was sought from literature that combines traditional design and computing technology, where digital sketching or its equivalents have been addressed.McCullough (1996) considered the hand-eye interplay of the pointing device and the screen as the beginnings of a digital craft.It is important to enable flow and continuity in informal tools.Sevaldson (2005) used Arnheim's concept of visual thinking to explain how designers work with dynamic visual representations He emphasizes the role of computer in producing generative, rich visual material with partially unexpected, emergent results.To Sevaldson, manual sketching is also a visual thinking tool, a graphic seed for design.Complex layering and dynamic manipulation can be quickly applied on digital sketches.Hummels (2000) studied hand gestures as a kind of primal visualization, and proposed ideative direct modeling tools more attuned to senses.Hummels evaluated digital sketchboards and criticized complex menu interfaces and 'button' style interaction as detrimental to the sketching process.According to Hummels, sketching, which is naturally a two-handed activity involving both the pen and the surface, has become one-handed in digital sketching.Also, computer sketching tends to force focus on single screen image, whereas multiple sketches can populate a paper surface.

The design brief
Our research has the following aims: Develop means for studying sketching and image making activity within design context, and develop a computerized sketch board that uses environment as source of colour.(figure 1) The outcomes are used for the purposes of studying computational design tools, as outlined by the above literature.
The initial system was tested with four designers.

RELATED WORK
The prototype work connects to two relevant developments.The first concerns augmenting sketching with computation.
The second complements digital drawing with physical interfaces for colour.

Augmenting sketching
Computer technology offers numerous ways for augmenting sketching."Magic paper" approaches support either engineering type whiteboarding, where sign elements are interpreted into working components within a virtual environment.(for example Alvarado and Davis, 2007) Also, model retrieval from sketches continues to be of interest.
(for example Chen et al., 2008).Block et al. (2008) presented ways of bridging between physical and digital in the sketching of interactive surfaces.
As the computer is made to interpret drawings, the drawing easily becomes an interface to a software function, whereas drawing as free-form activity can become overshadowed.The person using must adjust to the software more than would be required in true sketching.Our intent is to understand sketching and its digital counterpart as an activity in itself, only augmenting it step by step.

Tangible colour interfaces
A tangible colour brush, implemented with video camera, has been previously used in studying children's creativity.In the studies, the same device is used for both picking up colour and textures and painting them on a surface.(Ryokai et al, 2004.)Our device is portable and wireless, able to input colour and feed it in real time into the painting software.Our focus is on enabling a variety of interaction modes, such as two-handedness and using an immobile device.Enabling further modifications by the users is also welcome.

DEVELOPING THE DEVICE
The following describes the development of the device, the motivation behind design choices and challenges that were met in the development.

Technical components
The sketching system has two main units specifically created for the purposes of this research.The first unit is the physical colour input device itself, which uses a colour sensor to capture surface colours.The other is the sketching application, run in the computer.The application uses the colour values sent by the arrow unit.Wires between the parts are not shown in the schematics.
The development of the device began as experimentations with a custom built microcontroller board connected to a colour sensor module.The possibilities were explored by trying out different enclosures for the input module.An LCD screen connected to the microcontroller was helpful in initial stages for reading the sensor values directly.This helped build an understanding of the dynamic range and the responsiveness of the device.Minor but persisting technical problems and inconveniences prompted a move to more off-theself parts.The device was rebuilt with the Arduino Duemilanove board.The colour sensor module (figure 3), containing the LED lights, was retained from the earlier configuration.The Arduino building blocks also made simplified the transition to wireless.
The portable device contains: •

Design of the casing
The device casing is shaped like an arrow pointer, to be either held from the handle part, or placed sensor upwards on the table.The arrow shape was arrived at as a humorous gesture, resembling an operating system pointer.The shape has stuck as it fits hand and allows a good positioning of the different parts.
The colour sensor module is placed near the tip of the device, near to the surface to be read.The casing prevents outside light sources affecting the sensor reading.This means an internal light is needed for lighting the surface.(figure 2) Two LEDs are located near the colour sensor element.The 9V battery is placed within the handle part, and the board can be fit inside the bulge.
The clear separation of parts helped the development as the parts could be removed easily for maintenance.For the duration of user studies the casing top was covered with a cardboard lid, taped over the device.

The sketching software
The other part is the sketching software, made specifically for the purposes of the present study.
The primary intent of the study was not to develop new digital painting software.For the purposes of the study it was desirable to test the colour interface using simple software and to concentrate on the significance of colour while sketching in a very limited context.
The software was written with Processing.The program is a loop which reads the colour values from the serial port, and reads mouse input at the same time.The application shows an empty, light gray screen, with the currently input colour displayed at the corner.The colour indicator is updated constantly (20 times a second) as the device keeps sending sensor readings.The software does a simple linear calibration of the colour input using values derived from readings of white and black surfaces.
The software records all pen coordinate positions and colour input values into a log file.The drawing process can be played back or otherwise analysed at a later time.
In addition to the two components, a graphic tablet, touch screen or a mouse has to be used to actually draw the sketch.The present study uses a small A5 Wacom graphic tablet with two-ended pen.The pen tip is used to create fine line whereas the other end produces thicker, more brush-like line.The system presently does not support pressure sensitivity or pen angle input.
There is no undo, erase or retrieve features built into the software.If a person wishes to 'erase' a part of the sketch, the portion must be drawn over with the desired colour.The decision to omit undo and other conventional functions followed from the belief that sketching should remain as free-formed as possible.In principle, the log stores every sketch state into memory, but in this study the log contents were not accessible to the user.

TESTING THE DEVICE
Tests with designers were conducted to evaluate the success of the device.The study subjects were presented with the colour input device, graphic tablet and a laptop computer running the sketching software.Four people tried the device, all with a design background in industrial design, architecture or furniture design.
A dozen differently coloured cardboard sheets the size of a postcard were spread over the table surface.All the equipment was positioned on a table.The primary interest in these tests was the functioning of the colour input as a replacement of normal colour widgets commonly present in visual design software.

The three approaches
Three different approaches were tried within each test situation.At first, the people were told to hold the device in their hand, and position the device over surfaces of objects.The tablet would be used for drawing.This means they could leave the device on a coloured surface and change focus freely on the drawing process.Then the device was turned around, and instead of moving the device the subjects positioned coloured pieces of paper over the sensor.
In the third approach, the subjects were encouraged to try simultaneous scanning of colour values at the same time when they were drawing.This would require them to coordinate two-handed activity.This was achieved by changing the paper positioning while they drew.
Summary of the approaches: • Hold device: The device is moved over different coloured surfaces.• Hold cards: The device is stationary, and coloured cards are brought over the sensor • Two-handed: The device is stationary, and the subject attempts to draw and change the colour simultaneously The tests are not comparative in a rigorous sense.The three approaches were meant to produce a variety of responses and directions for development.As the people were designers it was hoped the situation could provoke further design insight for the purposes of new uses for the device.The subjects' opinions were collected during the tests.All sessions were recorded with video.

EVALUATION
The following gathers the impressions and results of the development work and the user tests, combined with possible future directions.The responses to the three approaches are gathered in Table 1.

Challenges in the use
The concept of using environment for colour input appeared to work, but becomes less useful in this type of sketching if the colour surfaces are not right at hand.Placing the cards further away necessitated a change of focus between screen and the table, a problem which becomes worse with the graphic tablet as an additional focus.

Continuous colour change
Most potential of the device comes from twohanded interaction, where the colour can be continuously changed at the same time when the picture is drawn.The two-handed interaction worked better when the device was upside down, on the table, and not held in hand at all.Holding the device in air is too cumbersome for two-handed use.
One exhibited more dexterity than others, and felt the device made the mixing of colours more 'real', reminding of the way colours are mixed in paintings.This user had previous long experience with graphic tablet, which was helpful.The two-handed technique helped in modulating the colour during drawing.In one test, a readymade colour gradient was also tried.The gradient was printed on a post-card sized surface, which was then be used as a two-dimensional input.

Directions for studying sketching
In retrospect, the initial starting point of replacing colour input with a physical device appears too simplistic when studying designers.Designers have wide interests into the possibilities of the tools.
There is no reason to cancel possibilities for building sketching environments, or other uses for the colour sensor not foreseen by the researchers.The line between sketch, prototypes and visualizations is blurry, and this needs to be addressed.The device could be expanded to a kit with more ready-made materials for building colour configurations.Creating a colour widget then becomes a mechanical rather than a software programming task.There are possibilities for altering the interface modality by using for example rotating and round objects with coloured surfaces.
The form of the device can have large influence on the ways it can be used.Two-person use seems another possible source for novel experimental approaches.The task of drawing can be split between two people, one drawing the lines, while the other provides colour values with the wireless device.Although this does not seem immediately practical, it can be taken as an indication of the flexibility of the device.
It should be noted, that although all managed to cope with the device, some comments indicated that it required effort.There is disparity between achieved results and expressed frustration.For example, they would handle two-handedness, talk at the same time, yet remark how difficult it was.It is not desirable to produce strain so this needs to be addressed.

Technical issues
The device should support larger variety of surfaces.The interaction with flat surfaces works well in the context of selecting and holding a colour, but picking colours from materials and objects becomes limited by object shapes and rough surfaces.The shaping of the device and the sensor opening needs to be redesigned.
Designing multiple software and hardware components can be laborious.It would also be more obviously practical to have the colour interface device work with already existing drawing and photo manipulation software.
For drawing, the graphic tablet could be replaced with a touch screen.Then the users would not have to cope with both unfamiliar graphic tablet and the new interaction device.

CONCLUDING REMARKS
The paper presented a development of a physical colour input device for designers to use in sketching type activity.The device was complemented by sketching software on the computer with a built-in logging of the drawing process.The user tests show the device can work as an alternative to colour widgets in the context of drawing simple sketches, although a level of manual dexterity is required.The results also suggest expanded possibilities for interfacing colour into design tools.

Figure 1 :
Figure 1: Basic concept.The hand held device scans colour values continuously.

Figure 2 :
Figure 2: Schematic positioning of the prototype parts.Wires between the parts are not shown in the schematics.

Figure 3 :
Figure 3: The custom colour sensor module placed within Styrofoam housing.The 5mm x 5mm sensor unit can be seen near the tip of the module.
The colour sensor module with RGB sensor • Arduino Duemilanove board • Arduino XBEE shield • Wireless Digi XBEE zigbee module • 9V battery The arrow shaped unit contains an analogue colour sensor, which sends RGB (Red, Green, Blue) values.These are translated by the Arduino board into digital values and sent wirelessly to the computer.An additional Arduino, connected to the computer, handles the reception of the wireless data.

Figure 4 :
Figure 4: A doodle made using a ready made printed gradient card, by subject S2.

Figure 5 :
Figure 5: A doodle showing use of colour for depth perspective effect (left), by subject S3.

Figure 6 :
Figure 6: A figure drawn from life, by subject S4.

Table 1 :
Responses to different modes of using.