A Robot That Draws and Shades with Tactile Force Feedback Sensed Through a Pencil

This paper presents the current technical aspects, design rationale and drawing capabilities of a drawing robot in development. The robot is fitted with a load-cell sensor to provide tactile feedback – a sense of touch. The paper provides preliminary results of a robot able to autonomously control the pressure placed behind a pencil when drawing. The robot is also able to dynamically adapt to irregularities of the drawing surface and its own mechanics. The paper presents technical studies, which demonstrate the various capabilities of the robot to draw and shade.


INTRODUCTION
This paper documents further developments of a drawing robot initially presented in (O'Dowd 2018), shown in Figure 1. This paper documents the current technical aspects, design rationale and drawing capabilities of the robot. The main contribution of this paper is to document the addition of a load-cell sensor to allow the robot to discern tactile feedback through a pencil during the process of drawing.
The development of the presented drawing robot fits within a broad aspiration to explore the extent to which a robot can be built and programmed to exceed our expectations. In order to make such an assessment of a robot, a robot must do something. In this case, the robot is being developed to draw. Drawing has been selected as a task because we, as humans, demonstrate a remarkable capability to discern implicit and abstract information from visual stimulus (Freedberg & Gallese 2007, Pelowski et al. 2017. A drawing robot is therefore of interest not only because it can autonomously do (act) but also because it leaves a lasting trace (product), which can be read as representative of the characteristics of the machine or the creator.
Recent relevant robots within the field of robotic drawing include a painting robotic arm (Lindemeier et al. 2015), a drawing robot arm (Tresset & Leymarie 2013), a robotic arm producing calligraphy (Mueller et al. 2013), a painting humanoid robot with multi-fingered hands (Kudoh et al. 2006), a Cartesian gantry robot producing calligraphy (Kwok et al. 2006) and a humanoid drawing robot (Calinon et al. 2005). Each of these implementations utilise visual information as the source of input for drawing. Furthermore, Kudoh et al. (2006), Kwok et al. (2006) and Calinon et al. (2005) all utilise visual information during operation to detect the position or deflection of the drawing tip upon the drawing surface. This paper is quite specifically interested in a sense of touch, and perception of physical aspects such as materials. The closest recent drawing robot with an applied sense of touch is a compliant-motion humanoid drawing robot producing graffiti (Berio et al. 2016). The robot is described as actively sensing and maintaining a consistent pressure behind a pen to translate a virtual drawing trajectory to the drawing surface.
The work of Berio et al. (2016) is also relevant because they implement 'low feedback gains' for position control in order that the robot behave more like a human arm, and to pursue 'natural motions' which are then assessed aesthetically. These elements link the motion of the robot body and the embodiment of the robot controller to the resultant drawn work.
The apparent simplicity of drawing makes for a fine example of the challenges still latent in robotics towards creativity, embodied artificial intelligence and automation. This paper reveals some of the complexity related to simply using a pencil (in pragmatic terms), which is easy to overlook from a human perspective. How to use a pencil expressively is a problem of greater complexity still.
In the context of expressivity, the presented drawing robot is being developed with a bottom-up methodology, beginning with physical construction upward toward autonomous behaviours. With a bottom-up approach, it is hoped that all facets of the robot (e.g., mechanical resonances, software constraints, material perceptions, etc.) can eventually be fully incorporated into its total operation, and then perceivable as expression in the drawn product.
The drawing robot has the form of a delta robot (Wikipedia s.v. "Delta Robot"), whereby three arms are configured to move an end-effector through 3D space. The delta configuration was chosen because the moving element has relatively low mass, which is conducive to rapid changes of movement. Rapid changes of movement are common within drawing and writing (Hollerbach 1980, Thomassen & Teulings 1985, Plamondon 1993. The drawing robot end-effector carries an assembly to present a drawing implement to a drawing surface. The drawing robot has previously been used to draw with either a rollerball pen or with pencil.
In prior work, the robot drew in open-loop control, meaning that it had no sensory perception or feedback explicitly related to the process of drawing. In informal terms, the robot had no sense of touch.
In prior work all sensory feedback related strictly to the position control of the robot. The robot was instructed to move in a pre-determined sequence, algorithmically generated by digital image processing. In which case, the robot physically operated with the assumption that a drawing implement was fitted, and with the assumption that a drawing would be successfully rendered to an extant surface. Importantly, the robot could not discern these aspects autonomously. Therefore, to make an open-loop implementation effective, the robot had to be calibrated with a priori knowledge of the height of a drawing surface relative to the functional end of a drawing implement.
A drawing robot with an open-loop control of the drawing end-effector presents several immediate problems, especially with a pencil.
First, a drawing implement such as a pencil wears down with its use. This problem is astonishingly obvious, but non-trivial to compensate for robotically. Wear is not consistent pencil-to-pencil. In Europe, the HB pencil-grading standard is common (Wikipedia s.v. "Pencil"). Numbers such as 2B, HB and 2H indicate the degree of graphite hardness. It also stands to reason that the pencil will wear relative to the force of application, to its pose, and in relation to the friction given from a drawing surface. Furthermore, wear can be abrupt, such as the pencil lead breaking away. These aspects are sensed and solved tacitly by humans, developed through formative experiences (Farokhi & Hashemi, 2011). When deemed necessary, a human drawer rotates the tip of the pencil, or uses a pencil sharpener. Developing a drawing robot serves as an immediate provocation on these tacit elements and the wonder of human drawing capability.
The drawing robot has previously been used to produce an edition of 60 prints made with pencil. Using a pencil required a recalibration of the robot every three prints. Beyond three successive prints, the pencil shortened and located incrementally higher above the drawing surface, and ultimately the point of contact for drawing was lost.
The shortcoming of pencil wear can be mitigated by utilising a ballpoint pen, or other implement with a stable functional tip replenished from a pigment reservoir. This approach has been commonly adopted, such as in (Berio et al. 2016, Tresset & Leymarie 2013. In which case, the remaining problem is to ensure a good point contact with the surface. Point of contact between implement and surface can be mitigated through well-considered mechanical design. A good example of such design is pen-plotters. Pen-plotters were utilised to computationally physically render images before other computer driven printers such as inkjet (Dietrich, 1986). Pen plotters work by moving a pen across the drawing surface. Pen plotters are designed with an exactness such that the drawing implement and drawing surface maintain a strict relation to one another, negating a calibration process. Pen plotters typically utilise an electronic solenoid to raise and lower a pen (to break surface contact), whilst a spring coupling creates a consistent pressure behind the pen when in application to the surface.
These elements of pen plotter design ensure a consistency of drawing. This consistency has been enjoyed by artists, such as the Algorists, who pair the elegance of the algorithm with a continuous, clean line (Verostko n.d.). This spirit is still alive today, with popular pen plotters such as the AxiDraw, artists such as (Hoff n.d.) and the #plottertwitter Twitter community (Twitter s.v. "#plottertwitter").
The aspirations for the presented drawing robot are that, in the future, the robot might be capable of successfully using drawing materials of even greater variability than pencil such as charcoal.
This paper takes a step towards this ambition in the autonomous robotic control of pencil tone in the resultant drawn mark. It is important context to note however, that in aspirational terms, it is not a consistency of mark, which is sought (a 'clean aesthetic'), rather a variability of mark which can be intelligently and autonomously exploited by the robot.

TECHNICAL SPECIFICATION
The following section describes the technical specification and design rationale of the drawing robot.
The robot has been conceived as an on-going project, whereby features and capabilities will be incrementally added and studied. The whole project is an exploration of how the holistic construct of a robot can be utilised and received as a form of creative expression. As such, the robot has been built by the author from the components in Figure  2, and all software is being iteratively codeveloped. This process provides the author with an intimate understanding of the robot at all technical levels, as both tool and medium, as we might understand the knowledge and skill of a craftsperson or artist.
The robot has been constructed from readily available materials. The robot has been intentionally designed for a low number of parts and relative mechanical simplicity ( Figure 2). This intention has been formed within the context of designing a robot to study the qualities of the marks rendered in relation to the robot itself.
In the context of studying marks made, it was decided that the robot should be feature-minimal as opposed to feature-rich, following the principle of Occam's razor (Wikipedia s.v. "Occams's Razor").
In being feature-minimal, it has been reasoned that it will be easier to discern which aspects of the robot are contributing to the quality and character of the drawn marks. Conversely, a feature-rich robot may obfuscate the key aspects and parameters that influence the drawn marks. In this way, the development of the robot straddles both the Rational and Romantic views of art and creativity (Sawyer 2011); the work attempts to be formally composed in pursuit, but encompassing of subjective qualities -a robot is being designed and studied to afford expression.
An obvious contention regarding complexity would be that a delta type robot is not the simplest or most elegant mechanical solution to draw ostensibly 2D marks. A straightforward solution would be the gantry style configuration common to pen-plotters. In opposition, the delta configuration was chosen because the moving element has relatively low mass, which is conducive to rapid changes of movement. Rapid changes of movement are common within drawing and writing (Hollerbach 1980, Thomassen & Teulings 1985, Plamondon 1993).
The robot is also expected to adapt to sudden changes in drawing implement length or to work on non-uniform surfaces, requiring movement in 3D space. The delta configuration also has a relatively unobstructed and large workspace between endeffector and surface. Whilst a robot arm could provide similar operation (Calinon 2005), a delta configuration provides an intrinsic mechanical stability with its parallel linkages. The delta configuration was adopted as a machine elegantly balanced between these requirements.
Whilst it would be possible to construct a complicated robot, which produces complex marks, the aspiration is to correlate the robot and drawn marks intelligibly. The author wishes to understand what makes marks appear interesting, and how the holistic construction of a robot can be utilised toward creative expression. In this way, the robot itself is an instrument of provocation on the act of drawing. In developing a drawing robot, the human act of drawing must be studied, related to a robotic form and described in robotic technological terms. The robot utilises three NEMA-17 stepper motors, which directly drive the arm-linkages to the endeffector. These motors are under significant load due to the lever forces, and the direct contact forces between pencil and surface. It is possible the motors may not achieve their commanded stepincrement when placing a drawing material under stress. Therefore, each stepper motor is close-loop controlled with respect to known rotation derived from CUI AMT120 rotary encoders (CUI Inc s.v. "AMT10").
The robot utilises a Microchip chipKit uC32 development board for computation. The uC32 is a relatively high capability microcontroller, with 80Mhz 32-bit MIPS, 512K flash and 32K RAM. The uC32 provides a sufficient solution to compute the forward and inverse kinematics at a consistent rate of 100hz, whilst performing the following other operations.
The uC32 receives movement commands from an off-board computer utilising an adapted GCode protocol. The position control of the robot has two complementary parts. First, the commands include the demanded Cartesian coordinates in millimetres -x, y describe a position on a 2D plane, and z being height. Secondly, the command includes a hardness value, denoting a demand of pressure to exert on the pencil in contact with the surface. Hardness is derived from the load-cell sensor readings. The load-cell is the FC22 3120 Compression Load Cell (Measurement Specialities s.v. "FC22").
The load-cell sensor can be thought of as an upside-down weigh-scale. When the pencil is brought into contact with the surface, the force exerted by the robot (the 'weight') is transmitted up through the pencil and measured by the load-cell (see Figure 3). The pencil is aligned perpendicular to the horizontal level. The pencil is free to slip through the end-effector assembly, but held closely enough to provided lateral stability. However, the pencil is pushed up against the load-cell by a spring and collet, which ensures that there is a stable nominal sensor reading when the pencil is not in contact with a surface. Any contact with the surface increases the load-cell reading from nominal. A spring and collet stop the pencil falling away from the load-cell under the influence of gravity, and therefore stabilises load-cell readings when the pencil is not in contact with a surface.
The load-cell sensor readings are low-pass filtered. The filtered reading informs a PID control algorithm with respect to the hardness demand. The PID algorithm operates at 100hz. The control signal output of the PID is used to adjust the demanded z position. In this way, a z demand of 0mm (contact with the surface) can be autonomously adjusted by the robot to increase or decrease the force detected through the load-cell to match the hardness demand. In practical terms, this can be thought of as the robot attempting to draw through the surface (negative z), or to lift the pencil away from the surface, if either are necessary.
The resultant x,y,z target is translated to stepper motor actuation. Because the kinematics and loadcell feedback are computed in real time at 100hz, it is possible to directly influence the robot operation by touching the pencil. If a human attempts to lift the pencil, the robot interprets this as a deviation of the load-cell from the demand hardness, and so lifts away from the surface. It is also the case that other dynamic changes, such as a break of pencil lead or an inclined surface, are fully autonomously compensated for.
At this stage of development, movement paths are streamed from an off-board computer and take the form of simple geometric paths. The drawings are lacking in any intentional expression, owing to the pragmatic study presented by the paper. The drawn paths have been determined to elucidate the performance capability of the robot in drawing and shading.

TECHNICAL STUDIES
The following section presents preliminary results from studies made of the drawing robot. In all these tests, the robot was instructed to draw in a scanning motion back and forth across the x-axis, whilst progressing across the y-axis when a line was complete. In effect, all tests take the form of a square drawn onto the surface. During these tests, the robot was instructed to draw with dynamic control of the pressure behind the pencil.

Shading reliability
The reliability to shade with a 2B pencil with various tone was tested by progressing the hardness demand with respect to the y-axis of motion. The hardness demand was incremented uniformly from 0 to 130, representing a total relative increase of 0.6 volts from the load cell. The sample, in Figure  4, was drawn by the robot, measures 100x100mm, and was produced 10 times. The 10 drawings were then digitally captured and processed at the pixel level to determine a dataset of HSV brightness. The mean and standard deviation of the brightness values were taken per row of the digital images, which are complete x-axis rows of the rendered drawing. The graph in Figure 5 illustrates a clear relationship between HSV brightness and the hardness demand placed on the drawing robot. The S.D error bars indicate the degree of variability between the 10 drawings.

Pencil types
The same test for shading reliability was constructed with 2B, HB and 2H pencils in order to ascertain if a distinction can be made between pencils. The robot was instructed with same progression of hardness demand [ 0 : 130 ] and travel across a 100x100mm square. Figure 6 indicates how the different grades of pencil produce different tones of marks across the hardness range.

Surface perception
For this test the robot was instructed to draw with a constant hardness demand whilst traveling the same 100x100mm square as in previous tests. The autonomous adjustments to the z-axis made by the robot were recorded in relation to x y position. This relative deviation of movement by the robot on the z-axis was a product of maintaining a stable pencil pressure. With a fixed hardness demand, the z-axis deviation indicates deviation of the surface and non-linearity in the robot mechanics. The deviation apparent in Figure 7 indicates a total range of 9mm adjustment required to keep the pencil at a stable pressure. Figure 7 has two components. First, the concave shape is a known by-product of the delta configuration mechanics. This characteristic is normally mitigated in software, but has not been implemented for this robot yet. The concave shape appears to lift approximate 5mm across the 100mm travel. The second component apparent in Figure 7 is a general elevation toward [100,100] (x and y), where z-axis deviation reaches the a maximum of 9mm. This indicates that the robot is not configured parallel to the drawing surface -indeed, the robot was not carefully positioned by the author owing to the knowledge that the robot should be able to compensate.

TECHNICAL DEMONSTATIONS
The following demonstrations were conceived to test the responsiveness of the capability of the robot to meet a varying demand of hardness.
First, the robot was instructed to travel the same 100x100mm square with a continuous progression of tone along the y-axis, but to suddenly invert the hardness demand. The occurrence of this inversion was mapped to a superimposed circle for algorithmic convenience. The effect of this is apparent in the drawn sample presented in Figure  8. Next, a digital image of a daisy was mapped to the same travel of a 100x100mm square. The pixel brightness of the image as used to set the hardness demand for the robot. The result is shown in Figure 9. As a last demonstration, a photograph of the author was used in the same process as the daisy, but across a larger 130x130mm area. In this drawing, more considerable distortion is apparent on the xyplane. This is likely caused by the irregularity of surface and mechanics shown in Figure 7. However, the drawing robot is able to make a reasonably faithful reproduction, indicating a useful dynamic response of the robot to hardness demand with respect to x and y travel.

DISCUSSION
This paper has presented the current technical aspects, design rationale and drawing capabilities of a drawing robot in development. The paper has provided preliminary results of a robot able to autonomously control the pressure placed behind a pencil when drawing. As such, the robot is also able to dynamically adapt to irregularity in the drawing surface and its own mechanics. In these ways, this paper presents a robot with tactile feedback -a sense of touch. Furthermore, the tactile feedback is usefully incorporated into the capability of the robot to draw and shade.
As a demonstration, the robot was applied to reproducing digital photographs. The digital photographs present a reasonable challenge to the robot to control the hardness, because the source photos have quickly varying tone across the 2D plane. However, photographs are very different representation than drawing. In future work, the drawing robot will be pursued with a returned emphasis of movement, gesture, and the quality of mark this may create.
The drawings of the photographs were produced with a continuous line of varying hardness. As such, the drawings are composed of a single layer of graphite, a single passage of the pencil. The subtlety of the drawings are likely related to the range of tone shown in the technical studies ( Figures 5 & 6), which places a HSV brightness value between 180 to 230. For darker tones (lower HSV brightness), multiple passes of a pencil could be employed.
During these technical studies, the qualitative performance of the robot was also observed. The range of hardness demand between [ 0 : 130 ] was appropriate for the physical capability of the robot. At greater demands, the stepper motors were unable to respond. There are several factors involved in this behaviour. For instance, longer pencils require the end-effector to move at a higher z position, and therefore effect the mechanics of the arms and resultant lever forces.
Furthermore, it has been observed that the pose of the pencil appears to have a significant effect on the quality of the travel over the page. At a perpendicular pose to the surface, there is a surprising amount of friction, occasional jamming, and the length of the pencil appears to amplify vibration. Reorientation of the pencil will require some re-design of the pencil assembly, and how forces can be detected through the pencil.
The capability of the robot to utilise touch creates new potential on how to inform what to draw. An obvious approach to drawing is to take visual information. However, the experience of materials appears to greatly inform or shape human expression. The work presented in this paper will be carried forward to investigate how materials and physicality can be understood by a drawing robot, and utilised in an intelligent expressive way.