A Rendering Model for Emotional In-air Handwriting

According to graphology, people’s emotional states can be detected from their handwriting. The purpose of our research is to propose a 3D handwriting rendering model which has emotional capabilities. We developed an in-air handwriting system and selected eight emotions (afraid, angry, disgusted, happy, sad, surprised, amorous and serious) to be conveyed using five rendering parameters (weight, smoothness, tip of stroke, ink density and ink dryness). We then conducted an online survey to determine the optimal parameters’ values. A first result shows that there is a positive correlation between the Euclidean distance between two emotions on the valence-arousal emotion two-dimensional model and the distance between the calculated rendering parameters’ values. Another online survey then revealed that four of the emotions (afraid, sad, serious and angry) are well conveyed. The results are less conclusive for the other four emotions, but further work is being conducted to improve the model when the handwriting is shown in conjunction with the other emotional channels of a social robot.


INTRODUCTION
According to graphology, the writer's emotional state, as a higher-level characteristic of personality, can be detected at the time of writing (Grewal et al., 2012;Fairhurst et al., 2015).Kedar et al. (2015) point out that handwriting features like baseline, slant (angulation), pen-pressure, size, margin and zone contribute to revealing one's emotions.
Meanwhile, the emergence of affordable hand tracking sensors enables a variety of applications based on spatial interaction, including in-air handwriting (Vikram et al., 2013;Xu et al., 2015;Kumar et al., 2017).Unlike writing on paper, which can be analysed through its on-surface properties such as pen pressure, stylus inclination and fluidity (Sesa-Nogueras et al., 2012;Likforman-Sulem et al., 2017), spatial interaction-based handwriting is entirely in-air.On-surface properties cannot be easily captured via 3D hand motion sensors, and consequently, the rendered characters do not display the corresponding characteristics that can reveal the emotions of the writer.In other words, the techniques used in graphology are not directly transferable to spatial interaction.Existing in-air handwriting systems used for user authentication (Bashir et al., 2011;Tian et al., 2017) or in place of touch screen-based displays and kiosks (Ding et al., 2017) are unable to render the emotions conveyed in 2D handwriting.The purpose of our research is to propose a 3D handwriting rendering model which has emotional capabilities.
In this work-in-progress paper we first describe the in-air handwriting system we developed.We then explain and justify our choice of emotions and 3D rendering parameters.In section 4, we report the results of a survey we conducted to associate emotions and rendering parameters' values.In section 5, we evaluate the emotional conveyance of our in-air handwriting rendering model.Finally, in the conclusion of the paper, we discuss our plans to apply our rendering system to a social robot and enhance its emotional capabilities.

THE IN-AIR HANDWRITING SYSTEM
We use the Leap Motion sensor (https://www.leapmotion.com)to capture hand motion; C# and the Unity 3D game engine (https://unity3d.com)for the 3D rendering of the handwritten characters.Users can write freely with their fingers in the air and observe in real time the rendering of their handwriting on a computer display (see Figure1).Alternatively, they have the option of immersing themselves in their handwriting by wearing a virtual reality headset.
Leap Motion captures the position of the user's hands and fingers and transmits the data into Unity, where the recorded (x, y, z) coordinates are then processed and rendered as a 3D object.Each individual object is constructed as a result of the writing of one single stroke, i.e., the continuum from pinching the fingers (to start drawing) to detaching them (to stop drawing).

Emotions
One of the most popular model is the valencearousal model, also known as the circumplex model of affect (Russell, 1980).It represents emotions using two dimensions: valence on the X axis and arousal on the Y axis; each emotional term is characterised by its (x, y) coordinates (see Figure 2).The Plutchik's emotion wheel (Plutchik, 1982) is a three-dimensional discrete model which projects 8 groups of emotions at different levels of intensity.
According to Plutchik and based on 10 postulates including varied similarity, varied intensity of arousal, applicability to animals, etc., the basic human emotions are: fear, trust, joy, anticipation, anger, disgust, sadness and surprise.
In (Ekman, 1992) the author, having summarised 9 common characteristics of emotions, such as rapid onset, short duration, unbidden occurrence, etc., argues that the basic human emotions are: anger, enjoyment, fear, sadness, disgust and surprise, followed by contempt, shame, guilt, embarrassment and awe.
For our study, we retained the six basic emotions common to Plutchik and Ekman: afraid, angry, disgusted, happy, sad and surprised; to which we added two more emotions: amorous and serious, to obtain a set of emotions that are represented by a large spectrum of (x, y) coordinates in the valencearousal model (see Figure 2).The figure is adapted from (Paltoglou & Thelwall, 2013).

Rendering Parameters
We aim to create a rendering model that can be universally applied to any handwriting and any alphabet: our choice of parameters is inspired by both Latin typography and Chinese calligraphy.
The PANOSE system, proposed by Bauermeister (1988), has been well suited to Latin-text based World Wide Web pages and other applications as a typeface standard.It uses ten parameters to describe a font type, including: serif style, weight and stroke variation.
In Chinese Calligraphy there are four elementary writing instruments, known as the "The Four Treasures of the Room of Literature", or Wên-Fang-Szu-Pao (文房四宝) (Yee, 1973;Zhang, 2004): the brush, the ink, the brush-stand and the ink-stone.
It has been previously shown that immersion provided by virtual reality benefits the design process in a variety of fields (Whyte et al., 2000;Heydarian et al., 2015).The final parameter selection process was carried out by immersing ourselves in our own in-air handwriting (using a virtual reality headset) and through numerous trials.
The five rendering parameters we chose are: (i) Weight (w): as in the PANOSE system and determined by the radius of the rendered stroke.(ii) Smoothness (s): Inspired by the brush and the brush-stand and determined by the minimum length of one segment that composes a stroke, or more basically, the minimum distance between two sampled vertices.(iii) Tip of stroke (t): Inspired by the serif style and the stroke variation (PANOSE) and determined by the ratio of the radius to the writing speed.When the writer is writing fast, the stroke will appear relatively thin while its tips are thick, and vice versa.(iv) Ink density (de): Inspired by the ink and the ink-stone and determined by the opacity of the rendering material.(v) Ink dryness (dr): Inspired by the brush and the ink and determined by the texture of the rendering material, which can be coarse or smooth.

DECIDING THE PARAMETERS' VALUES
Having implemented the 3D handwriting system and empirically determined five rendering parameters, we designed a survey to gather respondents' opinions on which rendering parameters' values are most effective at conveying the intended emotions.

Design and methodology
For each of the five selected parameters, we created three handwriting samples by varying the value of the parameter.For example, the parameter 'weight' was given the following values in meter: 0.001 for 'weak', i.e. thin; 0.004 for 'neutral'; and 0.008 for 'strong', i.e. thick.To avoid the combinatorial explosion of conditions and hence of the number of handwriting samples, each parameter is made to vary independently of the others, in other words, when testing the 'weight' parameter's values for a given emotion (e.g.happy), the other four parameters are kept to a fixed value.We obtain a total of fifteen handwriting samples to be tested against each of the eight emotions.Figure 3 shows the three samples obtained for the three values of the parameter 'weight'.The Chinese characters we used and which appear in Figure 3 convey no particular meaning, they are typically used as headings in an itemised list (such as A, B, C, D).The survey is made of eight sections, one for each emotion.In each section, five sets of three handwriting samples are shown (one set for each parameter), and the respondents are asked to choose the sample (one among three) that best convey the intended emotion.The same sets of handwriting samples are used for each of the eight emotions.To avoid a survey that is too long, we divided it into two independent surveys, each one covering four of the emotions.
The survey was distributed online to a group of University students on a Telecommunication Engineering Chinese/English bilingual programme in China.75 students took part in the questionnaire that covered the four emotions afraid, amorous, angry and disgusted; 53 students responded to the questionnaire that covered the four emotions happy, sad, serious and surprised.

Analysis and Results
The formula we used to calculate the optimal value of a parameter is as follows: Where i = 1, 2, 3 refers to the value of the parameter.Q is the total number of respondents, q i is the number of people who chose the handwriting sample with value i. P i denotes the parameter, i.e.,

[ ]
(2) R is thus the average value of a given parameter, for a given emotion.
Applying the R values to the 3D handwriting system in Unity, we obtain the eight emotional styles illustrated in Figure 4.

Discussion
We calculated the Euclidean distances between each pair of emotions using (1) their 2D coordinates (x, y) in the valence-arousal emotion model; and (2) their 5-dimensional vectors of normalised parameters' values.Across all pairs of emotions, there is a positive correlation (R=0.41) between the two distances.This is an interesting result, which seems to support the choice of parameters' values that was made in the model: when compared two by two, the distance between samples of emotional handwriting is comparable to the distance between the emotions they are meant to represent.

EVALUATION
We conducted another survey to evaluate the emotional capabilities of our rendering model.Handwriting samples in both Chinese ("甲乙丙丁") and English ("Pepper") were produced for each of the 8 emotions, making a total of 16 samples.42 undergraduate students (21 males and 21 females) on a Telecommunication Engineering Chinese/English bilingual programme in China (same as for the first survey) took part in the on-line questionnaire.For each handwriting sample, the respondents were asked to select among the 8 proposed emotions the emotion they perceived as being conveyed in the sample.
The four most notably recognised emotions are: afraid, sad, serious and angry.Binomial tests with a 95% confidence interval showed that for these four emotions the respondents' choices were significantly different from random chance (afraid: p=0; sad: p=0.0007; serious: p=0.0042; angry: p=0.0453).We note that these four emotions have all negative or neutral valence in the valencearousal model.The emotion afraid was particularly well recognised.The emotion happy was well recognised (the probability the observed proportion comes from a population not equal to .125 (i.e.1/8) being 88.32%, p=0.1378) but was also often confused for serious.The least correctly identified emotions are disgusted, amorous and surprised, which are only perceived by respectively 14.29%, 15.48% and 15.48% of the respondents.The Sankey diagram shown in Figure 5 provides a view on the confusion between intended emotions and perceived emotions.Serious is the most often selected emotion, and was often chosen instead of happy, amorous and disgusted, and to a less extent angry and surprised.Arguably, serious is the most "neutral" emotion (the closest to the centre of the valence-arousal model) and thus the easiest choice to make when in doubt.
To summarise, selecting one emotion among eight by observing a single word sample is a difficult exercise, but the results are rather encouraging.When considering the three most chosen emotions for each written sample, the intended emotion always belongs to the list.Half of the emotions (four) are top of the list, two are second choice and two are third choice.

CONCLUSION AND FUTURE WORK
In this paper, we proposed a 3D rendering model for emotional in-air handwriting.To our knowledge, the simple model we propose is the first model of its kind.We found a positive correlation between the distance between two emotions on the valencearousal model and the distance between their calculated rendering parameters' values.Evaluation results are less conclusive, but it is arguably very difficult to perceive emotion from a very short sample of handwriting.We found that the negative or neutral emotions (afraid, sad, serious and angry) are well conveyed.More thorough testing is still needed, the plan is to test the emotional handwriting in conjunction with other emotional channels such as body postures and movements.
Equipping social robots with emotional capabilities is a very active area of research (Breazeal et al., 2005;Claret et al., 2017).The social robot Pepper, developed by Softbank robotics (https://www.ald.softbankrobotics.com/en/robots/pepper) is known for its ability to convey emotions via a number of channels: posture, movements and voice.We now propose to enable Pepper to perform in-air handwriting using our system, which is technically easier to implement than on-surface handwriting (Potkonjak, 2012).The outcome will be displayed on its tablet (Figure 6).The extent to which the handwriting channel will add to the robot's emotional expressivity and how the rendering model can be refined will be studied.

Figure 1 :
Figure 1: The handwriting system using Leap Motion for hand capture and Unity for rendering.

Figure 2 :
Figure 2: The Valence-arousal emotion model and the eight selected emotions (marked in large capital letters).The figure is adapted from(Paltoglou & Thelwall, 2013).

Figure 3 :
Figure 3: Handwriting samples with three different values for the parameter 'weight'.

Figure 4 :
Figure 4: Handwriting samples with various emotional styles.

Figure 6 :
Figure 6: Emotional social robot Pepper without (left) and with (right) handwriting channel enabled.