Eliciting Writing-like Behaviour in Sign Language through Photographic Representation of Movement

This work constitutes a contribution to the emergence of a common writing for French Sign Language in a graphical or even a typographical framework. Our hypothesis is as follows: in its execution, the gestural sign contains a readable graphic trace that can be visualized with a photographic device. In order to evaluate this hypothesis, we gather a photographic corpus made of isolated elicited signs.


INTRODUCTION
In this paper, we present a new research approach for a writing system of sign language (SL) using visualisation of movements as an extensive technique of investigation and experimentation.Eventually, we will present how visualisation allows us to develop a critical graphic methodology and more importantly, how this method involves the deaf community in the research process by adopting writing-like behaviours.
The GestualScript project aims at contributing to the creation of a handwriting system for sign language through a visual approach inspired mainly by the "graphic method" of Étienne-Jules Marey (1883).Our working hypothesis is that gestural signs produce readable graphic structures.By projecting the signs in a specific graphic environment, we collect neighbouring productions that are used to define legible graphic components and parameters.
It is important to start with presenting the two main principles to consider when investigating a writing system for French Sign Language.One is the visuo-gestural modality which differs from the vocoacoustic modality that characterises spoken language.Sign language relies on gestures to produce signs and vision to perceive them.The entire upper body acts as a medium that combines different parameters: hands configurations, movements, orientation, chest, face to produce signs.This capacity of using several articulations at the same time brings us to the second principle: multi-linearity.Sign language can express multiple information simultaneously, as opposed to spoken languages which are mono linear.Because of that, researching sign language poses a great challenge of visualisation and no graphic system could until now offer the satisfying qualities in order to be used as handwriting although it would undoubtedly have a massive impact on the FSL speaking population.
Technologies and linguistic models are still being developed in order to provide efficient tools of representation and understanding.Some graphic transcription systems exist but most of them have been devised as scientific tools (LSColin 2001).They usually show limitations on the practical side which disqualify them as actual writing.While those systems visualise the linguistic parameters of sign language through either a visual decal of the body, or a symbolic transposition, our approach relies on capturing the trace left by the sign's completion.By claiming that FSL, in its gestural dimension, holds traces of graphic structures comparable to writing, our research attempts to do without the traditional parametric organisation of sign language described by Stokoe (1976) and Cuxac (2000).This way, we wish to explore the unique example of a possible analogy between a language and its writing through this common visuo-gestural modality.

CONTEXTUALISATION AND STATE OF THE ART
Although it would be appropriate to summon the history of writing to help us put the subject into context, it is important to stress the fundamental differences we presented in the introduction and how it is difficult to swap concepts, techniques and forms.
Latin alphabet is the result of a long evolution starting back in the Sumerian period with the cuneiform writing (pictograms).In the course of history, successive assimilation by different civilizations lead to a decisive mutation of writing: phonograms.Whether in the shape of Latin alphabet or Japanese kana, language was broken into syllables, consonants and vowels that a small amount of glyphs could represent.Unfortunately, such simplicity and economy can't apply to sign language.While spoken language holds less than a hundred phonemes, sign language is made of at least 5 gestural parameters which includes movement in 4 dimensions (xyz + time) resulting in hundreds of compounds that can be differentiated at every level.
Chinese writing system on the other hand appears to be closer with its phono-semantic compound nature.Fundamental pictograms form radicals that can be combined with other characters to represent new words while indicating semantic connection.With more than 100 000 characters, it is the oldest continuously used writing system in the world.It is interesting to see how it was able to grow while keeping its consistency using simple rules of construction and graphic semantic.At a formal level, Chinese characters use the notion of body in that sense that each glyph forms an entity that can exist alone or combine with others.It carries both semantic and phonetic information in a selfcontained dimension, a square that faces the reader.Calligraphy is another aspect of the Chinese writing worth considering when researching the connection to gestural visualisation.While calligraphy in Latin alphabet traditionally refers to the strict reproduction of a model, Chinese calligraphy develops a totally different philosophy regarding the act of writing.The supple Chinese brush is supposed to sense every modulation of the body and transfer the movement freely allowing an infinity of variation in the stroke.This connection between the body, the tool and the sign is interesting since it give the gesture authority over the sign and not the contrary.
At the other end of our problematic lies the question of the visualisation of the body.The work of the physiologist J.E. Marey is a milestone in many ways.Its graphic method and the technologies he developed help improve or give birth to physiology, psychology, cinematography, aerodynamics, aviation, ballistic, physical training, art, just to name the more important.Next to Eadweard J. Muybridge (1930Muybridge ( -1904) ) sensational chrono-photography, Marey gave birth to modern observation of the body in movement, refining photographic technology to the limit of the human eye (movement of air) and producing images with the rigorous mind of a dedicated scientist.His desire to picture the various factors of movement simultaneously in a single and legible representation couldn't have a stronger echo in our mind.Visualisation is prior to observation.In our case, it became both perception and action in a dynamic loop binding language and inscription.
At a formal level, the work of Anton Giulio Bragaglia (1890-1960) crystallized our reflections.His technique called photodynamism rejected Marey's analytical methods and focused on capturing the feeling of movement rather than breaking it apart.His pictures of gestures showed a visible trail left by the subject in movement.More than a sequence, this visualisation depicted movement in a indivisible reality and form.
Those references and consideration taken from the field of writing history and visualisation of the movement helps us to develop an original graphic thinking that differs from existing approach to writing systems for FSL.
There is currently several means of representation of FSL, whether suited for pedagogy, analysis or just static depiction, each with its own benefits.
In order to keep a written trace of FLS, annotation systems such as HamNoSys (Prillwitz et al. 1989) or SignWriting (Sutton 1995) have been created, the latter trying to be more visual that the other.In both cases, it is a extremely schematic system that can't properly represent the richness of sign languages.Those systems are very successful when used in narrow, specialized context and confirm the fact that traditional linear writing is not suited for SL (LSScript 2007).For that reason, we will have to research a transcription system able to cover various levels of complexity that are communicated simultaneously.
We focus exclusively on a graphic approach (as opposed to a verbal description) that is sufficient or multilinear communication.
Those systems use more or less arbitrary representations to depict the sign, with strong tendency for geometrisation.Until now, the most popular technique to capture SL in a nondestructive and subjective way is video recording.From those recordings many corpora have been created in order to perform general studies of SL and focusing more specifically on vocabulary or its emergence like in the project Creagest (Balvet et al. 2010).In other working cases, attention was brought on the grammatical structure or a comparison between signs from different countries as in the MARQSPAT project (Blondel et al. 2009).As important as a corpus might be, the question of the form in which it is recorded can greatly determine the type of analysis it will allow.Ours differs from existing corpora (LSScript 2007) by moving away from annotation purpose to concentrate on the visualisation of movements by the trace they leave.For that reason we are interested in the imprinting strategies that we suppose will influence a design build on the concept of a motivated trace through the analogy of signing and writing movements.
We are developing a multidisciplinary approach combining typography, photography, engineering and linguistic inspired by various theories from the field of design and science, from the view of G. Noordzij on the stroke of the pen to the new paradigm in cognitive science opened by enaction theory (Varela 1996).

PHOTOCALLIGRAPHIC VISUALISATION OF THE MOVEMENT
In the first part of our project, we focus on the visualisation of the hand movement parameter and without paying attention to the others (eyes, facial expression…).The team devised a photographic process allowing us to visualize the movement of the hands in space.We call this technique of visualisation "photocalligraphy" in reference to the formalism of our images.The photocalligraphy method focuses on hand movements and turns them into a graphic imprint.This way of representing the gestural dimension of SL envisions the hand as a graphic tool, a kind of living brush.Performative and dynamic parameters as well as the depth of the gestural do not show clearly in the graphic imprint.But this lack of detail forces the LS speaker to work on solutions to compensate this limitation.
Once we make graphic structures visible, we submit them to a test of legibility; that is their capacity to convey the original signification of the sign.This involves agreeing on a representative sample corpus, based on graphical (line dimension and dynamics, axes, symmetries, etc) and linguistic criteria (especially movement-related signs).Our working corpus (photographic) is made out of isolated signs.
We produce two photographic corpuses.The first one involved two deaf persons (a man and a woman).That exploratory corpus aimed at testing the legibility of the graphic traces and had to be elicited.The technical requirements linked to the set up left little freedom to them.However, despite those limitations, we could see a great involvement from the subjects.Motivated by the step back they could take to look at their own language through this process, they showed determination and a high level of expectation, as the numerous repetitions and reappraisals can testify for.As you will see, the results of those primary tests on legibility showed the importance of those decisions.

Figure 2: Difference in legibility between Bruno and Mathilde on a sign (chain)
Indeed, although we didn't count on such analysis, we saw a huge gap in legibility between the two SL speakers on some signs (see fig. 2).This observation helped us to identify what we call strategies of graphic inscription.Under this name, we mean all the techniques of production of the sign used in order to make its graphic representation closer (and more legible) to the mental visualisation the person has of the sign.
This new direction in the project pushed us to devise a second corpus that would motivate the formation of those strategies in order to test their efficiency and to list the possible techniques being used.

A graphic corpus: An exploratory work
It is necessary to insist on the exploratory nature of this work.Combining elicited single signs with graphic inscription represent an unprecedented approach of a language without a writing.
While orality and writing are traditionally separated by their different modality (voco-accoustic and gestuo-visual) in spoken language, SL has the theoretical potential to use a common canal for writing and orality.Eventually, such an experimental approach pushes the limit of how writing can be understood and put the writer in the situation of recalling his gestural language in the act of writing: an analogic graphic transcription of its oral sign.
The graphic dimension that we perceive in orality justifies the double nature of our corpus: collecting images that holds the mark of a gestural/oral execution and a scriptural performance.
Such a procedure confronts the SL speaker, who is an expert in his tongue, to a situation where he has to develop a critical sense over his scripting capacities.We name this double ability: signer/writer

Creating the corpus: choosing the signs and the speakers
In order to choose our elicitation, we made a list using illustrations from a dictionary that is considered as a reference in FSL: the IVT (International Visual Theater) dictionary (Moody et al. 1998).Those illustration are shown to the SL speaker as elicitation As mentioned earlier, our corpus is made of isolated signs (according to IVT segmentation rules) and stand for the variety of forms found in FSL.We choose the signs according to the following aspects: one or two hands, type of symmetry, type of rotation, type of forms, reach of the movements, occultation, repetition, change in the hand configuration, contact or not with a part of the body, depth of the sign, position.In this way, our corpus try not to be exhaustive but to cover, in a smaller scale, the proportions found in the language (Lefebvre-Albaret 2010).
This way of selection helped us gather about 300 signs that eventually became a hundred in order to accelerate the work of the LS speaker.We worked with 8 native speakers, men and women with various profiles.Some, from deaf families, had learn FSL since they were born.Others learned it in High school or even when they reached adulthood.Each of them performs 25 signs (a work session last about 3 to 4 hours).The resulting corpus is made of 200 pictures and 25+ hours of video (used as meta data).

Recording setup and working protocol
In order to isolate the hands and the face, the LS speaker wears a black garment with long sleeves and stands in front of a black background.The camera frames the upper body plus the necessary space to capture large signs, thus capturing the useful signing space.
We use a digital camera able to shoot long exposure images in order to catch the sign in its entire duration.This way, they can show a continuous trace of the hands in movement without the need of extra post-editing.For this corpus, we used a duration of two seconds.
A screen faces the LS speaker and displays with a slight delay the image just taken.This visual feedback makes possible a second look for the LS speaker on his production.It also gives to this process its scriptural dimension.Without being an actual writing tool, the photocalligraphy method and this setup demonstrate the concept that FSL gestuality holds a scriptural dimension.
Sessions are recorded on video in order to save the discussions, comments and the evolution of the behaviour of the SL speaker.Protocol of a session: • explanation of the project, presentation of the different working steps • first elicitation on the 25 signs with a initial live video recording • mastering of the photographic technique (short exercises in order to understand the requirements of the set up) • second elicitation of the same 25 signs with a photocalligraphique captation • playback of all the pictures taken during the session for a validation of the selected images • discussion on the working session The protocol is organized in order to allow the SL speaker to master the set up with a minimum of intrusion from us.Because he can't hear the noise made by the camera shot, we signify it by opening and closing a hand (start and end of the exposure).This is important to synchronize to the duration of the shot and to acquire a feeling of the time available to perform his sign.We don't express any subjective judgment on the quality of the images produced, even when asked by the SL speaker.When they experience doubts, we advise them to think of what they want to see in the image.Then, we can help them with technical advice to come close to their vision.When we feel that he/she develops a graphic strategy, we ask him/her to describe it.
At the end of the session, pictures shot are displayed again and we start a discussion about the question of legibility and the potential offered by the technique/set up.We also watch another series of pictures produced with a different SL speaker and ask him/she to determine their meaning, to pick out the most legible ones, sign by sign, and to explain why.

Graphic inscribing strategies
The specific strategies shown in photocalligraphy are: • various adaptation of the shape, the space or the time with which the sign is accomplished in order to reach a higher level of legibility • projection of the 4 dimensions of reality space (height, width, depth, time) into the two dimensions of photography The LS speaker tries different forms and can see the result on the graphic trail.If a form is considered effective, it is integrated to the strategy.
Because each of them has their own level of expectation, le number of shots may vary.
Next to the tactics used for each sign, more global strategies appear during the session.
As the SL speaker builds up an understanding of the set up and a dexterity at using it, he is able to upgrade his productions by recalling acquired strategies, showing that a learning process occurs.

A depository of collected strategies
Strategies mainly appear in reaction to two technical limitations of the set up: the exposure time (two seconds) and the frontal orientation of the camera lens.

Time related strategies
With an exposure duration superior to a fourth of a second, any moving object produces a motion blur.So the more an object stands still, the more it will appear sharp and bright.On the contrary, the more it is moving, the more it will be blurred and under exposed.This condition forces the LS speaker to adapt their sign in duration by using the following three methods: accentuation, decomposition, and repetition.
Accentuation allows to influence the sharpness and the brightness of portions of the sign.A hand kept still for a moment will produce a clear and sharp configuration on the picture.For example, most SL speakers choose to see the ending configuration of a sign brighter in order to hint at a direction of the movement.Also, in order to make it more legible, the signer/scriptor sometimes decides to break down their sign into key positions.
When a sign uses repetition in the movement, the SL speaker can choose to cancel it in order to avoid graphical overlays (hands or trail).3.5.2.Space related strategies They are developed to answer the question of the remaining three dimensions of the signing space.In order to create a consistent image of the sign within the signing space, the SL speaker can use several techniques: rotation, offset and emphasis.
As a reflex, he puts himself in front of the camera during the shot.People chose, most of the time, promote a face to face position similar to the natural configuration of communication in SL.In the case of movements lined up with the axe of the camera, the loss of information related to depth impairs the legibility of the sign.A line becomes a dot and the entire movement is blurry form.Here, the LS speaker can choose to turn aside in order to re-view the movement in a better angle.(see Figure 4).
Use of emphasis helps to improve the expressivity of a pictured movement by adding volume to the gesture.For example, if a sign traces a curve, it can be exaggerated becoming almost a caricature.

The dilemma of the signeur/scripteur
The gathering of the second corpus shows a disparity in the signs taken with the photocalligraphic technique.Some items are more problematic when their visualisation produces less readable results and that the usual strategies are not able to solve them without causing too much distortion.The dilemma of the signeur/scripteur is the result of a desire to inscribe a mark that would respect the natural shape of the sign and achieve legibility on the final picture.SL speakers themselves are not equals in front of this visualisation process.Some need more time than others to acquire the necessary strategies.However, we noticed that similar strategies (presented earlier) would recur spontaneously among all participants without any direction from us.This would prove that those techniques are a logical answer to the set up and not a singular invention from every person

CONCLUSION
By sharing our corpus and this methodology, we aim at offering a more graphic approach to the question of a script and making it accessible to other teams working on sign languages.This is indeed a multidisciplinary field of study where we hope to prove the importance of exploratory graphic design.In this respect, we believe the first results are encouraging.For the moment, we haven't tested in detail the readability and comprehension of the photocalligraphies, or the relevancy of graphic inscription strategies.The post viewing done by the SL speakers still let us estimate the part of subjectivity and objectivity in the perception of what is readable and therefore made us realize the relativity of the inscription strategies' efficiency.We are using Elan to observe and analyze with more detail the graphic inscription strategies in order to confirm/infirm our previous observations.This software let us annotate several media in a same space (photographs, video recordings and natural signs) traces a curve, it can be exaggerated becoming almost a caricature.

Perspectives
Our corpus will let us test the comprehension and readability of our graphic representation of FSL on better quality images compared to our first corpus (this parameter greatly influences readability, as proved by our first corpus).We will test our hypothesis about the compositionability of signs, and the impact of graphic inscription strategies.The photocalligraphic inscription system shares certain characteristics with writing tools (the gesture leaves a graphic print, the possibility of touching up).we suppose that the rules developed on a medium will be found back on the other.There are numerous scriptural representation of sign languages and some of them use similar strategies.We are interested in learning from the strategies developed through our visualisation technique and apply this knowledge to a writing system.Those strategies would be translated into rules of composition, harmony, balance, etc.
Another exploratory dimension will be the creation of a graphic system/tool that will visualize gestures in real time, allowing a simultaneous perception of the graphic sign and the scriptural performance.Our team works in parallel to develop such equipment.It uses a technology specifically developed by us to visualize hand movements from sign language.While extending the research initiated with the photocalligraphic technique, this project will develop its own specific research topics.We will have to overcome some problems if we want to propose relevant answers to the challenge of graphic visualisation of FSL.For example, how do we choose the meaningful parameters in a special performed gestural sign in order to translate it into a graphical sign that is static and flat?Also, photocalligraphy captures the hand in a uniform way when the tracing dimensions of its portions (fingers, palm, back) differ from one sign to the other.We can as well question the status of those images.Are they a representation of the gesture or a representation of the language?).How do they affect the cognitive model of language?But more fundamentally, the articulation of the signs in between themselves and their position on the page raise the questions of segmentation and enunciation.Those are questions that our research will have to ponder in the future.

Figure 3 :
Figure 3: Picture of the setup

Figure 4 :
Figure 4: shooting Chou -three pictures of the three steps required for the sign [Forest]