Touchscreen User Interface Design for Content Based Image Retrieval

Speed is one of the key demands while performing image from background extraction through automatic shape search and dealing with images on highly textured paper. The CBIR-to-art-history application stimulates development of specialised tools and procedures in this field (Otal et al. 2008, Automatic Watermark Detection Tool for the Bernstein Project 2008). Some of these are distinguished by great precision, while being time consuming and demanding a higher grade software, while others run faster at the cost of precision (Jaworska 2007, Liu et al. 2007). Both semi-automatic and fully automatised methods are applicable.


INTRODUCTION
Speed is one of the key demands while performing image from background extraction through automatic shape search and dealing with images on highly textured paper.The CBIR-to-art-history application stimulates development of specialised tools and procedures in this field (Otal et al. 2008, Automatic Watermark Detection Tool for the Bernstein Project 2008).Some of these are distinguished by great precision, while being time consuming and demanding a higher grade software, while others run faster at the cost of precision (Jaworska 2007, Liu et al. 2007).Both semi-automatic and fully automatised methods are applicable.
The majority of presently used systems have a limited capacity to engage users in defining the area of specific interest in visual retrieval.Interactive options are usually limited to image loading and/or edge cropping (see Oxford Ballads Online visual search or BSB Bildsuche).Query-bysketch, which used to be one of the most common interactive strategies, formulated in the very early days of the CBIR (Di Sciascio et al. 1999) has now become marginalised.This is due to technological advances as searching for an image can be initiated with an easily obtainable photo or a scan.
The recent proliferation of touchscreen devices has generated new user habits (Cichoń et al. 2013).It also creates an opportunity to redesign some of the visual search strategies.As the display screen becomes a working surface, we gain freedom to find the object of interest in the most natural and simplest of ways.Interfaces designed for such touchscreen, human-computer interaction should be designed to facilitate image segmentation and labelling techniques.Their procedures should be flexible enough to combine the qualities of queryby-sketch and query-by-image.The interface we present here is designed to use a basic set of hand movements to replace costly and inflexible automation techniques with more flexible and userfriendly methods.

THE INTERFACE
JoAnka interface is built with HTML5 and React.jslibrary components.The core processing concerning shape description and similarity measure is performed in Matlab.Elements of the UI are limited to: load button, dialogue box, highlighter tool, reset and search functions.The highlighter's size is adjusted automatically to the final format of the image and its diameter equals 1/3 of the shortest image edge length.The adjustment function should stimulate the user to highlight the image with one swift touch that would cover the area of interest instead of outlining it (see Figure 1).
While building the interface we aimed to minimise the time dedicated to image extraction by designing a flexible, one-touch, no-words image editing procedure, suitable for displays of all sizes.Details of the editor's default settings have been designed based on a focus group research survey involving group of seven users, with an age range between 9 and 56 years old, both with (3) and without (4) visual communication/art history background.The framework has been inspired by the conclusions of a user experience survey conducted in autumn 2016.This involved two groups of digital natives, undergraduate students aged 19 to 22, who had basic experience either in computer science (37 individuals) or culture studies (23 individuals).

CONCLUSIONS
The interface can be applied to a range of visual search purposes including shape and multi element object labelling in historical prints and drawings.Some limitations of the method have been observed and these are: the one-touch method may appear inconvenient for users with low hand movement capacity.Accidental omissions while highlighting the desired shape are 0.18 instances in the test set (see Figure 1b and 2k).This can be minimised by the introduction of morphological dilation filtering which would sustain the continuity of the figure outline.

Figure 1 :
Figure 1: The highlighter's scaling to the image allows the user to cover the area of interest instead of having to outline it.See two test Images marked by the same user each a) b) and c) d) while differed sizes of markers provided.

Figure 2 :
Figure 2: Examples of the test images while in the following stages of the interaction: original a) b) c); edited d) e) f); subtracted g) h) i) thresholded j) l) k) The preprocessing described have guaranteed complete shape extraction in 0.18 instances only.It should be enforced with further image filtering.However it has proved to eliminate 0.26 more artefacts than the average cropping operation.As part of further development it should be considered giving the user an option to control the threshold level while the subtracted image is being thresholded.Marking a shape with touch within has appeared to be faster than cropping interaction (6 seconds per image vs 8/9 sec.).The test was performed on the set of 260 examples of historical watermarks and collector's marks, 0.25 are photo, 0.75 sketch documentation.