User Interaction Evolution in the SmartFactory KL

In the SmartFactory KL the intelligent factory of the future, the consortium of companies and research facilities explores intelligent future technologies. Being a development and demonstration center for industrial applications, the SmartFactory KL is arbitrarily modifiable and expandable (flexible), connects components from multiple manufacturers (networked), enables its components to perform context-related tasks autonomously (self-organizing), and emphasizes user-friendliness (user-oriented). This paper presents the outcomes of research in different user interaction methodologies for industrial environments. Today, in industrial environments most devices have their own stationary user interface directly attached to the device. Therefore users have to learn many different user interfaces and interaction devices from different vendors. To climb the ladder of evolution regarding interaction in future factories a mobile universal interaction device has been developed that is capable of communicating with various field devices and plant modules of an industrial facility via common wireless communication standards. A mobile Human Computer Interface like that has one user interface for all purposes, is non-proprietary and can be designed individually dependent on its owner«s needs and desires. Furthermore a first approach on integrating technologies from Augmented Reality (AR) into industrial facilities recognizing natural interaction through an AR-tablet configuration is presented. Eye-tracking, awareness tools, machine learning, coordina tion, expertise Human-Computer-Interaction, Universal Interaction Device, SmartFactory KL , Bluetooth, Augmented Reality, Hand Gesture Recognition, Hand detection, Hand tracking.


INTRODUCTION
Technical staff members in future plants will have to cope with a huge variety of technical devices every day, stemming from multiple vendors and providing entirely different user interfaces.It seems likely to assume that the number and complexity of technical devices in industrial plants and manufacturing facilities will keep growing.With the current lack of explicit and obligatory industry standards harmonizing interaction styles and their look-andfeel workers have to learn how to use each single device.In the SmartFactory KL , the first intelligent factory of the future [1], we investigate if all those devices must be equipped with their own, stationary user interfaces -or not.Already today, mobile control panels and bus systems may allow replacing many stationary displays and proprietary user interfaces with only a few remote interaction devices.A master control station providing collected and aggregated status data as well as means for remote control is common practice in factory automation.Yet, these master control stations are still stationary.A mobile control panel might provide the same quality of data and services to workers as the master control station, but at any location inside the factory.For remotely coupling a mobile panel like that with a certain device, plant module, or a whole process line of a factory, Wireless Local Area Network (WLAN) and Industrial WLAN (IWLAN) are the preferred network based wireless communication technologies.Yet, there are lots of other network communication standards such as Bluetooth and ZigBee but also bidirection standards like Radio Frequency Identification (RFID) and Near Field Communication (NFC) that offer certain advantages depending on the situation [2] -for example, Bluetooth is more suitable than WLAN when numerous connection lines must be kept up in parallel [3] [4].Especially when interacting directly with a product or product component, RFID and NFC are the current state-ofthe-art technologies [5] [6].Consequently, a universal interaction device would need to master multiple wireless communication technologies.The SmartFactory KL achieved the aim to develop a mobile universal interaction device which is able to interact with products and product components as well as devices from different vendors and tools communicating over different types of connections (e.g.Profibus [7]) in an interoperable way.Earlier research used mobile phones to control devices in production environments [8].A universal interaction device was developed in cooperation between the SmartFactory K , the German Research User Interaction Evolution in the SmartFactoryKL Gerrit Meixner, Nils Petersen, Holger Koessling Center for Artificial Intelligence (DFKI) and two other companies.The harmonization of plant control systems has started with central stationary master control stations which allowed centralized control and information query with a consistent front end for all connected machines.With a universal interaction device these devices are now becoming mobile.This enables in situ control while at the same time preserving global plant control.The next step is to improve usability and worker's efficiency while using these systems.A promising technology for this is Augmented Reality.This can provide seamless, less distracting information presentation.Together with hand gesture recognition this offers a better way for more intuitive control and interaction without necessarily requiring any hand-held objects.This paper presents both the outcomes of a research project concerning the mobile universal interaction device and the combination of user specific information with Augmented Reality techniques to head for the next user interaction generations (see Figure 1).While section 2 gives an introduction to the SmartFactory KL , section 3 provides an overview of the universal interaction device for the domain of production automation.Section 4 describes the general idea of the next evolution step for a more natural interaction in the domain of production automation±adapting Augmented Reality techniques into future factories.Section 5 concludes this paper and highlights future goals of the universal interaction device.

THE SMARTFACTORY KL
In the mid 1980s, Computer Integrated Manufacturing (CIM) became one of the most important technological research topics worldwide.During these days, so-called 'CIM Centres' were established to demonstrate new ideas and technologies and received substantial governmental funding.The classical aim of automation was to replace the disturbing factor 'human' with his control and problem solving ability.At the top of this hype, fully automated factories that work without any human interference seemed to be within grasp.Now, almost thirty years later, full automation has not been achieved and probably will never be achieved in the foreseeable future.In the last three decades, however, the complexity of plant structures, processes and control systems has further increased, resulting in inflexible, monolithic production systems, while the market demands for lean, flexible and agile production lines and manufacturing have grown at the same time.It is obvious that we must develop technologies that help us to speed up the planning and setup of factories, which let us adapt plants and processes quickly and allow switching from the production of one product to another on demand.Today, technologies like that do exist and begin to spread.It is time for a new attempt to building smart factories±while keeping in mind the lessons learned during the CIM era.The SmartFactory KL in Kaiserslautern-Siegelbach, Germany, is the first intelligent factory of the future [9], built on two visions of a highly technological future that focus on the human himself.Ubiquitous Computing, introduced by Mark Weiser, predicts that technology will move into the everyday, the small and the invisible [10].Computers and technology will be everywhere, but stay unobtrusive.The second vision, known as Ambient Intelligence, is similarly socially oriented.It envisions the augmentation of nearly all physical items with communication (ad-hoc networking) abilities and a basic 'intelligence', thereby, whole environments become intelligent, easing people's everyday lives by supporting them in natural ways when needed and desired.In both visions, it's not the human that disappears into the background -it is the technology.Several research and living labs for both these visions have been established all around the world in the last ten years [11].There are smart homes, smart classrooms, smart kindergartens, smart offices, future stores, and many more.The Korean government even plans to build 60 so-called ' Ubiquitous Cities (U-Cities) until 2015 [12]!However, one type of living lab was still missing -a smart factory.The first of its kind was the SmartFactory KL , a research, development and demonstration facility established as a private-public-partnership already in 2005.Being a non-profit organization, the technology initiative is open to public and private research facilities as well as industrial companies from local small businesses to global corporations (see Figure 2).The predefined goals of the association are to support the development, application, and distribution of innovative industrial plant technologies in different economic branches, and to provide a basis for their extensive usage in science and practice.The SmartFactory KL is the first vendor-independent demonstration factory for industrial applications of modern and future technologies.
User Interaction Evolution in the SmartFactoryKL Gerrit Meixner, Nils Petersen, Holger Koessling Figure 2. Current partners of the Smart Factory KL  Technology Initiative.
Demonstrating the future of industrial automation, the SmartFactory KL is arbitrarily modifiable, flexible and expandable.It combines components from multiple vendors and enables them to perform context-related tasks autonomously and selforganizing.Last but not least the technology initiative emphasizes user-friendliness.
In this section the SmartFactory KL demonstration plant and its modular design is technically illustrated.Figure 3 shows the modular concept including a continuous flow process and a discrete process part for bulk goods in a 200 square meter industrial facility producing colored soap.
In the continuousflow process a certain quantity of transparent soap has to be pumped in a heating and mixed with colorant pigments according to the customer«s order.All modules are controlled by one single Programmable Logic Controller (PLC), connected to a Supervisory Control and Data Acquisition (SCADA)-System.
In the subsequent discrete process part (bulk goods process) empty bottles have to be inserted on the conveyor belt, filled with colored soap, closed with a dispenser and tagged with a label, containing customer specific information, before they can be commissioned to the next logistical step like packaging and sending.Here the modules communicate in a decentralized way, meaning each of them is controlled by a different PLC without a control station.

Figure3.Floor plan of theSmartFactory KL (top view on modules).
All product relevant lifecycle data, e.g. the amount of bottles, soap color or just customer specific data is stored directly and solely on an RFID label, tagged on each single bottle (see Figure 4).They also determine which steps of the production process have already been passed at a certain time, which ones still have to be taken or which problem has occurred during the production.All these data can be made available to the user of the universal interaction device by scanning the RFID label tagged on a product.

Project Objectives & Goals
The primary goal of the project of the universal interaction device in the SmartFactory K was to develop a prototypical mobile panel for universal access to field devices, plant modules and programmable logic controllers in industrial production environments.Consequently, it must be able to provide the user with access to these devices according to his role, tasks, and goals (user-and task-orientation).Relevant tasks are for example parameterization, control, monitoring, diagnosis and maintenance.A universal interaction device must communicate with devices from various vendors using different communication paths and technologies such as Bluetooth, WLAN, RFID and NFC.Finally, it must provide functions to the user depending on his current location.All these goals aim to support the user in efficiently and effectively fulfilling his tasks.However, in addition to the objective of perfectly supporting the user, 16 important corollary scenarios had been identified in the early analysis phase of the project, resulting in additional use cases and crucial requirements.Especially, a universal interaction User Interaction Evolution in the SmartFactoryKL Gerrit Meixner, Nils Petersen, Holger Koessling device must be accessible to several users belonging to several user groups.
A prototype of a universal interaction device was built by unipo ® GmbH, an expert company in visual man-machine communication, manufacturing process controls, control panels, and industry PCs and ISO-certified developer of software and hardware components for visualization of processes.Figure 5

The UCP450 Prototype
The unipo ® Control Panel 450 (UCP450) as a universal interaction device for industrial environments (see Figure 5) presented in this section employs Bluetooth as the primary wireless communication technology connectable to single field devices and interacting with plant modules and PLCs.Since Bluetooth is not common in industrial environments, however, an adapter or gateway is needed as an intermediary between the UCP450 and its targeted device.

The Communication Gateway
This role is taken over by an intelligent 'Factory Gateway' that possesses both a Bluetooth interface and a universal asynchronous receiver transmitter (UART) for digital serial communication, e.g.via RS-232 and RS-485 (see Figure 6).Additionally, the gateway (see Figure 7) provides TeleTYpe (TTY), Local Area Network (LAN) and Universal Serial Bus (USB) interfaces.Furthermore, WLAN communication is used to interconnect multiple factory gateways to each other.This way, the gateways can communicate with each other and forward requests from the UCP450.Effectively, this enables a UCP450 to access several gateways at the same time, even over long distances via a single Bluetooth connection.This not only reduces the battery usage of the universal interaction device, but also allows for accessing distant devices.Further, it can provide the UCP450's users and applications with internet access.
Finally, the UCP450 will be equipped with an NFC module.In order to help the user to connect to one of the numerous plant devices and modules, he only needs to hold the UCP450 onto an NFC tag representing the respective device.Since these tags can be read out only within a range of up to 9 cm, the gesture of hold the UCP450 onto this tag can be interpreted as a connect command.But of course, the user can also retrieve the user interface for a certain device by navigating through graphical menus or device lists.The gateway itself is enclosed by a compact, robust metal chassis as small as 150 x 85 x 55 mm (see Figure 7).Likewise, the gateway can send settings from any UCP450 to any field device, and thereby control them remotely, provided that the PLC accepts and forwards these settings as the second intermediary in line.

Ad-hoc user interface visualization
Furthermore, even the user interface (UI) specification (see Figure 8) can be uploaded to the UCP450 via Bluetooth.In fact, the UCP450 fulfills all requirements for a run-time generated UI via e.g.user interface models [15].This means, that a PLC or field device can store the UI specification for a single device or a whole plant module.Thereby it can enable the user of a UCP450 to remotely control any device that supplies the specification -even if it is a newly installed device, it does not require any manual installation or set-up on the UCP450.The user might have to wait for a few milliseconds until the UI specification has been transmitted, but in ost cases he won't notice any delay.

Figure 8.Exampleof a user interfacefor remotely controlling
The UI can be adapted according to the user group the user belongs to.Currently, four user groups are being defined: The 'guest' is the only one, who does not need to log in, while every 'operator', 'service technician' or 'administrator' needs to enter a log-in code.Then the UCP450 adapts the UI in a way that each user is provided with appropriate information and interactive widgets according to his tasks and rights.

AUGMENTED REALITY
Another area of ongoing research within the SmartFactory KL is the investigation of improved interaction using Augmented Reality.The term Augmented Reality (AR) denotes the addition of computer generated content into the user's perception of reality.Although the term is not limited to visual perception most AR applications as well as the system presented in this work solely deal with visual augmentations.In this context the term is describing the addition of virtual 2D and 3D imagery in a way that they appear to be fixed on objects in the real scene.Additional hardware is required to display these augmentations in the user's field of view [16][17] for an overview.In principal there exist two techniques for this: Head mounted displays (HMD) and ARtablets.
For using HMDs the user wears video-goggles that display the video feed of a head-worn camera where the augmentations are added.The camera should ideally provide exactly the same view as the user would have looking directly at the scene.
A widespread variant of this is using see-through goggles equipped with a transparent screen to display the augmentations.The camera is hereby only used to track the environment.The advantage of this variant is that the user keeps his natural vision.The disadvantage is that the augmentations will always have a delay with respect to the viewed environment.The problem is design-immanent and can be reduced but not entirely vanquished with faster hardware since every frame has to go through the additional processing pipeline camera, computer vision system, image generation, and display.Due to the delay between augmented imagery and environment resp.other sensory information like the human sense of balance, HMD systems have the tendency to lead to motion sickness after longer periods of use.
The other approach to display the augmentations is to use a tablet display with a camera on its back like e.g. an accordant slate tablet-pc or a cell phone.Again, the camera should approximately have the same field of view as the user would see if the tablet was just an empty frame.This already foretells the way of usage: the user holds the tablet like a picture frame through which he sees the scene behind in User Interaction Evolution in the SmartFactoryKL Gerrit Meixner, Nils Petersen, Holger Koessling an altered way.Although this setup delivers not the same amount of immersion as HMDs do, the benefits are that no calibration step is necessary prior to use, the system can quickly be handed over to other people, and users are not exposed to symptoms like dizziness or motion sickness [16].
In the following sections we will explain how Augmented Reality presentation can improve the efficiency as well as the ergonomics of a worker within an industrial scenario and describe our first implementation of such a system.

Usecases in industrial environment
There are a great number of use cases of AR in industrial environments: For instance AR can enable an improved workflow by offering an in situ interface for information querying.Conventional kinds of presentation like displays on a terminal or manuals on paper require users to actually interrupt their workflow to look at the provided information.With AR that content can be inserted directly in the field of view of the user.Furthermore, since this information is displayed right next to the associated object their relationship can be expressed in an intuitive way for the user.Another use case is a context aware step by step instruction, e.g. in maintenance tasks.Through the camera, the AR-system can identify the current state of a maintenance process and show an instructing animation for the next step directly in the field of view.
A third case is using AR for discrepancy checks between a IDFWRU\ ¶V/ -model and the actual plant.By overlaying the CAD-model perspectively correct on the actual view discrepancies can quickly be identified.
For a more detailed overview on use cases of Augmented Reality applications in industrial environments see [18].

Technical aspects
To identify the object that the system is looking at and to be able to display the augmentations as if fixed on objects of the real scene, the system precisely has to track the camera's viewpoint.
To this end we use optical fiduciary markers (particularly ARToolKit markers [19]) attached to parts in the factory (see Figure 9).The exact camera pose can be deferred from a single image where the marker is visible.The center area of a marker contains a distinctive machine readable code to identify each marker.With the knowledge about a seen marker's global position and orientation in the factory the system can localize itself.While positioning systems like Ubisense provide a location of about 15 cm [20] without any rotational information, the fiducials provide up to subcentimeter precision optically.Another merit is that the precision increases with the camera getting closer to the marker.This can be used to compensate for decreasing precision of factory scale positioning systems like Ubisense that suffer from measurement artifacts near metal objects.So, another beneficial side effect coming from the requirements of an AR-system is that location based services of much higher quality can be offered.This can be used to lead a factory's maintainer directly to the source of an event even if it has the size of a screw.The respective part is hereby highlighted in AR as soon as it comes into view of the user, resp.the AR-System as seen in Figure 9.
Though systems for tracking exist that make use of features naturally occurring in the environment (e.g.[21]) they all suffer from the problem of finding the initial pose.Furthermore these systems are less robust and computationally far more demanding.So for environments where it is feasible to attach markers their usage is recommended.

Image-based interaction using hand gestures
To complement the in situ kind of visualization it would be desirable if the user could also interact in a natural way.The most obvious way to do so is to use bare hand gestures.In this section we will focus gesture based interaction with a common electropneumatic positioner (see Figure 10 and Figure 11) which is a common device in industrial environments.
User Interaction Evolution in the SmartFactoryKL Gerrit Meixner, Nils Petersen, Holger Koessling Figure 10.An electropneumatic positioner in an industrial environment.A pointing gesturecan query an information sheet.
We build upon the hand tracking method described in [22] for hand detection and posture recognition.
The method uses an efficient way to extract scan lines that are likely to contain finger cross-section from the image.Those scan lines are then scored independently on a set of distinctive features of finger appearance.The output of the algorithm is a set of curvilinear segments along the finger centers and the principal axis of the hand, i.e. the direction from the center of the palm towards the finger area.
We chose this approach since it can quickly reinitialize when tracking is lost (which always happens when the hand leaves the camera image), is robust against partial (self-)occlusion of the hand and can be tuned to only track the hand if held in a certain distance from the image sensor.This last property is very important to us since we want to rely on a single camera setup where it is hard to recover reliable depth information, especially about non-rigidly moving objects like hands.The alternative would be to use a setup similar to [23] where the authors use a wide-base stereo camera setup to track the hands in full 3D using the same hand tracking algorithm.To make it less possible that unintentional hand movements in front of the camera are recognized as a gesture we have limited the range of depth where the hand is detected to approximately the distance of an outstretched arm.
We distinguish two different hand postures: a pointing finger and a grabbing gesture.Since the hand tracking step provides us with high level information, the classification is done by simply counting the number of approximately parallel finger segments aligned with the hand's principal axis.This simple heuristic classifies the two detectable hand postures reliably.The user can query information from parts of his environment by pointing on the respective part (see Figure 10).Subsequently, the user can point to the selectable controls indicated in green.When doing a grabbing posture while pointing at a control the view changes and a slider GUI-element is shown (see Figure 11).The knob follows the hand as long as the user is continuing this grabbing posture.Due to the camera tracking the virtual slider is hereby anchored in 3D.
While grabbing, the hand position is asumed to be located at the same depth as the slider's knob.
Through that the AR-system can partly compensate the camera movements for the user to get the impression that the slider is really fixed in space.

Implementation details
For our first implementation we decided for the tablet-pc approach (see Figure 12) since the user's acceptance of head worn hardware is much lower and requires a calibration process prior to use.The downside of this decision is that the main benefit-the possiblity of 'hands-free'-operation -is not offered to the user.For a final implementation we therefore propose to use an HMD to provide this.Nevertheless, for a development rather than evaluation platform, the tablet approach is more feasible.
We use a Sony VAIO UMPC equipped with a U1500 Core Solo processor running at 1.33GHz, a 4.5 inch WSVGA display, and a rear side (low quality) VGA camera.The benefit is that this setup is quite compact while having sufficient calculative performance and all necessary peripheries for a complete tablet-like AR-system.In order to maintain hands-free operation the UMPC would also be small and light enough to be carried at the wrist.While the marker detection is lightweight enough to run directly on the UMPC, the hand tracking is computationally more demanding.Since the UMPC has anyways to be connected to a server to provide the queried data, this connection is being used to outsource parts of the hand tracking algorithm.The UMPC selects all scan lines that are likely to contain a User Interaction Evolution in the SmartFactoryKL Gerrit Meixner, Nils Petersen, Holger Koessling finger cross section in the image, sends those to the server and receives back the respective scores.The UMPC processes the remainder of the algorithm and generates perspectively corrected imagery for the Augmentations.The whole system currently runs at 16 frames per second with a Core 2 Quad 2.5GHz, connected through 802.11g 56Mbit/s Wireless LAN.

RELATED WORK
The number and complexity of technical devices in industrial plants are constantly growing.What is common are thousands of field devices and plant modules in a single facility [1].Since these devices often stem from multiple vendors, they possess user interfaces of different types, different complexity, different look, and different interaction styles.The factory workers need to cope with this diversity, thus having to learn how to use all these different devices.
On the other hand, it is common practice to collect status data from field devices and plant modules, and to remotely control them from a master control station employing a process control system such as Siemens Simatic PCS7 [24].This allows for creating a dedicated, schematic, graphical user interface (GUI) for plant modules or even whole plants, using, for example, Siemens Simatic WinCC [25].
What is missing, however, are mobile interaction devices.The Personal Universal Controller (PUC) [26] approach in the Pebbles project, for example, aims at using handhelds such as mobile phones or PDAs for remote controlling household and office appliances -but not in industrial environments.The GUI is automatically generated by transforming a function model, which must be provided by the manufacturer of the appliance.Until now the ideas of the PUC approach are still not available as real hardware-based universal interaction devices.Furthermore, even today, radio-based communication with process control systems, field devices, and plant modules, e.g. for diagnoses, maintenance, and parameterization, are not common in industrial appliances, although the required wireless communication technologies already exist [2].This is also due to a lack of security associated with wireless communication²while secure communication is, of course, a critical requirement for industrial devices.
Only few wireless interaction devices are available on the market for industrial applications, among them the SIMATIC Mobile Panel 277F IWLAN [27] which employs Industrial Wireless LAN.

SUMMARY AND OUTLOOK
This paper presented the outcomes of research in different user interaction methodologies for industrial environments.Today, in industrial environments most devices have their own stationary user interface directly attached to the device.Therefore users have to learn many different user interfaces and interaction devices from different vendors.To reach the next step of evolution regarding interaction in future factories the UCP450, a mobile universal interaction device, was developed.The UCP450 is the first prototype of a universal interaction device developed by the members of the SmartFactory KL technology initiative.It neatly combines numerous wireless communication technologies, thereby providing access to production lines, technical devices, programmable logic controllers, products and product components.This versatility makes the UCP450 a truly universal interaction device, enabling users to access all kinds of products and devices, no matter what type they are or which vendor they come from.Furthermore a first approach on integrating technologies from Augmented Reality (AR) into industrial facilities was presented.The system recognizes bare hand gestures and shows queried information directly in the user's field of view.
A future main topic will be the evaluation of the universal interaction device in real world scenarios with different users and user groups in comparison with other mobile devices.One further direction of development is to incorporate continuous workflows as described in [23] in the User Interaction Evolution in the SmartFactoryKL Gerrit Meixner, Nils Petersen, Holger Koessling interaction paradigm, see Figure 13.That includes paper-bound extensions of the user -interface using 'intelligent paper' that synchronizes all hand written annotations instantaneously with the instances still anchored at the respective parts in the factory.This would be one step further towards the vision of ubiquitous computing.

ACKNOWLEDGMENTS
The

Figure 4 .
Figure 4. Colored soap with smart RFID label.
shows a picture of the prototype from unipo¨ GmbH.Furthermore the universal interaction device employs Bluetooth technologies from Stollmann Entwicklungs-und Vertriebs-GmbH, an ISO-certified developer of protocol stacks and wireless communication products.The project was led and controlled by the German Research Center for Artificial Intelligence (DFKI) that operates the SmartFactory KL in Kaiserslautern-Siegelbach, Germany.The research and development project started in April 2007, ended in March 2009, and received funding from the German Government (see Acknowledgments).

Figure 5 .
Figure 5. UCP450 prototype with touch screen showing the status of a soap agitator.
Its software runs on Linux and unipo ® ProDiagXE ® [13]; data can be stored and loaded from a micro SD card.A unipo I/O bus ® module (24V, 500mA) allows for connecting up to sixteen input as well as eight output lines.Thereby, signal lights, switches, and external control elements can be attached, for example.Finally, of course, a Bluetooth antenna and a class 1 Bluetooth module are required.The gateway's Bluetooth module is able to keep up several Bluetooth connections at the same time.Therefore, multiple universal interaction devices can connect themselves to the same gateway in parallel.A multi-point interface (MPI) card enables the gateway -and thereby, the universal interaction device(s) -to connect to a Profibus DP network[7][14].Given that a PLC provides respective data blocks, the gateway can receive data about other field devices from the PLC even when being a Profibus slave or when running in multi-master mode.

Figure 7 .
Figure 7 .Rear view of the Factory Gateway.

Figure 9 .
Figure 9. Screenshot from the A R-system with fiduciary markers used for tracking and a visible augmented label at the top of the scene.

Figure 11 .
Figure 11.Making use of hand gesture based interaction.The value of the positioner can be set by dragging the sliderusing horizontal hand movements.

Figure 12 .
Figure 12.Example of using the A R-system.

Figure 13 .
Figure 13.Example of a Continuous Natural User Interface (C NUI) showing information instances in realworld (right) compared to digital counterpart in an AR-view[22].
work in section 3 has been sponsored by the German Federal Ministry of Economics and Technology (BMWi) and supported by the German Federation of Industrial Research Associations (AiF) within the PRO INNO II program (grant number KF0476901WD7).The work in section 4 been supported by the German Federal Ministry of Education and Research (BMBF) within the AVILUSplus project (grant number 01IM08002).