Davide Mazza Haptic and Visual Rendering for Multi-Modal Exploration of Molecular Information

The paper presents a system for the multi-modal rendering of molecules. Chemists typically deal with phenomena that are not directly experienceable and usually described by huge amount of data awkward to understand directly. Software tools have been introduced to support data interpretation, without a high degree of interactivity and explorability of the data. This work goes beyond the classical visual representations of molecules and introduces some new ways of exploration and navigation of information on which inter-molecular interactions are based. Target users are researchers and teachers, as integration to their activities. Two different modalities are used: haptical and visual. The former modality consists in the addition of a haptic interface that enables users to feel the interaction forces exerted by the electric field around a molecule. The latter modality shows the features of the same electric field by allowing the user to navigate the information about the values of the field using a visual color-based rendering and adding other visual cues. Implementation details, case studies, and results on the two modalities are described. Eye-tracking, awareness tools, machine learning, coordina tion, expertise


INTRODUCTION
Molecular analysis has acquired a great importance in recent years, in terms of study of the interactions among molecules and the ways in which they can be aggregated to obtain new chemical compounds with desired properties.This is at the basis of the field called supramolecular chemistry [19].In this context it is fundamental to obtain a deep understanding of the inter-molecular forces that govern this kind of processes.The field of haptics, that deals with sensing and manipulation of virtual objects through touch and force feedbacks [11], can greatly help in this field, by providing the sensation of the forces involved in such phenomena and by experiencing many related properties.This paper describes a framework for the haptical and visual rendering of molecules which goes beyond the classical tools for representation of chemical compounds and provides a way to explore and navigate the same information more intuitively.An analyzed molecule is presented to the user and rendered in a graphical way: the electric field that surrounds the molecule can be explored, to understand the interactions of the molecule with the environment.Along with this representation, it is possible to interact with the molecule through a haptic device to feel the forces generated by the electric field.The employed haptic device is the Sensable PHANTOM Omni [7], widely used in other works presented in literature.
The framework has been developed with two target users in mind: researchers, who can employ the tool as a support for their scientific activities, and chemistry course teachers, who can make students better understand a set of selected topics with a learning-by-doing approach.The paper is organized as follows: Section 2 reports the state of the art of the field; Section 3 introduces the visualizations currently used in the chemistry field, to highlight the improvement given with this work; Section 4 shows the architecture of the design system and provides some details on the implementation of the data computation process; Sections 5 and 6 show how haptic and visual interactions take place; Section 7 shows the results in terms of system performance and users' feedback, while Section 8 draws some conclusions and future work.

RELATED WORK
VMD [15] is probably the most used program in the bioinformatics field due to its high customization possibilities and features.It is a molecular visualization program for displaying and analyzing large biomolecular systems using 3D graphics and built-in scripting.Haptic interaction is supported with strong limitations in VMD, due to the scarce use of this technology, not so wide-spread at the time of its development.Explorations of data is also not possible as explained in this article, but just rely on showing numerical values upon user requests.Beside VMD, there are other tools, specifically designed for protein visualization, such as PyMol [4], MoilView [9], MD Display [14], and Chimera [10].These tools offer a wide range of representations, some of them geometrically-based, others based on the calculations to determine the surfaces of interests (e.g.potential isosurfaces).For some representations, it is possible to have the option of exploring the molecular surface haptically, but haptic support is limited to feel attraction or repulsion, while our system gives also the magnitude of the interaction.A prototype Molecular Visualiser (MV) application, based on Web3D standards and extended to support haptic interaction, was developed by Davies et al. [13].To combine haptics with 3D molecular visualization, a Reachin Display [5] has been used.This tool is targeted to students as well as to researchers, but suffers from performance bottlenecks for large molecules.[18] implements a system termed Interactive Molecular Dynamics (IMD) that supports the manipulation of molecules in molecular dynamics simulations with real-time haptic force feedbacks and a graphical display.Compared to our work, it uses a more sophisticated hardware configuration (not a commercial haptic device like in our case) and its focus is more on performances issues rather than on the interaction modalities.Considering the proposals in the field of technology-enhanced learning, [16] shows new ways for displaying features of complex molecules to students, that cannot be rendered exhaustively with the typical molecule representations; in particular, it proposes a collaborative project where multi-modality enhancements of tangible molecule models are obtained by superimposing graphical information on top of these models.Anyway, the use of such physical entities makes the work difficult to reproduce and scale easily, which is instead possible with a completely-virtual simulation such as the one proposed in this paper.[17] present an e-learning system that incorporates a multimodal haptic simulator.The haptic simulator facilitates student understanding of difficult concepts (e.g.physics concepts) and has the potential to augment or replace traditional laboratory instruction with an interactive interface offering enhanced motivation, retention and intellectual stimulation.The work is target on simplest physical principles and is more suitable for secondary school students, while ours is deeply theoretically-modeled and targets a more technical and high-skilled audience.

MOLECULAR VISUALIZATIONS
Molecules can be described in different ways, according to the properties one is interested in.For example, a molecule can be described at a first instance by the absolute positions of the atoms in the space in which the molecule is placed, and the inter-distances between the same atoms.This is a geometrical description of a molecule, and even if it could be the more immediate, it is not the only one.Over the years a series of properties have been defined and associated to molecules, e.g., referring to dimensions being measured empirically or to other properties defined theoretically to explain the reaction of the molecule in a particular context (e.g., electronegativity).All these descriptions are represented as series of numerical data that define the values of the property in the surrounding space or on a particular area of the molecular surface.This huge quantity of information is hard to interpret as is also by experts of the field, who usually rely on specific software tools to better understand the same information.Plenty of ways of representation of molecules have been developed to represent synthetically the many features that can be associated to a molecule.These representations could be divided in two sections: structural representations and properties representations.The former render a molecule paying attention to its geometry and space occupation, in terms of nature of atoms (i.e., which chemical element is composing the molecule), positions of the same atoms, distance among them, existence and kind of bindings between each pair of atoms, etc.; the latter focus on the properties of a molecule, like, e.g., the values of the electric field in the space surrounding the molecule.For a survey on molecular representations, see [6].Figures 1 a) and b) show the ball-and-stick and space-fill representations, two of the most used structural renderings.The ball-and-stick representation depicts atoms as spheres of different radius according to the atomic number of the chemical elements, while ligands are represented as small-radius cylinders connecting spheres.This representation gives an idea of the connections among the atoms composing the molecule.The space-fill representation shows each atom as a sphere with different radius (called van der Waals radius)and each sphere occupies the volume where the electrons moving around each atom can be found.It gives thus an intuitive representation of the extension of the electrostatic interactions of the molecule.These two visualizations are built up on the basis of the geometrical information of a molecule, i.e., by considering the positions of atoms and their sizes.Typically the former information (position) is provided as a set of coordinates of atoms center in a given reference system; the latter (size of each atom) is derived by considering the atomic number (the greater the number, the bigger the atom radius).Other representations instead are based on the values of some properties of the molecule, and try to reproduce this information visually.Typically, the rendering of properties consists in coloring the surface of a molecule on the basis of the values of the property to represent.Values of the property are usually divided in subranges of values, and each sub-range is associated with a color.Figure 1 c) shows the representation of the molecular electrostatic potential (MEP), i.e., of the electric field around the molecule.Two different colors are associated with the positive and negative values of the electric field around the molecule, in order to highlight their polarity.These properties representations are especially useful when they are described by great amount of information, which need to be structured in some way and then represented to be interpreted.The electric shell around a molecule, for example, cannot be described precisely by means of analytical expressions, but requires computational tools, which usually return a huge amount of numerical values representing the magnitude of the electric field in the space around the molecule.These data are not easy and direct to understand, therefore some representations have been introduced.The representation exemplified in Figure 1 c) is static and depicts a state of the molecule only on its surfaces.However, the property could have values in other points of the space (e.g., the electric field has significant values in the whole space around the molecule, not only on the molecular surface).In this work, we extend such a representation by providing different modalities to browse and navigate the data of the electric field of a molecule, in the whole space around the molecule and considering its interaction with an external electric charge.In particular, to browse the information we employed two possible modalities of navigation: • Haptic navigation of the forces due to the interaction generated by the electric field around the molecule.
• Visual navigation of the values of the electric field in the space around the molecule.
These two modalities will be described in Sections 5 and

4. THE ARCHITECTURE OF THE SYSTEM
The architecture of our framework is sketched in Figure 2 and includes: • The user interface, composed of a haptic subsystem that allows the user to feel the simulated interaction forces involved in the real interactions, and a visual subsystem that shows to the user the information about the electric field of the molecule and allows him to browse such information graphically.
• A repository of molecules, containing the data needed for the visual and haptic rendering of the geometrical information and of the electrostatic field.
• A process to obtain the needed data from computational chemistry tools.

Process for the computation of the force
The right-hand side of Figure 2, describes how the electrostatic data of a molecule can be obtained.The input for their computation is the structural information of the molecule: we have chosen the chemical PDB format [3], storing data obtained from a crystallographic analysis.This format can be manipulated by OpenBabel [1], an open-source, chemical library for reading/writing the many file formats available in the chemical field.OpenBabel provides also the software objects to correctly manage the structural information for molecular visualization.The electrostatic field data can be computed starting from the geometrical information of the molecule: computational chemistry tools like GAMESS [12] and wxMacMolPlt  In order to obtain the value of the electric field, each voxel's MEP value must be spatially derived.From physics we have indeed: where V (x, y, z) is the MEP value in a particular position (x, y, z) of the virtual space and F(x, y, z) is the force vector felt by a unitary positive charge.
In order to compute the gradient starting from the numerical values of the MEP, we need to calculate the derivatives using approximation as finite differences, computed along each direction as follows: For the sake of simplicity, we have adopted a syntectic notation where V has to be intended as the value of the potential associated with the current voxel (V (x, y, z)) and the different Vindex refer to adjacent voxels, where index represents the difference w.r.t. the considered direction (e.g., Vx•1 = V (x • 1,y, z)).
Along the border of the spatial grid we assume that the points outside the grid have a null MEP value, which means that electrostatic field interactions are not felt so far from the molecule.This is a realistic assumption because electrostatic field is felt till a certain distance from the molecule.In our simulation the threshold is determined by the grid dimensions (number of voxels along each axis), considering the geometrical center of the molecule as the center of the grid.Gradient computation is a quite time-consuming task.In order to preserve real-time constraints that guarantee user interactiveness, it is pre-computed and saved in the repository where force values are retrieved when the haptic proxy enters the particular voxel.

HAPTIC INTERACTION
The haptic interaction defines the way in which the user interacts with the system and, in particular, how force feedbacks are rendered.To get the user feel the forces of interaction of the electric field, a reference object that interacts with the molecule must be considered, because electric interactions can be described only in terms of a couple of interacting objects.For this reason we consider a unitary positive electric charge as reference object, associated to the probe of the haptic device (in our case a Sensable PHANTOM Omni [7]).The user can move the charge in the virtual space via the haptic device.The value of force exerted on the electric charge in its current position is the one felt by the user through the haptic device.The intensity of forces are very small in scale (forces in nanoscale worlds are of the order of nano-newtons) and therefore not felt by the user with the haptic tool.For this reason we need to amplify the force values to be rendered.Amplification is done by considering the range of forces exertable by PHANTOM (MAX neg<Fexertable <MAX pos, where MAX pos is the maximum positive force and MAX neg is the maximum negative force), and by proportionally scaling over this interval the force values obtained from the computational process.If we call Fmin(x, y, z) and Fmax(x, y, z) the minimum and maximum forces of the electric field of a molecule returned by the computational process, i.e., for all the force values F (x, y, z) • [Fmin(x, y, z),Fmax(x, y, z)], then, we can define two constants of proportionality Kpos and Kneg to amplify positive and negative forces, to obtain forces that exploit the whole interval of exertable forces, as follows: The actually rendered force will be:

The discontinuity problem
The matrix of values associated with each voxel can present differences of the values magnitude for adjacent voxels that could even be great, leading to jumps and discontinuities in the electric field.Such discontinuities are due to approximations introduced by the quantization of grid of MEP values and are not present in nature, where molecular electric field is continuous and does not present singularities.This could produce bumps and instabilities in the force feedback to the user.In particular, discontinuities due to the sudden change of the sign of the values of adjacent voxels (from positive to negative values, or vice versa) make the haptic feedback unrealistic and the probe highly instable and difficult to handle, causing it to strongly vibrate around the discontinuity.Moreover, these discontinuity points happen in particular spots of the electrostatic surface, such as in its maxima or in equilibria, which are worth great interest in molecular exploration and knowledge (e.g., equilibria are the points where usually a binding with another atom or molecule takes place).Figure 5 a) shows an example of a grid with such a discontinuity in its values, where the force direction (sign) suddenly changes from one to the opposite between adjacent voxels.The problem has been solved by modifying the way in which the force feedback is generated (amplification, mentioned before).The force is no more computed just based on the value of the gradient for the current position, but the force exerted on the device is smoothly changed during the passage between two different haptic frames.In this way, the passage between the last and the current force feedback is made continuous.Every component of the force vector applied to the device for each frame is computed as: where Ct•1 represents the component of the force applied in the previous frame, Ct the force component to be actually applied in the current position (i.e., the final value to reach at the end of transition), and C is the force component applied in the current haptic frame.S is a transition constant that determines how fast the final value Ct is reached.

VISUAL INTERACTION
The visual interaction treats those aspects related to the rendering of the environment in which the user operates.It is important that the user does not feel lost in moving in the environment and gets some hints to better understand what he is doing during his interaction.The representation must not just be correct but it also needs to be meaningful, i.e., through the graphical rendering the user must be able to obtain information about the electric field of the molecule and about the magnitude of the interactions.
The main object of the system is the molecule, represented as a set of 3D spherical surfaces.
The sphere is depicted pixel-by-pixel and each point is colored according to the value of the electrostatic potential in that position.The potential is segmented from a minimum (maximum negative) to a maximum (max positive) value into a given number of intervals and a color is associated with each interval.The haptic force sensation is represented also visually, as shown in Figure 7: a graphical vector shows the intensity and direction of the force felt in the current point: intensity is represented by the length of the vector, while the direction shown by the arrow indicates if there is an attraction or repulsion force towards/from the molecule.This allows a further multi-modal interaction and conveys the same information in different ways.

RESULTS
The tool has been applied on different molecular interaction scenarios in order to determine if typical electrostatic features of well-known molecules can be recognized and determined also with the simulation supported by the tool.Indeed, with such a tool it is possible to understand which part of a molecule takes part in a binding, how bindings happen and which are the best candidate sites on the molecular surface, or how the molecule must be oriented to make the binding happen, etc.At this aim the following electric field explorations have been experimented: • Polar and non-polar molecules have been analyzed.In the former case molecule makes its attraction/repulsion effect sensible in all the environment; in the latter the effect on an electric charge at a high distance is negligible, thus defining an area of influence of the molecule.Moreover, non-polar molecules can be polarized with an external positive/negative charge, thus changing the intensity of attraction/repulsion force of the charge.
• Critical points in the electrostatic potential surface in the space around a molecule have been explored (e.g., to find minima, maxima values or "holes" that give stability to the charge position when reached).This experience is particularly interesting with molecules having a closed symmetric geometrical structure, such as the benzene (C6H6) molecule, shown in Figure 8.In this case, holes can be formed inside the molecule, thus trapping the charge, as shown for the negative electric charge in the right part of the Figure.This mechanism is typically used for the docking technique when designing new drugs.
• Molecule showing anisotropy have been considered.Molecular anisotropy represents the difference in interaction of a molecule with respect to the direction of approaching.
Indeed, a molecule exerts different influences (in intensity and sign of forces) depending on the direction from where the charge comes In all these cases some aspects can be explored simply by looking at the color of the surfaces or by navigating the electrostatic data around the molecule (e.g., the area of influence, minima and maxima values etc.); others require the exploration considering an external charge and its interaction with the molecule.The former can be rendered visually on the (possibly enlarged) atom surfaces; the latter are rendered visually on the proxy representing the charge.From an haptic point of view, in all the cases force feedbacks are returned during the exploration.

Users feedback
The tool has been tested on different molecules taken from the repository (e.g., water, ammonia, or also complex such as DNA parts) by two kinds of users: students and chemistry technicians/researchers.The former experienced the interaction as a didactic tool; chemical technicians and researchers appreciated the improved awareness of the phenomena 1 they are used to work with, which are otherwise not so intuitive, and tested how the tool is able to render well-known interactions.According to the users, the first experiment can be clearly understand with both modalities; the second one is more understandable haptically, while for the last case study, the direction and intensity of force rendered visually better explain the anisotropy of the molecule.Both modalities are therefore useful, depending on the phenomena or the particular aspect to render.For molecules of great dimensions, the users claimed the lack of possibility of having a first coarse exploration of the electrostatic field on the global molecule, and then to further refine the felt forces on selected areas of interest of the molecule, where a more detailed and precise haptic response should be provided.This drawback could be solved by employing multi-resolution grids, planned as an extension of the tool (see Section 8).

Performances
The developed tool has been tested to obtain some indications about its performances.In particular, we measured the rate at which the simulation runs both from the visual and haptic points of view.We have been interested in preserving the realtime feature of the user interaction, both from haptic (force output frequency) and visual (visualization update frequency) point of view.Figure 9 shows a table where two parameters strictly important for real-timeness preservation have been 1 Notice that the developed framework is based on data retrieved with the help of a computational chemistry and therefore on data that simulate the behavior of interactions in a very realistic way.This is not just another implementation of the interaction between a charge and a molecule modeled from a theoretical point of view, as can be found in many applets all around the Web or in other similar applications.
measured.We have compared the complexity of a molecule in terms of atoms and bonds, and obtained a complexity index simply given by the sum of these two quantities.We have then measured the Graphical Frames per second (GFPS) and the Haptical Frames per second (HFPS) at which the tool can run, by applying it on different molecules on increasing complexity.Remembering that human operation can see discontinuities in the interaction if GFPS is lower than 25-30 fps2 and HFPS is lower than 1000 fps [2], the performance exhibited by tool is sufficient to preserve the real-time behavior we needed, as shown by the table in Figure 9, where all the values of GFPS and HFPS are over the previously-mentioned limits.
Figure 10 shows that also for complex molecules the graphical fps value does not go under 30 fps, which is enough for realtime rendering.

CONCLUSIONS AND FUTURE WORK
In this work we have developed a system for the study of the molecular interactions that exploits multimodal ways for conveying information to users.The modalities of interactions are haptical and visual, and allow the user to surf the data about the electric field around a molecule in a more immersive and involving way.In particular, we have shown how the data on which the interactions are based have been obtained through computational chemistry tools to get realistic interactions.Then, we have detailed how the forces felt by the user through the haptic device have been rendered and how data can be browsed in the virtual environment where the molecule is placed.In particular, the electric field data can be explored in a visual way, by introducing a fictitious structural modification of molecular geometry to reach the zone of the space around and inside the molecule and by adding visual information about the forces on the proxy rendering the charge.The rendering of the forces in two different modalities (haptic and visual) conveys information to users in an effective way: as reported by our testers, the two modalities make them more aware of what they are experiencing.
A little drawback of the current graphical representation of the environment is the difficulty for the user to get the sensation of depth, i.e., to get the impression of the z-coordinate of the current position.
The PHANTOM allows to move in 3D real space (having 3-dof) but it is difficult for the user to get this depth sensation by looking at the screen when moving the haptic probe.A solution for this could be a representation of the virtual environment that provides some auxiliary virtual objects (e.g., walls) where fictitious spatial properties of the molecule can be rendered (e.g., its shadow).
As future work we plan to investigate also moleculemolecule interactions, as already present in some tools in literature.Anyway we will keep the possibility of the exploration of the electrostatic surface of a molecule, because this is a feature not exploited in previous works and it can be very useful for studying molecular interactions.Finally, we will introduce a multi-resolution grid, which would be particularly useful in case of complex molecules, where the user may start with a "coarse" exploration of the electrostatic surface, and when zooming on selected segments of the molecule, the interaction may be refined using a finer grid resolution in order to obtain a more detailed and precise haptic response.

Figure 1 :
Figure 1: Different representations of the same molecule.

Figure 2 :
Figure 2: The architecture of the designed system.

Figure 3
Figure 3 shows a snapshot of our tool: the central part of the interface shows the graphical rendering of the molecule; the bottom part contains some setting commands to move/zoom the molecule; on the right-hand side the user can open a new molecule, choose among different molecule representations, and set some parameters for the visualization and haptic rendering of the electrostatic field data.

Figure 3 :
Figure 3: A screenshot from our tool.
[8]  can provide in output the electric field values of the molecule that are needed to model the force interaction.The use of these tools is necessary because the data related to the electric field around a molecule can only be analytically derived using quantum physics computations.In input these tools require data about the structural (atoms' center coordinates) and chemical (atoms' element type, like oxygen, hydrogen, etc.) characteristics of the molecule; in output they return the Molecular Electrostatic Potential (MEP), organized as a threedimensional grid G[Nx][Ny][Nz] of small-dimension cubes (voxels), where Nx,Ny,Nz represent the number of voxels for each spatial direction and each g[i][j][k] (i =1..Nx, j =1..Ny, k =1..Nz)isthe magnitude of the MEP in the voxel.For example, the result for the H2O molecule, with a grid of dimensions 42x65x42, will look like:

Figure 4
Figure 4 gives the idea of subdivision of the space around the molecule into a grid of voxels.

Figure 4 :
Figure 4: The idea of a grid surrounding a molecule.
Figure 5 b) tries to give an idea of the smooth transition of the force values by passing from the vector force on left-hand side to the one on the right-hand side.

Figure 5 :
Figure 5: Solving discontinuity problems by smoothing the force feedback.

Figure 6 a
) shows an example of this initial representation applied to a water molecule.It is colored according to the polarity (sign) of the field and the intensity of the color is proportional to the magnitude.In particular, the red colors highlight the negative part of the molecule, while blue colors indicate the positive parts; green colors indicates the near-zero values.Starting from this initial representation, the tool gives the user the possibility to explore the electric field around the molecule.The user can select an atom and then enlarge / reduce its radius.The associated sphere traverses the electric field and becomes colored accordingly.Even if it could seem unrealistic, because of the arbitrary modification of sizes and proportions of atoms' dimensions, this modality of interaction allows to surf the field around the molecule in a graphical, visual way.

Figure 6 :
Figure 6: (a) the original dimension of a molecule of water (b) the center atom enlarged to explore the electric field around the molecule.

Figure 6 b
Figure 6 b) shows the same molecule of water with enlarged atoms to browse the electric field values.Compared to Figure 6 a), the user can get the impression of how the electric field changes around the molecule space.

Figure 7 :
Figure 7: A graphical vector represents felt forces in a visual way.
Both modalities of interaction can be applied on the different use cases: a) for the determination of molecular polarity, the force vector represented visually and rendered haptically can help in understanding where molecule poles are placed; b) in critical points detection haptic and visual rendering of forces can help the user feeling the stability of the point (with the equilibrium of attraction and repulsion forces in such points) and visualizing the (almost) null intensity of forces in electrostatic holes; c) finally, in testing anisotropy, the visual and haptic representations of the force can be used to show and feel the different intensities of force in the explored point of the space.

Figure 9 :
Figure 9: Visual and haptic performance of the developed tool.

Figure 10 :
Figure 10: A plot of the performance of the tool.