Real-Time Mobile Emotional Content Player Using Smartphone Camera-Based PPG Measurement

Abstract

A real-time mobile content player was developed that can recognize and reflect emotions in real time using a smartphone. To determine effective awareness, a photoplethysmogram (PPG), which is a biological signal, was measured to recognize emotional changes in users presented with content intended to induce an emotional response. To avoid the need for a separate sensor to measure the PPG, PPG signals were extracted from the red ( R) values of images acquired by the rear camera of a smartphone. To reflect an emotion, the saturation ( S) and brightness ( V) levels, which are related to the ambience of a content, are changed to reflect the emotional changes of the user within the content itself in real time. Arousal- and relaxation-inducing scenarios were conducted to validate the effectiveness. The sample t-test results show that the average peak-to-peak interval (PPI), which is the time interval between the peaks of PPG signals, was significantly low when viewing the content under the arousal-inducing scenario as compared to when watching regular content, and it was determined that the emotion of the user was led to a state of arousal. Ten university students (five males and five females) participated in the experiment. The users had no cardiac disease and were asked not to drink or smoke before the experiment. The average PPI was significantly higher when the content was viewed in the relaxation-inducing scenario compared to regular content, and it was determined that the emotion of the user was induced to a state of relaxation. The designed emotional content player was confirmed to be an interactive system, in which the video content and user concurrently affect each other through the system.