A Survey on Multimedia Quality of Experience Assessment Approaches in Mobile Healthcare Scenarios

Background There are over 165,000 mHealth apps currently available to patients, but few have undergone an external quality review. Furthermore, no standardized review method exists, and little has been done to examine the consistency of the evaluation systems themselves. Objective We sought to determine which measures for evaluating the quality of mHealth apps have the greatest interrater reliability. Methods We identified 22 measures for evaluating the quality of apps from the literature. A panel of 6 reviewers reviewed the top 10 depression apps and 10 smoking cessation apps from the Apple iTunes App Store on these measures. Krippendorff’s alpha was calculated for each of the measures and reported by app category and in aggregate. Results The measure for interactiveness and feedback was found to have the greatest overall interrater reliability (alpha=.69). Presence of password protection (alpha=.65), whether the app was uploaded by a health care agency (alpha=.63), the number of consumer ratings (alpha=.59), and several other measures had moderate interrater reliability (alphas>.5). There was the least agreement over whether apps had errors or performance issues (alpha=.15), stated advertising policies (alpha=.16), and were easy to use (alpha=.18). There were substantial differences in the interrater reliabilities of a number of measures when they were applied to depression versus smoking apps. Conclusions We found wide variation in the interrater reliability of measures used to evaluate apps, and some measures are more robust across categories of apps than others. The measures with the highest degree of interrater reliability tended to be those that involved the least rater discretion. Clinical quality measures such as effectiveness, ease of use, and performance had relatively poor interrater reliability. Subsequent research is needed to determine consistent means for evaluating the performance of apps. Patients and clinicians should consider conducting their own assessments of apps, in conjunction with evaluating information from reviews.

Mobile health apps are health and wellness programs available on mobile devices such as smartphones or tablets. In three systematic assessments published in BMC Medicine, Huckvale and colleagues demonstrate that widely available health apps meant to help patients calculate their appropriate insulin dosage, educate themselves about asthma, or perform other important functions are methodologically weak. Insulin dose calculators lacked user input validation and made inappropriate dose recommendations, with a lack of documentation throughout. Since 2011, asthma apps have become more interactive, but have not improved in quality; peak flow calculators have the same issues as the insulin calculators. A review of the accredited National Health Service Health Apps Library found poor and inconsistent implementation of privacy and security, with 28 % of apps lacking a privacy policy and one even transmitting personally identifying data the policy claimed would be anonymous. Ensuring patient safety might require a new approach, whether that be a consumer education program at one extreme or government regulation at the other. App store owners could ensure transparency of algorithms (whiteboxing), data sharing, and data quality. While a proper balance must be struck between innovation and caution, patient safety must be paramount. Please see related articles: http://dx.doi.org/10.1186/s12916-015-0444-y, http://www.biomedcentral.com/1741-7015/13/106 and http://www.biomedcentral.com/1741-7015/13/58

The first time I heard the term "e-health" I was at the 7th International Congress on Telemedicine and Telecare in London, at the end of November 1999. John Mitchell from Sidney, Australia, spoke about a national government study whose main result was the recognition that "cost-effectiveness of telemedicine and telehealth improves considerably when they are part of an integrated use of telecommunications and information technology in the health sector." [1]. This led to the identification of "e-health" as an umbrella term, with definitions such as "a new term needed to describe the combined use of electronic communication and information technology in the health sector... the use in the health sector of digital data - transmitted, stored and retrieved electronically - for clinical, educational and administrative purposes, both at the local site and at distance" [2]. In this talk, e-health was introduced as the death of telemedicine, because - in the context of a broad availability of medical information systems that can interconnect and communicate - telemedicine will no longer exist as a specific field. The same could also be said for any other traditional field in medical informatics, including information systems and electronic patient records. e-health presents itself as a common name for all such technological fields. Mitchell also pointed out that "e-health can be considered to be the health industry's equivalent of e-commerce," and this could be one key for understanding the sense of e-health: just medical informatics and telematics on the shop shelves, a fashionable name for something already existing but otherwise difficult to sell. Without arguing anything about the consequentiality of the facts, in December 1999 the subtitle of Telemedicine Today- a non-peer-reviewed journal - changed from "Where healthcare + telecommunications converge" to "The eHealth Newsmagazine," and just some months later, even the Telemedicine Journal- a scientific, peer-reviewed journal - added an "and eHealth" to its title. Nice name? Fear of being left out of a possibly-new field? The Ace Allen editorial that introduces the change in the subtitle of Telemedicine Today [3] sounds slightly bitter: during the time the telemedicine market exploded, Telemedicine Today's name suddenly changed, perhaps to satisfy the hundreds of healthcare-related dotcoms looking for a buzzword. Shortly after the above changes, E. Rosen [4] rationally explained some differences related to the use of the words "telemedicine" and "e-health." Investors look for investments that can produce high returns even after several years. From this point of view, the specific term of telemedicine seems inadequate, as it identifies a market niche, while e-health, as any "e-thing", seems more open and promising (just like anything without a clear meaning). Rosen also points out the hardware-centric aspects of telemedicine, which is based on the traditional equipment sales model, while e-health is apparently oriented to service delivery, which is more interesting on the business side. The final remark of Rosen is that almost all "e-things" will again become simply things as soon as we become acquainted with the novelty of the Internet; after all, we all know that e-commerce is just commerce... Allen, in a further editorial [5], discovered a new difference: telemedicine remains linked to medical professionals, while e-health is driven by non-professionals, namely patients (or, in the e-health jargon, consumers) that with their interests drive new services even in the healthcare field-mostly for their empowerment through access to information and knowledge . Interestingly enough, even after the name change, Telemedicine Journal and eHealth did not publish any paper directly mentioning e-health; also, the other major scientific journal related to telemedicine, the Journal of Telemedicine and Telecare, seems not to care much about e-health. This could be related to the business role the term e-health seems to have: when researchers describe their work, the classical categories of, for example, Medical Informatics, Telemedicine, and Electronic Patient Records are more meaningful than the generic term e-health. As a researcher, I can see some sense in the term e-health; coming from the integration perspective it suggests: integrated-healthcare-systems' properties, possibilities, and consequences that are (in a holistic approach) more than the sum of the single-component outcomes. However, even these aspects are already studied in some computer science fields - for example Artificial Intelligence (at least inside the multi-agent paradigm), Information Economics, and Dynamic Systems; thus, there is nothing new again, except for the specific interest in healthcare. Vincenzo Della Mea Institute of Pathology, University of Udine, Italy