+1 Recommend
1 collections
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Measuring implementation in global mental health: validation of a pragmatic implementation science measure in eastern Ukraine using an experimental vignette design


      Read this article at

          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.



          There is mounting evidence supporting the effectiveness of task-shifted mental health interventions in low- and middle-income countries (LMIC). However, there has been limited systematic scale-up or sustainability of these programs, indicating a need to study implementation. One barrier to progress is a lack of locally relevant and valid implementation measures. We adapted an existing brief dissemination and implementation (D&I) measure which includes scales for acceptability, appropriateness, feasibility and accessibility for local use and studied its validity and reliability among a sample of consumers in Ukraine.


          Local qualitative data informed adaptation of the measure and development of vignettes to test the reliability and validity. Participants were veterans and internally displaced persons (IDPs) recruited as part of a separate validity study of adapted mental health instruments. We examined internal consistency reliability, test-retest reliability, and construct and criterion validity for each scale on the measure. We randomly assigned half the participants to respond to a vignette depicting existing local psychiatric services which we knew were not well regarded, while the other half was randomized to a vignette describing a potentially more well-implemented mental health service. Criterion validity was assessed by comparing scores on each scale by vignette and by overall summary ratings of the programs described in the vignettes.


          N = 169 participated in the qualitative study and N = 153 participated in the validity study. Qualitative findings suggested the addition of several items to the measure and indicated the importance of addressing professionalism/competency of providers in both the scales and the vignettes. Internal consistency reliabilities ranged from α = 0.85 for feasibility to α = 0.91 for appropriateness. Test-rest reliabilities were acceptable to good for all scales ( rho: 0.61–0.79). All scales demonstrated substantial and significant differences in average scores by vignette assignment ( ORs: 2.21–5.6) and overall ratings ( ORs: 5.1–14.47), supporting criterion validity.


          This study represents an innovative mixed-methods approach to testing an implementation science measure in contexts outside the United States. Results support the reliability and validity of most scales for consumers in Ukraine. Challenges included large amounts of missing data due to participants’ difficulties responding to questions about a hypothetical program.

          Electronic supplementary material

          The online version of this article (10.1186/s12913-019-4097-y) contains supplementary material, which is available to authorized users.

          Related collections

          Most cited references18

          • Record: found
          • Abstract: found
          • Article: not found

          Assessing organizational readiness for change.

          A comprehensive assessment of organizational functioning and readiness for change (ORC) was developed based on a conceptual model and previous findings on transferring research to practice. It focuses on motivation and personality attributes of program leaders and staff, institutional resources, and organizational climate as an important first step in understanding organizational factors related to implementing new technologies into a program. This article describes the rationale and structure of the ORC and shows it has acceptable psychometric properties. Results of surveys of over 500 treatment personnel from more than 100 treatment units support its construct validity on the basis of agreement between management and staff on several ORC dimensions, relationships between staff organizational climate dimensions and patient engagement in treatment, and associations of agency resources and climate with organizational stability. Overall, these results indicate the ORC can contribute to the study of organizational change and technology transfer by identifying functional barriers involved.
            • Record: found
            • Abstract: not found
            • Article: not found

            Significance Testing of the Spearman Rank Correlation Coefficient

              • Record: found
              • Abstract: found
              • Article: not found

              Pragmatic measures: what they are and why we need them.

              Pragmatic measures are important to facilitate implementation and dissemination, address stakeholder issues, and drive quality improvement. This paper proposes necessary and recommended criteria for pragmatic measures, provides examples of projects to develop and identify such measures, addresses potential concerns about these recommendations, and identifies areas for future research and application. Key criteria for pragmatic measures include importance to stakeholders in addition to researchers, low burden, broad applicability, sensitivity to change, and being actionable. Examples of pragmatic measures are provided, including ones for different settings (e.g., primary care, hospital) and levels (e.g., individual, practitioner, setting) that illustrate approaches to produce broad-scale dissemination and the development of brief, standardized measures for use in pragmatic studies. There is an important need for pragmatic measures to facilitate pragmatic research, guide quality improvement, and inform progress on public health goals, but few examples are currently available. Development and evaluation of pragmatic measures and metrics would provide useful resources to advance science, policy, and practice. Copyright © 2013. Published by Elsevier Inc. on behalf of American Journal of Medicine.

                Author and article information

                BMC Health Serv Res
                BMC Health Serv Res
                BMC Health Services Research
                BioMed Central (London )
                29 April 2019
                29 April 2019
                : 19
                : 262
                [1 ]ISNI 0000 0001 2171 9311, GRID grid.21107.35, Department of Mental Health, , Johns Hopkins Bloomberg School of Public Health, ; 624 N. Broadway 8th fl, Baltimore, MD 21205 USA
                [2 ]ISNI 0000 0001 2171 9311, GRID grid.21107.35, Department of International Health, , Johns Hopkins Bloomberg School of Public Health, ; Baltimore, USA
                [3 ]ISNI 0000 0000 9136 933X, GRID grid.27755.32, University of Virginia Curry School of Education, ; Virginia, USA
                [4 ]Center for Mental Health and Psychosocial Support National University of Kyiv-Mohyla, Kyiv-Mohyla, Ukraine
                © The Author(s). 2019

                Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

                : 18 December 2018
                : 12 April 2019
                Funded by: FundRef http://dx.doi.org/10.13039/100000200, United States Agency for International Development;
                Award ID: AID-OAA-LA-15-00003
                Funded by: National Institute of Mental Health
                Award ID: K01MH116335
                Research Article
                Custom metadata
                © The Author(s) 2019

                Health & Social care
                dissemination & implementation,measurement,validation,mixed-methods,global mental health


                Comment on this article