How might we go about assessing and, where needed, improving the quality of psychotherapy
practice delivered in usual care? Given the numbers of mental health specialists practicing
in the U.S. (recently estimated at one-half million), the diversity of professional
training and licensure that establish credentials to practice therapy, a broad range
of practice settings, and variations in the types of individuals treated, the task
of answering this question seems daunting (Institute of Medicine 2006a). This special
issue includes a range of studies that have made first steps toward tackling this
challenging task. Not surprisingly, given the groundbreaking nature of this research,
these papers overwhelmingly raise more questions than they answer. A large part of
the value of these studies is lessons learned that inform next steps to assess and
improve the quality of psychotherapy in usual care.
Quality of Care Framework
Donabedian’s Quality of Care model, commonly used in healthcare quality assessment
and improvement research, is useful for putting the issues addressed in this special
issue into broad context (Donabedian 1988). In this framework, healthcare can improve
the expected outcomes of individuals who access care through two broad domains: (1)
structural aspects of care, such as facilities, staff competencies, equipment, organization
of care; and (2) processes of care—the specific evaluation and treatment encounters
experienced by the individual. In order to improve the quality of healthcare, we need
to establish causal links from structural aspects of care to processes of care, and
from processes of care to outcomes. With key links established to outcomes, we can
assess relevant structural or process components of usual care, target healthcare
improvement efforts to those areas where care deviates from what the evidence holds
to be best for producing health outcomes, and evaluate whether our quality improvement
efforts had the intended effects of closing the gap between usual care and best practices,
and improving outcomes. Although it is not an easy matter to establish causal linkages
between aspects of healthcare structure, process and health outcomes, the field of
health services research, and in particular, clinical effectiveness research, is contributing
to a growing body of empirical evidence that has identified some of these important
relationships for specific populations and medical, including behavioral health, conditions.
Within the Donabedian framework, the papers in this issue are concerned with the problem
of how to assess the processes of mental health care when psychotherapy is a component
of treatment delivered in usual care. They are not trying to establish the causal
links between processes of care and outcomes. Instead, they are trying to describe
usual care psychotherapy practice in all of its diversity, and determine how patterns
of usual care relate to what has been learned from clinical literature about effective
psychotherapy practices. If researchers are successful in developing methodologic
approaches and measurement tools to assess usual care psychotherapy, then these tools
can provide a platform upon which to develop feedback systems to support routine quality
assessment and continuous quality improvement.
Measurement Challenges
The measurement challenges are considerable, as these papers attest. There are numerous
approaches that can be taken, and trade-offs to consider in each decision regarding
approach. The investigators have wrestled with these decisions and made choices in
their approaches, helping us consider the variety of approaches and strengths and
limitations of alternatives. Below, we summarize five questions regarding measurement
approach that emerged as prominent themes in this issue.
1) At what level should the unit of observation be set, given the possible range from
small and finely grained behavioral interaction units to more encompassing and broadly
defined therapeutic orientations? The research examples in this issue all have selected
a middle-ground, with the unit of observation being techniques, strategies, or goals
that are specific components of broader therapeutic approaches. Garland and colleagues
provide an insightful discussion of why this level of observation is preferable over
other choices that might be made (Garland et al. 2009). Kelley and colleagues suggest
that many therapists can more easily report on the topic of their focus in a therapy
session (e.g., a behavioral problem) than on the techniques they use (Kelley et al.
2009).
2) How broadly should assessment cover the range of practice in psychotherapy? Approaches
in this issue have ranged from a broad scope that attempts to cover most of what is
observed in mental health care for children (Garland et al. 2009; Hurlburt et al.
2009; Kelley et al. 2009) to a more narrow focus on identifying elements of prominent
therapeutic approaches in the treatment of depression (Hepner et al. 2009a; Miranda
et al. 2009). Differences in breadth of the assessment approach reflects differences
in study aims: a broad assessment was suited to the goal of describing natural variation
in psychotherapy practice and the associations between that natural variation and
outcomes; the narrow focus on specific psychotherapy techniques for depression treatment
aimed to understand the extent to which evidence-based practices were delivered in
usual care. Both approaches appear to be fruitful ways of generating knowledge about
usual care psychotherapy practice.
3) From whose perspective—clinician, client, objective observer—should the assessment
of psychotherapy be made? Each of these perspectives has strengths and limitations,
with examples of each provided among the studies in this issue. The perspective of
the clinician gives insight into the intent of practice that other perspectives cannot
capture; the client perspective is unique in reflecting understanding or awareness
that clients took from their therapeutic experience, and the observer perspective
is unique in capturing observable behavior and verbal interactions that occurred during
therapy. There is no gold standard against which to evaluate the validity of alternative
perspectives. Each perspective is relevant and it remains to be seen whether one or
another, or perhaps a combined approach, will emerge as a more useful and practical
way to assess psychotherapy practice in usual care.
4) Can assessment approaches incorporate important aspects of the changing, dynamic
nature of psychotherapy over time? While some therapy elements or techniques may be
present in every therapy session (e.g., agenda setting or agreeing upon ‘homework’
in a cognitive behavioral therapy session), some therapy elements may be more commonly
used during a particular stage of therapy. For example, within cognitive behavioral
therapy, educating the client about the cognitive model for depression would typically
be a primary focus during the first few sessions of therapy and would receive less
emphasis in later stages of therapy. One-time assessments that look back on the full
course of therapy to date may need to take into account the number of sessions the
client has received at the time of the assessment. Alternatively, approaches that
characterize a single session in a client’s therapy may have difficulty capturing
key aspects of therapy that change over time.
5) Is it possible to develop low-cost, low-burden assessment tools that could feasibly
be used routinely and widely as part of information system support for improving the
quality of mental health care? While we want the answer to this question to be “yes”,
it is clear that the assessment task is sufficiently complex to warrant further measurement
development, evaluation, and refinement before we can hope that acceptable tools will
be available. We join others (Bickman 2008) in emphasizing the need for and critical
importance of developing information systems that provide feedback on outcomes and
processes of care to support mental health care improvement. Further research is needed
on methods and measures that are appropriate components of these information systems.
Opening the Black Box
Psychotherapy, as practiced in usual care, has long been considered a black box—a
process in which the inner components and processes aren’t easily known. Some studies
have reported on the number of psychotherapy visits received by those getting mental
health care, as an indicator of the quality of care that was received (Wang et al.
2005). While information about number of visits is often available in administrative
data or medical records, it is clearly limited for understanding the content and quality
of the therapy received. Researchers can reasonably infer that those receiving very
few visits were unlikely to have received even minimally adequate care, but cannot
distinguish poor from high quality therapy among those receiving many visits. Clinician
credentials, such as educational background and professional licensure, indicate a
minimum level of professional competency, but provide little indication of the therapy
processes practiced by the clinician. Certification in a specific therapeutic approach
is one approach to ensure clinician competency to practice it. For example, the Academy
of Cognitive Therapy (www.academyofct.org) certifies clinicians as competent in cognitive
therapy following an application process that includes an independent reviewer coding
an audio taped session using the Cognitive Therapy Rating Scale (Young and Beck 1980).
While certification does not guarantee that a clinician will continue to competently
practice cognitive therapy, it does indicate that the clinician has demonstrated an
ability to do so. Many clinicians report that they use eclectic (that is, multiple)
or integrative (combinational) approaches (Norcross et al. 2005), but it is not clear
how techniques from multiple approaches might be selected and combined in any specific
episode of treatment. So while clinician training, licensing, and credentialing are
integral to the production of therapist workforce skills and capacities, we cannot
infer the content or quality of psychotherapy from knowing them.
What do we see when we look into the black box? Several articles in this issue have
provided a first empirical glimpse into usual care psychotherapy. Hepner et al. (2009b),
examined usual care psychotherapy for adults with depression, finding that clinicians
often used some techniques associated with evidence-based therapies—Cognitive Behavioral
Therapy (CBT) and Interpersonal Therapy (IPT)—but other core techniques from these
therapeutic approaches were also among the least endorsed. The results suggested that
many clinicians are using an eclectic approach to treating depression, that is, using
some techniques from a variety of approaches. In contrast to pure forms of CBT and
IPT, the effectiveness of such eclectic approaches is largely unknown. Brookman-Frazee
et al. (2009), similarly found that empirically supported practice elements were used
with low intensity in usual care psychotherapy for youth. And these authors found
no clear pattern of characteristics associated with delivery of empirically supported
practice elements, highlighting the complexity of understanding usual care psychotherapy
practice. The work from Hurlburt et al. (2009), suggests that clinicians may find
it difficult to accurately report on their own therapy practice in the absence of
training in how to do so, based on a study that compared these reports to coded observations.
Taken together, these attempts to open the black box suggest that a wide range of
techniques are used in usual care psychotherapy, and that clinicians are generally
not using empirically supported approaches with the same purity and intensity demonstrated
to be effective in clinical trials. Consistent with this view, Landry et al. (2009)
found that, among a national sample of adults who reported receiving some form of
mental health or substance abuse care, a minority claimed to have received specific
counseling consistent with components of evidence-based treatments for the most common
disorders.
Future Research
The measurement and description of psychotherapy in usual care will provide a platform
for addressing other important research questions. Below, we list several of these
questions.
What is the relationship of workforce training and competencies to psychotherapy practice
patterns? A professional coalition on mental health workforce issues made a strong
case for reform in graduate training to better prepare mental health specialists for
practice in today’s healthcare environment (Hoge et al. 2005), and SAMHSA has called
for discussion of “An Action plan for Behavioral Health Workforce Development” (Annapolis
Coalition 2007), but to our knowledge, no federal efforts are yet underway to implement
the plan. Proposals to reform workforce training and development implicitly assume
the link between structural aspects of mental health care (workforce composition and
competency) and the processes of care (delivery of treatments that are more appropriate
and effective to those individuals who seek care). This is a reasonable assumption,
but there are important questions to be addressed regarding the kind of workforce
training and development that is needed to improve usual care practices. In addition,
it is likely that other organizational supports and incentives are needed to facilitate
the effectiveness of a well-trained workforce.
What is the relationship between practice patterns and outcomes? Processes of care
should also be linked to outcomes. Specifically, we expect that practices based on
the best available clinical evidence, such as evidence-based guidelines, will lead
to better outcomes in usual care than when practice deviates from evidence-based guidelines.
Some studies have demonstrated that specific approaches to improving the delivery
of mental health care in primary medical care settings results in better outcomes
for patients (Meredith et al. 2006; Wells et al. 2000), but there is still much to
be learned about how to improve the outcomes of psychotherapy in usual care, in both
specialty behavioral health and general medical care settings. Controlled clinical
studies are necessarily limited in their generalizability across broader client populations.
Studies of outcomes of psychotherapy in usual care can identify areas in which the
evidence base needs to be strengthened to address clinical challenges. Socially and
culturally diverse clients, complex health and mental health conditions, and diverse
preferences and life circumstances are all realities of usual care that may moderate
the effectiveness of evidence-based care. Identifying those for whom existing evidence-based
practice is ineffective will point to important ways that clinical practice needs
to be improved. Studies of the relationship between psychotherapy practice and outcomes
in usual care could also suggest promising new treatment approaches that deserve further
evaluation.
How can structural aspects of the healthcare environment improve the practice of psychotherapy?
There is growing understanding of the importance of aligning financing incentives,
and of putting into place organizational structures and informational systems that
support best practices. Producing high quality care requires more than a well-trained
workforce; mental health specialists need an environment in which excellent care is
rewarded and the environment is organized to support the delivery of excellent care
(Institute of Medicine 2006a). In spite of a growing literature on the ways that financing
incentives, organizational models, and informational systems can improve the quality
of health and mental health care, we know very little about structural aspects of
the health care environment that can support the delivery of high quality psychotherapy.
Will development of more informed consumers of psychotherapy improve the practice
of psychotherapy in usual care? The importance of consumer-oriented care, where consumers
are informed partners in their decisions about and management of healthcare is increasingly
recognized in general and mental health care (Katon et al. 1995; Mueser et al. 2002;
Wagner et al. 2001). For psychotherapy, it is perhaps obvious that consumer engagement
and participation in the therapeutic process is essential, and will influence the
unfolding of the psychotherapy process as well as its results. But there are many
impediments to the free flow of information about and transparency of mental health
care to consumers, especially given the stigma and discrimination associated with
mental illness (Institute of Medicine. 2006b). Further research is needed to understand
how to promote more informed and actively participating consumers of psychotherapy.
Conclusions
These seminal papers begin to lay the important foundation needed to improve the quality
of psychotherapy provided to our nation’s vulnerable populations in need of mental
health care. Although much work remains ahead of us, these papers begin to develop
a methodology for defining the psychotherapy that is provided to millions of Americans
each year. As we understand psychotherapy in practice, we develop the tools needed
to routinely monitor the processes and outcomes of mental health care. These tools
provide the platform to improve care and provide relief from suffering and opportunities
for growth of the many individuals suffering from mental disorders.