Back to top Open menu arrow
The Journal of Clinical Psychiatry The Journal of Clinical Psychiatry

The Journal of
Clinical Psychiatry
Access icon

This article is freely available to all

ABSTRACT

Objective: Measurement-based care (MBC) improves patient outcomes. However, there has been minimal focus on MBC for psychotic disorders. This study examines the use of patient-reported outcome measures (PROMs) in the Veterans Health Administration (VHA) to characterize their use among Veterans with psychotic disorders and to inform candidate measures for psychosis-related MBC.

Methods: Data on Veterans with and without ICD-10 psychotic disorders and at least 1 PROM during fiscal years (FYs) 2016–2019 (FY16–FY19) were collected. The sample included 3,935,504 PROM administrations among 1,192,897 Veterans. Included PROMs spanned multiple symptom and non-symptom domains. Percentages of total PROM administrations were calculated by aggregating across time and diagnosis. Facility-level statistics were also calculated. Absolute change in the percentage of unique Veterans administered a particular and repeated PROMs over time were calculated.

Results: The core PROMs for VHA MBC (Patient Health Questionnaire-9 [PHQ-9], General Anxiety Disorder-7 Scale, PTSD Checklist-5, and Brief Addiction Monitor) accounted for the majority of PROMs for Veterans with (88.18%) and without (92.56%) psychotic disorders. The PHQ-9 accounted for the largest proportion (psychotic disorder: 45.89%; other diagnosis: 46.70%). The absolute changes in percentages of repeated PROM administration were similar over time across groups.

Conclusions: The use of PROMs in VHA mental health care increased during FY16–FY19 for Veterans with and without psychosis. The rates of PROM use were similar for both groups, and PROM use predominately consisted of the core measures. While the similar rates of PROM administration are encouraging, these findings highlight the need for psychosis-specific measures to tailor MBC for Veterans with these diagnoses.

J Clin Psychiatry 2022;83(2):21m14274

To cite: Liebmann EP, Resnick SG, Stacy MA. Measurement-based care for psychotic disorders in the Veterans Health Administration: current practices and future directions. J Clin Psychiatry. 2022;83(2):21m14274.
To share: https://doi.org/10.4088/JCP.21m14274

© Copyright 2022 Physicians Postgraduate Press, Inc.

aDepartment of Psychology, VA Connecticut Healthcare System, West Haven, Connecticut
bYale School of Medicine, Department of Psychiatry, New Haven, Connecticut
cOffice of Mental Health and Suicide Prevention, VA Central Office, Washington, DC
*Corresponding author: Meaghan Stacy, PhD, Department of Psychology, VA Connecticut Healthcare System, 950 Campbell Ave – 116B, West Haven, CT 06516 ([email protected]).

CME Background

Articles are selected for credit designation based on an assessment of the educational needs of CME participants, with the purpose of providing readers with a curriculum of CME articles on a variety of topics throughout each volume. Activities are planned using a process that links identified needs with desired results.

To obtain credit, read the article, correctly answer the questions in the Posttest, and complete the Evaluation. This activity is free.

CME Objective

After studying this article, you should be able to:

  • Incorporate measurement-based care into treatment for psychotic disorders

Accreditation Statement

The CME Institute of Physicians Postgraduate Press, Inc., is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.

Credit Designation

The CME Institute of Physicians Postgraduate Press, Inc., designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 Credit™. Physicians should claim only the credit commensurate with the extent of their participation in the activity.

Note: The American Nurses Credentialing Center (ANCC) and the American Academy of Physician Assistants (AAPA) accept certificates of participation for educational activities certified for AMA PRA Category 1 Credit™ from organizations accredited by the ACCME.

Release, Expiration, and Review Dates

This educational activity was published in March 2022 and is eligible for AMA PRA Category 1 Credit™ through April 30, 2024. The latest review of this material was February 2022.

Financial Disclosure

All individuals in a position to influence the content of this activity were asked to complete a statement regarding all relevant personal financial relationships between themselves or their spouse/partner and any commercial interest. The CME Institute has resolved any conflicts of interest that were identified. In the past 3 years, Marlene P. Freeman, MD, Editor in Chief, has received research funding from JayMac and Sage; has been a member of the Independent Data Safety and Monitoring Committee for Janssen (Johnson & Johnson), Novartis, and Neurocrine; and has served on advisory boards for Eliem and Sage. As an employee of Massachusetts General Hospital (MGH), Dr Freeman works with the MGH National Pregnancy Registry, which receives funding from Alkermes, Aurobindo, AuroMedics, Johnson & Johnson/Janssen, Otsuka, Sage, Sunovion, Supernus, and Teva, and works with the MGH Clinical Trials Network and Institute, which receives research funding from multiple pharmaceutical companies and the National Institute of Mental Health. Dr Freeman has also received royalties through MGH for the Massachusetts General Hospital Female Reproductive Lifecycle and Hormones Questionnaire. No member of the CME Institute staff reported any relevant personal financial relationships. Faculty financial disclosure appears at the end of the article.

“When one is dealing with human lives and life opportunities, it is immoral to adopt a mode of decision-making which has been demonstrated repeatedly to be either inferior in success rate or, when equal, costlier to the client or the taxpayer.”1(p. viii)

Providers regularly collect and interpret a range of clinical data to make treatment decisions, but their ability to predict outcome based on clinical judgment alone is inferior to methods based on structured quantitative assessment.1–4 One such approach is measurement-based care (MBC), a multistep process that includes frequent assessment and timely feedback of outcomes during the clinical encounter, which is then used to guide clinical care.5–8 Within Veterans Health Administration (VHA) mental health services, MBC is defined as a clinical process in which a Veteran independently completes patient-reported outcome measures (PROMs) immediately prior to the clinical encounter, the results of which are discussed in the appointment and used to inform shared decision-making.8 MBC is acceptable to clients and providers9 and has numerous benefits,10–19 including improved treatment response and outcomes.13,15,16,18,20 Several organizations have recommended or required the implementation of MBC,21–24 and the VHA began the national MBC in Mental Health Initiative in 2016 with the goal of establishing MBC as the standard of care in mental health services.8

To date, the MBC literature has largely focused on depression, anxiety, and posttraumatic stress disorder (PTSD).25,26 Psychotic disorders, including schizophrenia and bipolar disorders, have significant consequences for those with the diagnosis and for society.27–32 Incorporating MBC into treatment for psychotic disorders may be beneficial given that it has the potential to reduce variability in mental health care and improve patient outcomes14 and may facilitate symptom remission33 and increase mental health literacy.34 Adding MBC to usual care can improve symptoms, interpersonal problems, social role functioning, and quality of life, especially for clients likely to experience treatment nonresponse.17 This is especially important given that people with psychotic disorders are often underserved and experience health disparities.35 In this way, MBC can be viewed as evidence-based and recovery-oriented, a priority in systems serving individuals with psychotic disorders.26,35,36

Despite the benefits of MBC and the fact that providers view it favorably,37 it has not been widely adopted.2,7,38 Beyond the obstacles that impede the adoption of MBC generally,7,8,39 implementing MBC for psychotic disorders is further complicated by a lack of well-established recommendations for psychosis-specific PROMs,25,40 challenges in using self-report measures,41 and a lack of research on measures of functioning or quality of life specifically as part of MBC for psychotic disorders. When research is limited, current clinical practices can be informative.

Understanding the use of PROMs with Veterans with psychotic disorders in VHA mental health programs may be illustrative and informative to current practitioners. This article fills a gap in the literature by examining the frequencies and proportional use of PROMs to identify which are commonly used by VHA providers when working with Veterans with psychotic disorders. We also compare the relative frequency and proportional use of the same measures among all other Veterans receiving PROMs in VHA. Characterizing use patterns will serve as an initial step to identifying potential candidate measures for psychosis-related MBC practice and studies.

METHODS

Data Acquisition and Sample

The study was approved by the Institutional Review Board at VA Connecticut Healthcare System. Data on Veterans with at least 1 mental health encounter and at least 1 completed PROM administered during fiscal years (FYs) 2016–2019 (FY16–FY19) were extracted from the VHA Corporate Data Warehouse (CDW), including variables describing age, sex, race, mental health diagnosis, and presence of standard categories of service utilization within each FY. Veterans were considered to have a diagnosis of a psychotic disorder if 2 or more outpatient mental health visits or 1 bed day of care indicated an ICD-10 diagnosis of schizophrenia, schizoaffective disorder, bipolar disorder, or other psychotic disorders.

The initial dataset included 4,106,267 PROMs from 1,214,347 Veterans. Of these, the psychotic disorder status of 21,450 (1.77%) unique Veterans changed at least once, so they were excluded to preclude counting these Veterans twice in aggregate analyses. The final analytic sample included 3,935,504 PROM administrations and 1,192,897 Veterans.

Patient Reported Outcome Measures (PROMs)

We examined a wide range of PROMs available in Mental Health Assistant (MHA) and/or Behavioral Health Laboratory (BHL), software systems that integrate with the VA’s electronic health record and store data in CDW.8 Measures recommended by a US federal interagency task force for use in MBC have been promoted as core measures in VHA24,25: the Patient Health Questionnaire-9 (PHQ-9)42 the General Anxiety Disorder-7 Scale (GAD-7)43; the PTSD Checklist-5 (PCL-5),44 and a measure of substance use created for MBC in VHA, the Brief Addiction Monitor (BAM).45 Three versions of the BAM were combined: BAM, BAM-Revised (BAM-R), and the BAM for intensive outpatient programs (BAM-IOP).

Additional PROMs spanned multiple constructs. Depression measures included the Beck Depression Inventory-II (BDI-II),46 the Center for Epidemiologic Studies Depression Scale (CES-D),47 and the Zung Depression Scale (ZUNG).48 Those measuring anxiety/stress included the Beck Anxiety Inventory (BAI)49 and the Perceived Stress Scale (PSS).50 Multisymptom measures included the Symptom Checklist-90-Revised (SCL-90-R)51 and the Behavior and Symptom Identification Scale-24 item (BASIS-24)52; given low usage, we combined the BASIS-24 scale and BASIS-4 Psychosis subscale. Insomnia-related PROMs included the Sleep Need Questionnaire (SNQ)53 and Insomnia Severity Inventory (ISI).54 Quality of life/well-being scales were the full and short versions of the World Health Organization Disability Assessment Schedule (WHODAS),55 the Quality of Life Inventory (QOLI),56 the Veterans RAND 12-Item Health Survey (VR12),57 the full and short versions of the Warwick-Edinburgh Mental Well-being Scale (WEMWBS),58 the Medical Outcomes Study Short Form 36 Health Survey (SF-36),59 the Quality of Life and Satisfaction Questionnaire–Short Form (Q-LES-Q-SF),60 and the World Health Organization Quality of Life abbreviated version (WHOQOL-BREF).61 Other measures were the Pain Outcomes Questionnaire (POQ),62 the Working Alliance Inventory–Short Revised (WAI-SR),63 the Internalized Stigma of Mental Illness Inventory (ISMI),64 and the full and short versions of the Mental Health Recovery Measure (MHRM).65

All PROMs were available for the full study period except the ISI and PCL-5. The ISI was added in June 2018. The PCL-5, which replaced the civilian (PCL-C) and military (PCL-M) versions of the PCL, was added in November 2016. In this dataset, the earliest observed administrations of the ISI and PCL-5 for both groups occurred in the second quarter of FY18 and the first quarter of FY16, respectively.

Analysis

Percentages of total PROM administrations were calculated by aggregating across time and Veterans and calculated for Veterans with and without psychotic disorders. The number and percentage of facilities administering any PROMs were calculated by aggregating the data pooled across time and diagnosis to the facility level (total = 141 facilities) and determining which sites had a total count greater than zero. Facility means, standard deviations, and minimums and maximums of the percentage total for each PROM were calculated to characterize variability in PROM utilization at the systems level.

Trends in the percentage of unique Veterans administered a particular PROM over time were examined by determining the percentage of the total number of unique Veterans per quarter who had received a particular PROM. Absolute change (%UniqueFY19Q4 – %UniqueFY16Q1) was calculated for each PROM to characterize the direction and magnitude of change over time. Trends in the repeated administration of PROMs over time were also examined. Analyses of repeated administration were limited to Veterans who had > 1 visit within a quarter (n = 675,989, 56.67%). This constraint was imposed to include in the denominator for only those Veterans for whom it would have been possible to have a repeated PROM administration (ie, had multiple visits). Percentages for each quarter were created by calculating the proportion of Veterans with > 1 PROM administration to those with > 0 PROM administrations. Absolute change in the percentage of repeated PROMs (%RepeatedFY19Q4 – %RepeatedFY16Q1) was also calculated.

RESULTS

Sample characteristics are presented in Table 1. Veterans with psychotic disorder diagnoses had lower rates of being married (30.10% vs 48.09%) and an Operation Enduring Freedom/Operation Iraqi Freedom/Operation New Dawn (OEF/OIF/OND) Veteran (17.33% vs 32.24%) compared to Veterans without psychotic disorders. A greater proportion of Veterans with psychotic disorders had alcohol use disorders (35.71% vs 21.30%) and drug use disorders (35.50% vs 14.17%) compared to Veterans without psychotic disorders. A larger percentage of Veterans with psychotic disorders had histories of outpatient substance use disorder treatment (25.12% vs 15.11%), inpatient treatment (27.51% vs 6.28%), residential treatment (12.69% vs 3.93%), and acute care (22.27% vs 3.65%).

Pooling data across time and diagnosis, all 141 VA facilities administered the 4 core MBC measures during FY16–FY19 at least once, but with varied percentages of administration (Table 2). The PHQ-9 accounted for 44% of PROMs across facilities and nearly 85% of PROMs at maximum. In comparison, the PCL-5, GAD-7, and BAM accounted for 20.27%, 16.42% and 10.28%, respectively. The 4 core PROMs accounted for 88.18% of all PROMs administered to Veterans with psychotic disorders and 92.56% of PROMs administered to Veterans without psychotic disorders.

Among the non-core measures, only the BDI-II and the BAI were administered at all facilities and accounted for the largest percentage of PROMs at the facility and diagnostic levels compared to other non-core PROMs. The quality of life/well-being PROMs accounted for the greatest number of PROMs for a single category of non-core PROMs and exhibited wide variation in systems-level use across measures. The WHODAS was administered by 99% of facilities, whereas the WHOQOL was administered at < 3% of facilities. Even among quality of life/well-being measures with nearly 100% of facilities administering them, the mean percentage of total PROMs administered across facilities did not exceed 1% for any of these measures. At the patient level, while the WHODAS, QOLI, VR-12, and WEMWBS accounted for a larger percentage of PROM administrations among Veterans with psychotic disorders compared to those without, the contribution of these measures across both diagnostic groups was low.

Among the remaining non-core PROMs, rates of use by facilities varied from 99% for the ISI to 11% for the MHRM. The mean percentage total for the measures across facilities did not exceed 0.72% for any of these measures. At the patient level, the percentages of total administration were higher among those with psychotic disorders, except for the BRS and SNQ; however, these PROMs accounted for less than 1% of all administrations for both groups.

Trends Over Time

Figure 1 depicts trajectories by group of the quarterly percentage of unique Veterans administered a core PROM and the top 6 non-core PROMs based on the percentage of total PROMs administered. The PHQ-9 had an absolute change of 4.62% and 1.47% for Veterans with and without psychotic disorders, respectively. The absolute change for Veterans with and without psychotic disorders for the PCL-5 (17.9% and 29.00%) and the GAD-7 (23.99% and 26.69%) exceeded that for the PHQ-9. The BAM had a slightly negative trajectory over time for Veterans with and without psychotic disorders (−4.00% vs −1.77%). Trajectories for the BDI-II and BAI indicate decreasing administration; the absolute change for Veterans with and without psychotic disorders were negative for the BDI-II (−8.90% vs −9.33%) and the BAI (−2.79% vs −3.64%), respectively. The absolute change for Veterans with and without psychotic disorders was 3.85% versus −0.46% for the WHODAS, 2.62% versus 0.90% for the QOLI, 2.64% versus 0.44% for the BASIS-24, and 1.09% versus 3.46% for the ISI.

The quarterly percentages of repeated PROM administrations by diagnostic group are presented in Figure 2. The percentage of repeated core PROM administrations increased during FY16–FY19. The absolute change in percentage of repeated PROM administration for Veterans with and without psychotic disorders was 16.16% versus 11.15% for the PHQ-9, 29.63% versus 25.96% for the PCL-5, 12.99% versus 9.18% for the BAM, and 9.38% versus 11.35% for the GAD-7. The absolute change for the BDI-II and BAI among Veterans with psychotic disorders was 4.11% and 8.66%, respectively, and −1.09% and 1.63%, respectively, for Veterans without psychotic disorders. The WHODAS exhibited positive absolute change for Veterans with and without psychotic disorders (9.44% vs 10.39%), while the QOLI exhibited negative absolute change for both groups (−0.86% vs −4.67%).

DISCUSSION

MBC is a low-cost, potentially low-burden, and transdiagnostic intervention that enhances the speed with which a patient receives appropriate interventions and obtains optimal results. It can improve the quality of care by influencing patient-physician interactions and encouraging shared decision-making.5 Despite its promise, MBC has not been fully adopted, and minimal literature has focused on MBC for psychotic disorders. Although there has recently been effort to highlight potential PROMs for use with psychotic disorders,40 to our knowledge, there are no research studies of self-report measures being used for MBC with psychotic disorders. This study bridges that gap by assessing what measures VHA providers are using in the treatment of psychotic disorders.

The use of PROMs in VHA mental health care for Veterans has increased in recent years, consistent with the MBC in MH Initiative and mandates from health care organizations and accrediting bodies.8 Notably, the measures used by providers did not differ based on diagnostic category; most PROMs administered to Veterans with and without psychotic disorders were 4 core measures (PHQ-9, BAM, PCL-5, and GAD-7) identified by VHA for MBC, particularly the PHQ-9. Further, the proportion of unique Veterans who completed repeated measures did not differ between those with and without psychotic disorders. While the MBC in MH Initiative required that at least one core measure be adopted, it also strongly encouraged the use of quality of life, functioning, and other non-symptom measures. However, our results suggest that, when working with Veterans with or without psychotic disorders, little experimentation occurred beyond the 4 core PROMs, and no clear consensus emerged outside of the core measures. This is encouraging in that it suggests Veterans with psychotic disorders receive MBC at similar rates to those without psychotic disorders; however, it highlights the need for psychosis-specific measures to tailor MBC for Veterans with these diagnoses.

While the pattern of PROMs usage for Veterans with psychotic disorders largely mirrored that for Veterans without psychotic disorders, it is noteworthy that the BAM was the second most common measure for individuals with psychotic disorders, but ranked fourth for those with non-psychotic disorders. This potentially reflects the higher prevalence of substance use among those with psychotic disorders66,67 and suggests providers are tailoring assessments to individual Veterans.

Given that MBC is, in part, based on the premise of using repeated measures to inform treatment, understanding the use of repeated measures within this sample is critical. For both diagnostic groups, the use of repeated measures increased over time, reflecting the increasing implementation of MBC throughout VHA. The repeated measures with the greatest increases over time were also the 4 core measures, again highlighting the importance of clear guidelines and need for psychosis-specific PROMs to inform implementation of MBC for psychotic disorders.

Psychiatric rehabilitation promotes the pursuit of personal goals despite the existence of symptoms.68,69 As such, MBC in the context of psychotic disorders is an opportunity for measures of quality of life, well-being, and recovery to guide treatment. In this study, the proportion of Veterans with psychotic disorders who completed the QOLI, WHODAS, VR-12, WEMWBS, Q-LES-Q-SF, and MHRM were larger than the proportion of Veterans without; however, the proportions who were administered these measures were small. While this finding again suggests providers are tailoring assessments to each individual patient, it also highlights the opportunity for providers to incorporate recovery-oriented measures, especially when considering the importance individuals with psychosis place on personal recovery as an outcome.40 It also suggests a potential need for additional training or specific organizational commitment to incorporating recovery-oriented measures more broadly for individuals with psychosis. Although such measures exist, they have not been systematically examined in an MBC paradigm (eg, sensitivity to change as part of treatment), and anecdotal reports suggest that providers may not know how to integrate these PROMs into clinical practice. PROMs used for MBC should measure what is targeted in treatment. If practitioners are largely focusing on symptom reduction, tracking measures that are unlikely to change over time without direct intervention, such as quality of life, may actually be detrimental to treatment outcomes and the working alliance.

For MBC to be successful, measures must be self-report, current, and actionable during the encounter5,70 and used to guide care. Ideal PROMs are brief, low-cost, publicly available, easily administered, acceptable to clients and clinicians, psychometrically sound, able to be repeated frequently, sensitive to change, and able to differentiate between desired and undesired outcomes.14,71–74 While there are candidates for measures for psychotic disorders, few meet criteria for MBC. Recently, McKenzie et al40 proposed a standardized set of 9 PROMs to support MBC for psychotic disorders, though the selection criteria did not require thresholds to depict meaningful change or severity, so their utility in making treatment decisions may be limited. It is promising that there is some overlap between this set of 9 measures and the measures identified by the current study, specifically the PHQ-9; however, it is noteworthy that the remaining 8 measures recommended by McKenzie et al40 are largely absent in current VHA practice and suggest potential measures for future investigation and implementation. Future work is needed to evaluate existing measures for MBC in psychotic disorders, create and validate brief measures for MBC in psychotic disorders, and assess the effectiveness of incorporating MBC into self-management interventions to maintain gains and recognize signs of relapse.75

As with all studies, there are several limitations to note. This study uses VHA data, which may not be representative of other settings. PROMs in VHA administrative data are limited to the assessments in MHA and BHL software. There may be other candidate measures not included in MHA/BHL that VA clinicians would use during treatment of psychotic disorders or are using but not able to document in MHA/BHL. We also cannot confirm that these measures were utilized according to the principles of measurement-based care. The data convey only that measures were administered; we do not know what transpired within the clinical encounter. Providers may not have provided feedback or used the information to make shared decisions about treatment. Future qualitative research or natural language processing studies of clinical documentation might inform our ability to understand the frequency of “share and act” within clinical encounters. Despite these limitations, this study is the first, to our knowledge, to evaluate the relative use of measures in the setting of MBC for those with and without psychotic disorders, a critical step for MBC to be widely and effectively implemented throughout mental health care systems.

Submitted: September 30, 2021; accepted December 22, 2021.
Published online: March 1, 2022.
Disclosure of off-label usage: Dr Stacy has determined that, to the best of her knowledge, no investigational information about pharmaceutical agents or device therapies that is outside US Food and Drug Administration–approved labeling has been presented in this article.
Financial disclosure: Drs Liebmann, Resnick, and Stacy have no conflicts to disclose.
Funding/support: The authors are employees of the Veterans Health Administration (VHA). This material is based upon work supported in part by the Department of Veteran Affairs Office of Academic Affiliations, Inter-Professional Advanced Fellowship in Addiction. The authors have no conflicts of interest.
Role of the sponsor: The VHA had no role in the conduct or reporting of this study.
Previous presentation: None.
Additional information: Data from the VHA Corporate Data Warehouse are stored within VHA servers and are not on a URL or accessible by the public.

Clinical Points

  • Measurement-based care (MBC) is an evidence-based intervention; however, there is little existing guidance for its use among individuals with psychotic disorders.
  • The Patient Health Questionnaire-9 (PHQ-9), General Anxiety Disorder-7 (GAD-7), PTSD Checklist-5 (PCL-5), Brief Addiction Monitor (BAM)–Revised (BAM-R), and BAM for intensive outpatient programs (BAM-IOP) are appropriate for Veterans with and without psychotic disorders.
  • Further work in developing MBC for individuals with psychotic disorders is necessary to make it consistent with recovery-oriented and person-centered care.
Editor’s Note: We encourage authors to submit papers for consideration as a part of our Focus on Psychosis section. Please contact Ann K. Shinn, MD, MPH, at [email protected].
CME LogoCME Logo

View Related Activity

Go to Activity
  1. Department of Psychology, VA Connecticut Healthcare System, West Haven, Connecticut
  2. Yale School of Medicine, Department of Psychiatry, New Haven, Connecticut
  3. Yale School of Medicine, Department of Psychiatry, New Haven, Connecticut
  4. Office of Mental Health and Suicide Prevention, VA Central Office, Washington, DC
  5. Yale School of Medicine, Department of Psychiatry, New Haven, Connecticut
  6. Office of Mental Health and Suicide Prevention, VA Central Office, Washington, DC
  7. Corresponding author: Meaghan Stacy, PhD, Department of Psychology, VA Connecticut Healthcare System, 950 Campbell Ave – 116B, West Haven, CT 06516 ([email protected]).
  1. Meehl PE. Clinical versus Statistical Prediction: A Theoretical Analysis and a Review of the Evidence. University of Minnesota Press; 1996.
  2. Hannan C, Lambert MJ, Harmon C, et al. A lab test and algorithms for identifying clients at risk for treatment failure. J Clin Psychol. 2005;61(2):155–163. PubMed CrossRef
  3. Hatfield D, McCullough L, Frantz SHB, et al. Do we know when our clients get worse? an investigation of therapists’ ability to detect negative client change. Clin Psychol Psychother. 2010;17(1):25–32. PubMed
  4. Yalom ID, Lieberman MA. A study of encounter group casualties. Arch Gen Psychiatry. 1971;25(1):16–30. PubMed CrossRef
  5. Fortney JC, Unützer J, Wrenn G, et al. A tipping point for measurement-based care. Psychiatr Serv. 2017;68(2):179–188. PubMed
  6. Lavallee DC, Chenok KE, Love RM, et al. Incorporating patient-reported outcomes into health care to engage patients and enhance care. Health Aff (Millwood). 2016;35(4):575–582. PubMed CrossRef
  7. Lewis CC, Boyd M, Puspitasari A, et al. Implementing measurement-based care in behavioral health: a review. JAMA Psychiatry. 2019;76(3):324–335. PubMed CrossRef
  8. Resnick SG, Hoff RA. Observations from the national implementation of measurement based care in mental health in the Department of Veterans Affairs. Psychol Serv. 2020;17(3):238–246. PubMed CrossRef
  9. Steinfeld B, Franklin A, Mercer B, et al. Progress monitoring in an integrated health care system: tracking behavioral health vital signs. Adm Policy Ment Health. 2016;43(3):369–378. PubMed CrossRef
  10. Dowrick C, Leydon GM, McBride A, et al. Patients’ and doctors’ views on depression severity questionnaires incentivised in UK quality and outcomes framework: qualitative study. BMJ. 2009;338:b663. PubMed CrossRef
  11. Duffy FF, Chung H, Trivedi M, et al. Systematic use of patient-rated depression severity monitoring: is it helpful and feasible in clinical psychiatry? Psychiatr Serv. 2008;59(10):1148–1154. PubMed CrossRef
  12. Eisen SV, Dickey B, Sederer LI. A self-report symptom and problem rating scale to increase inpatients’ involvement in treatment. Psychiatr Serv. 2000;51(3):349–353. PubMed CrossRef
  13. Guo T, Xiang YT, Xiao L, et al. Measurement-based care versus standard care for major depression: a randomized controlled trial with blind raters. Am J Psychiatry. 2015;172(10):1004–1013. PubMed CrossRef
  14. Harding KJK, Rush AJ, Arbuckle M, et al. Measurement-based care in psychiatric practice: a policy framework for implementation. J Clin Psychiatry. 2011;72(8):1136–1143. PubMed CrossRef
  15. Knaup C, Koesters M, Schoefer D, et al. Effect of feedback of treatment outcome in specialist mental healthcare: meta-analysis. Br J Psychiatry. 2009;195(1):15–22. PubMed CrossRef
  16. Lambert MJ, Harmon C, Slade K, et al. Providing feedback to psychotherapists on their patients’ progress: clinical results and practice suggestions. J Clin Psychol. 2005;61(2):165–174. PubMed CrossRef
  17. Lambert MJ, Whipple JL, Hawkins EJ, et al. Is it time for clinicians to routinely track patient outcome? a meta-analysis. Clin Psychol Sci Pract. 2003;10(3):288–301. CrossRef
  18. Lambert MJ, Whipple JL, Vermeersch DA, et al. Enhancing psychotherapy outcomes via providing feedback on client progress: a replication. Clin Psychol Psychother. 2002;9(2):91–103. CrossRef
  19. Swift JK, Callahan JL, Rousmaniere TG, et al. Using client outcome monitoring as a tool for supervision. Psychotherapy (Chic). 2015;52(2):180–184. PubMed CrossRef
  20. Yeung AS, Jing Y, Brenneman SK, et al. Clinical Outcomes in Measurement-based Treatment (COMET): a trial of depression monitoring and feedback to primary care physicians. Depress Anxiety. 2012;29(10):865–873. PubMed CrossRef
  21. Kroll D, Young A, Alter C. Position Statement on Utilization of Measurement Based Care. American Psychiatric Association website. https://www.psychiatry.org/File%20Library/About-APA/Organization-Documents-Policies/Policies/Position-Utilization-of-Measurement-Based-Care.pdf. Published online December 2018. Accessed August 23, 2021.
  22. Kennedy Forum. Fixing Behavioral Health Care in America: A National Call for Measurement-Based Care in the Delivery of Behavioral Health Services. The Kennedy Forum website. https://www.thekennedyforum.org/app/uploads/2017/06/KennedyForum-MeasurementBasedCare_2.pdf. Accessed January 29, 2020.
  23. The Joint Commission. Revised Outcome Measures Standard for Behavioral Health Care (Standard CTS. 03.01.09). The Joint Commission website. https://www.jointcommission.org/accreditation-and-certification/health-care-settings/behavioral-health-care/outcome-measures-standard/. 2018. Accessed January 29, 2020.
  24. U.S. Department of Veterans Affairs. Measurement Based Care in Mental Health Initiative. US Department of Veterans Affairs website. https://www.mirecc.va.gov/visn4/docs/MBCinMHInitiative.pdf. 2016. Accessed August 18, 2021.
  25. Department of Defense. Interagency Task Force on Military and Veterans Mental Health: 2013 Annual Report. Department of Veterans Affairs, & Department of Health and Human Services website. https://www.mentalhealth.va.gov/docs/2013_ITF_Report-FINAL.pdf. 2013. Accessed January 29, 2020.
  26. U.S. Department of Veterans Affairs. VHA Handbook 1160.01 Uniform Mental Health Services in VA Medical Centers and Clinics. US Department of Veterans Affairs website. https://www.va.gov/vhapublications/ViewPublication.asp?pub_ID=1762. 2008. Accessed January 29, 2020.
  27. Insel TR. Assessing the economic costs of serious mental illness. Am J Psychiatry. 2008;165(6):663–665. PubMed CrossRef
  28. Kessler RC, Aguilar-Gaxiola S, Alonso J, et al. The global burden of mental disorders: an update from the WHO World Mental Health (WMH) surveys. Epidemiol Psichiatr Soc. 2009;18(1):23–33. PubMed CrossRef
  29. Laursen TM, Nordentoft M, Mortensen PB. Excess early mortality in schizophrenia. Annu Rev Clin Psychol. 2014;10(1):425–448. PubMed CrossRef
  30. Olfson M, Gerhard T, Huang C, et al. Premature mortality among adults with schizophrenia in the United States. JAMA Psychiatry. 2015;72(12):1172–1181. PubMed CrossRef
  31. Palmer BA, Pankratz VS, Bostwick JM. The lifetime risk of suicide in schizophrenia: a reexamination. Arch Gen Psychiatry. 2005;62(3):247–253. PubMed CrossRef
  32. Tandon R, Nasrallah HA, Keshavan MS. Schizophrenia, “just the facts” 5. Treatment and prevention. Past, present, and future. Schizophr Res. 2010;122(1-3):1–23. PubMed CrossRef
  33. Rush AJ. Isn’t it about time to employ measurement-based care in practice? Am J Psychiatry. 2015;172(10):934–936. PubMed CrossRef
  34. Valenstein M, Adler DA, Berlant J, et al. Implementing standardized assessments in clinical care: now’s the time. Psychiatr Serv. 2009;60(10):1372–1375. PubMed CrossRef
  35. Mongelli F, Georgakopoulos P, Pato MT. Challenges and opportunities to meet the mental health needs of underserved and disenfranchised populations in the United States. Focus Am Psychiatr Publ. 2020;18(1):16–24. PubMed CrossRef
  36. President’s New Freedom Commission on Mental Health. Achieving the Promise: Transforming Mental Health Care in America. Suicide Prevention Resource Center website. https://www.sprc.org/resources-programs/achieving-promise-transforming-mental-health-care-america. 2003. Accessed January 29, 2020.
  37. Oslin DW, Hoff R, Mignogna J, et al. Provider attitudes and experience with measurement-based mental health care in the VA Implementation Project. Psychiatr Serv. 2019;70(2):135–138. PubMed CrossRef
  38. Zimmerman M, McGlinchey JB. Why don’t psychiatrists use scales to measure outcome when treating depressed patients? J Clin Psychiatry. 2008;69(12):1916–1919. PubMed CrossRef
  39. Boswell JF, Kraus DR, Miller SD, et al. Implementing routine outcome monitoring in clinical practice: benefits, challenges, and solutions. Psychother Res. 2015;25(1):6–19. PubMed CrossRef
  40. McKenzie E, Matkin L, Sousa Fialho L, et al; Psychotic Disorders Working Group of the International Consortium for Health Outcomes Measurement. Developing an international standard set of patient-reported outcome measures for psychotic disorders. Psychiatr Serv. 2021;s202000888. PubMed CrossRef
  41. Kølbæk P, Dines D, Holm T, et al. Clinical validation of ratings on the six-item Positive and Negative Syndrome Scale obtained via the Simplified Negative and Positive Symptoms Interview. J Psychopharmacol. 2021;35(9):1081–1090. PubMed doi.org/10.177/269881121996890
  42. Spitzer RL, Kroenke K, Williams JB. Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. JAMA. 1999;282(18):1737–1744. PubMed CrossRef
  43. Spitzer RL, Kroenke K, Williams JBW, et al. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006;166(10):1092–1097. PubMed CrossRef
  44. Weathers FW, Litz BT, Keane TM, et al. The PTSD Checklist for DSM-5 (PCL-5). US Department of Veterans Affairs website. https://www.ptsd.va.gov/. Accessed August 18, 2021.
  45. Cacciola JS, Alterman AI, Dephilippis D, et al. Development and initial evaluation of the Brief Addiction Monitor (BAM). J Subst Abuse Treat. 2013;44(3):256–263. PubMed CrossRef
  46. Beck AT, Steer RA, Ball R, et al. Comparison of Beck Depression Inventories -IA and -II in psychiatric outpatients. J Pers Assess. 1996;67(3):588–597. PubMed CrossRef
  47. Radloff LS. The CES-D scale: a self-report depression scale for research in the general population. Appl Psychol Meas. 1977;1(3):385–401. CrossRef
  48. Zung WW. A self-rating depression scale. Arch Gen Psychiatry. 1965;12(1):63–70. PubMed CrossRef
  49. Beck AT, Epstein N, Brown G, et al. An inventory for measuring clinical anxiety: psychometric properties. J Consult Clin Psychol. 1988;56(6):893–897. PubMed CrossRef
  50. Cohen S, Kessler RC, Gordon LU. Measuring Stress: A Guide for Health and Social Scientists. Oxford University Press; 1997:xii, 236.
  51. Derogatis LR. SCL-90-R, Administration, Scoring and Procedures Manual-II for the R(Evised) Version and Other Instruments of the Psychopathology Rating Scale Series. Clinical Psychometric Research; 1992.
  52. Eisen SV, Normand SLT, Belanger AJ, et al. BASIS-32and the Revised Behavior and Symptom Identification Scale BASIS-R. In: Maruish MM, ed. The Use of Psychological Testing for Treatment Planning and Outcome Assessment. 3rd ed. Lawrence Erlbaum; 2004:79–113.
  53. Spielman AJ. Sleep Need Questionnaire. Adapted from an unpublished work from Arthur J. Spielman, PhD, The City College of New York. https://www.mirecc.va.gov/docs/visn6/Improve_Your_Sleep_Self-Guided_Approach_for_Veterans_with_Insomnia-March-2017.pdf. Accessed September 14, 2021.
  54. Bastien CH, Vallières A, Morin CM. Validation of the Insomnia Severity Index as an outcome measure for insomnia research. Sleep Med. 2001;2(4):297–307. PubMed CrossRef
  55. Üstün TB, Kostanjsek N, Chatterji S, et al, eds. Measuring Health and Disability: Manual for WHO Disability Assessment Schedule WHODAS 2.0. World Health Organization; 2010.
  56. Frisch MB. Quality of Life Assessment/Intervention and the Quality of Life Inventory TM (QOLI). In: The Use of Psychological Testing for Treatment Planning and Outcomes Assessment. Lawrence Erlbaum Associates Publishers; 1999:1277–1331.
  57. Iqbal SU, Rogers W, Selim A, et al. The Veterans RAND 12 Item Health Survey (VR-12): What it is and how it is used. CHQOERs VA Medical Center Bedford. MA: CAPP Boston University School of Public Health; 2015:1–12., https://www.bu.edu/sph/files/2015/01/veterans_rand_12_item_health_survey_vr-12_2007.pdf.
  58. Tennant R, Hiller L, Fishwick R, et al. The Warwick-Edinburgh Mental Well-being Scale (WEMWBS): development and UK validation. Health Qual Life Outcomes. 2007;5(1):63. PubMed CrossRef
  59. Ware JE, Snow KK, Kosinski M, et al. SF-36 Health Survey: Manual and Interpretation Guide. Health Institute, New England Medical Center; 1993.
  60. Endicott J, Nee J, Harrison W, et al. Quality of Life Enjoyment and Satisfaction Questionnaire: a new measure. Psychopharmacol Bull. 1993;29(2):321–326. PubMed
  61. The WHOQOL Group. Development of the World Health Organization WHOQOL-BREF quality of life assessment. Psychol Med. 1998;28(3):551–558. PubMed CrossRef
  62. Clark ME, Gironda RJ, Young RW. Development and validation of the Pain Outcomes Questionnaire-VA. J Rehabil Res Dev. 2003;40(5):381–395. PubMed CrossRef
  63. Hatcher RL, Gillaspy JA. Development and validation of a revised short version of the working alliance inventory. Psychother Res. 2006;16(1):12–25. CrossRef
  64. Ritsher JB, Otilingam PG, Grajales M. Internalized stigma of mental illness: psychometric properties of a new measure. Psychiatry Res. 2003;121(1):31–49. PubMed CrossRef
  65. Young SL, Bullock WA. The Mental Health Recovery Measure. Department of Psychology, University of Toledo; 2009.
  66. Mueser KT, Bennet M, Kushner MG. Epidemiology of substance use disorders among persons with chronic mental illnesses. In: Lehman AF, Dixon LB, eds. Double-Jeopardy: Chronic Mental Illness and Substance Use Disorders. Harwood Academic Publishers; 1995:9–25.
  67. Center for Behavioral Health Statistics and Quality. Key Substance Use and Mental Health Indicators in the United States: Results from the 2015 National Survey on Drug Use and Health. Substance Abuse and Mental Health Services Administration. 2016.
  68. Anthony WA. Integrating psychiatric rehabilitation into managed care. Psychiatr Rehabil J. 1996;20(2):39–44. CrossRef
  69. Kern RS, Glynn SM, Horan WP, et al. Psychosocial treatments to promote functional recovery in schizophrenia. Schizophr Bull. 2009;35(2):347–361. PubMed CrossRef
  70. Bickman L, Kelley SD, Breda C, et al. Effects of routine feedback to clinicians on mental health outcomes of youths: results of a randomized trial. Psychiatr Serv. 2011;62(12):1423–1429. PubMed CrossRef
  71. Bech P. Clinical Psychometrics. Wiley-Blackwell; 2012.
  72. Bech P. Measurement-Based Care in Mental Disorders. Springer International Publishing; 2016.
  73. Morris DW, Trivedi MH. Measurement-based care for unipolar depression. Curr Psychiatry Rep. 2011;13(6):446–458. PubMed CrossRef
  74. Zimmerman M, Young D, Chelminski I, et al. Overcoming the problem of diagnostic heterogeneity in applying measurement-based care in clinical practice: the concept of psychiatric vital signs. Compr Psychiatry. 2012;53(2):117–124. PubMed CrossRef
  75. Youngstrom EA, Choukas-Bradley S, Calhoun CD, et al. Clinical guide to the evidence-based assessment approach to diagnosis and treatment. Cognit Behav Pract. 2015;22(1):20–35. CrossRef  
Download Free PDF

Save Save

Share Share

Cite Cite

Celebrating
85 Years

Read the Journal of Clinical Psychiatry

Subscribe to read
more articles

Subscribe now