Validity of Posttraumatic Stress Disorder Service Connection Status in Veterans Affairs Electronic Records of Iraq and Afghanistan Veterans

Validity of Posttraumatic Stress Disorder Service Connection Status in Veterans Affairs Electronic Records of Iraq and Afghanistan Veterans

Brian P. Marx, PhDa,b,*; Michelle J. Bovin, PhDa,b; Derek D. Szafranski, PhDa,c; Eden Engel-Rebitzer, BAa; Matthew W. Gallagher, PhDa,b; Darren W. Holowka, PhDa,b,d; Paula P. Schnurr, PhDe,f; Raymond C. Rosen, PhDg; and Terence M. Keane, PhDa,b

ABSTRACT

Objective: This study examined the extent to which veterans’ posttraumatic stress disorder (PTSD) service connection (SC) status corresponded to their PTSD diagnostic status, as determined by a semistructured diagnostic interview.

Method: Participants were 834 veterans in the Veterans After-Discharge Longitudinal Registry (Project VALOR), an observational registry of veterans with and without PTSD who are enrolled in the Veterans Affairs (VA) health care system. PTSD diagnostic status was confirmed using the Structured Clinical Interview for DSM-IV (SCID).

Results: Concordance between PTSD SC status and current and lifetime PTSD diagnosis was 70.2% and 77.2%, respectively. Individuals with PTSD SC were twice as likely as those without PTSD SC to have a current SCID PTSD diagnosis (OR = 2.11 [95% CI, 1.47-3.04]; P < .001) and almost 3 times as likely to have a lifetime SCID PTSD diagnosis (OR = 2.72 [95% CI, 1.67-4.41]; P < .001). For current PTSD, results showed a slightly higher proportion of false positives—individuals who did not meet SCID criteria but who did have SC for PTSD—than false negatives—individuals who met SCID criteria but did not have SC for PTSD. For lifetime PTSD, the proportion of false negatives was approximately twice the proportion of false positives.

Conclusions: PTSD diagnostic and SC status are discordant for a significant minority of veterans. Findings revealed that both the number of veterans who are service connected without meeting criteria for PTSD and the number of veterans who meet PTSD criteria but have not been granted SC status are concerning.

J Clin Psychiatry 2016;77(4):517-522

dx.doi.org/10.4088/JCP.14m09666

© Copyright 2016 Physicians Postgraduate Press, Inc.

aVA National Center for PTSD, VA Boston Healthcare System, Boston, Massachusetts

bBoston University School of Medicine, Boston, Massachusetts)

cRalph H. Johnson VA Medical Center, Charleston, South Carolina

dRhode Island Hospital, Providence

eVA National Center for PTSD, VA Medical Center, White River Junction, Vermont

fGeisel School of Medicine at Dartmouth, Hanover, New Hampshire

gNew England Research Institutes, Watertown, Massachusetts

*Corresponding author: Brian P. Marx, PhD, National Center for PTSD, VA Boston Healthcare System (116B-4), 150 South Huntington Ave, Boston, MA 02130 ([email protected]).

The prevalence of claimed mental disorder-related disability among United States military veterans has risen sharply over the past decade. Many of those with such disabilities have sought compensation from the Department of Veterans Affairs (VA) on the grounds that those claimed disabilities resulted from or were aggravated during military service. By far, the most frequently claimed service-connected mental disorder is posttraumatic stress disorder (PTSD).1,2 By the end of fiscal year (FY) 2013, 648,992 veterans were receiving disability compensation for PTSD, a 72% increase from FY 2008.2 According to a VA Office of Inspector General report, growth in PTSD claims exceeds that of applications for other conditions.3

The sharp increase in veterans seeking and receiving compensation for service-connected PTSD has bolstered concerns that financial incentives lie at the heart of many of these disability claims.4,5 Others have countered that, although there are certainly documented cases of fraud, we do not have sufficient data to establish the prevalence of false service-connected PTSD claims among veterans.6 For many reasons, it may never be possible to determine the prevalence of false PTSD disability claims. However, it may be possible to determine the extent to which veterans’ PTSD service connection (SC) status (and concomitant eligibility for disability compensation and other VA benefits) corresponds to their PTSD diagnostic status as determined by a "gold standard" clinician-administered PTSD interview at a given point in time, without necessarily knowing the reasons for discrepancies. Having such information could provide us with a better understanding of the degree to which we should be concerned about veterans receiving PTSD disability benefits when they should not have them as well as veterans not receiving PTSD disability benefits when they should.

In this investigation, we used data from an ongoing study, the Veterans After-Discharge Longitudinal Registry (Project VALOR),7 to examine the extent to which veterans’ PTSD SC status in VA’s electronic medical record (EMR) was concordant with the results of an independent, semistructured diagnostic interview conducted to determine PTSD diagnostic status. Because not all veterans with PTSD seek PTSD SC, we focused only on those veterans who had documentation of an SC examination for PTSD. To better interpret the results of these analyses, we also examined the extent to which there were significant differences in self-reported functional impairment and number of PTSD symptoms endorsed between participants classified as (1) true positive versus false negative, (2) true negative versus false positive, (3) true negative versus false negative, and (4) true positive versus false positive. Finally, we examined whether participants classified as true positive differed from participants classified as false positive in the percentage of PTSD SC received.

METHOD

Participants

Participants in the current study were a subsample of US Army or Marine veterans enrolled between 2009 and 2012 in the baseline assessment of Project VALOR, a registry of VA mental health care users with and without PTSD who deployed in support of Operation Enduring Freedom, Operation Iraqi Freedom, or Operation New Dawn. To be included in the cohort, veterans must have undergone a mental health evaluation at a VA facility. Veterans with probable PTSD according to VA medical records (ie, at least 2 instances of a PTSD diagnosis by a mental health professional associated with 2 separate visits) were oversampled at a 3:1 ratio, and female veterans were oversampled at a rate of 1:1 (female:male). Potential Project VALOR participants were recruited from a roster of veterans who met inclusion criteria. The roster was provided by the VA Environmental Epidemiology Service. Potential participants (N = 4,331) were contacted by phone, and, of these, 2,712 (62.6%) consented to participate. Of participants who gave consent, 2,169 (80.0%) completed the questionnaires and 1,649 (60.8%) completed both the questionnaires and the diagnostic interview. The data from these 1,649 male and female veterans were included in Project VALOR.

• It is unknown whether veterans’ posttraumatic stress disorder (PTSD) service connection (SC) disability status corresponds to their PTSD diagnostic status based on a diagnostic interview.
• We found a significant minority of veterans for whom PTSD SC status and PTSD diagnostic status were mismatched.
• Both the number of veterans who were service connected but did not meet criteria for PTSD and the number of veterans who met PTSD criteria but had not been granted SC status were concerning.

Our sample consisted of all Project VALOR participants who reported a military-related trauma during the Structured Clinical Interview for DSM-IV (SCID),8 were assessed for both current and lifetime PTSD diagnostic status, and had documentation of a compensation and pension (C&P) PTSD examination in their EMRs. We excluded 780 participants because they did not have this documentation and an additional 35 participants because they were not assessed for both current and lifetime PTSD. Our sample (N = 834) ranged in age from 22 to 67 years (mean = 38.1; SD = 9.9), and the majority of the sample (84.2%) had completed at least some college. Fifty-four percent (n = 454) of participants were men. Ninety-two percent of the sample (n = 766) served in the Army, and 8.2% (n = 68) served in the Marines. Respondents who did not meet the inclusion criteria were younger (mean age = 36.6, SD = 9.5; range, 22-68 years; P = .003), less likely to be male (45.4%, n = 370, P < .001), and more likely to serve in the Marines (11%, P = .05).

Procedure

Participants provided informed consent verbally over the telephone in accordance with the research protocol approved by all local institutional review boards and the Human Research Protection Office of the US Army Medical Research and Materiel Command. After receiving verbal consent, study staff scheduled the telephone interview and reminded the participant to complete an online self-administered survey. Participants were compensated $50 for their participation.

Measures

PTSD diagnostic status. Trained, doctoral-level diagnosticians assessed both current (past month) and lifetime PTSD via telephone using the PTSD Module of the Structured Clinical Interview for DSM-IV (SCID).8 The PTSD SCID module has demonstrated excellent psychometric properties.9 To remain consistent with PTSD SC examinations, diagnosticians assessed all participants for their worst military-related traumatic event.

Interviewers were blind to PTSD SC status. Regular meetings with assessment personnel were held to ensure diagnostic reliability and to prevent rater drift. Interrater reliability for the SCID computed on the basis of a randomly selected subsample was excellent for both lifetime (κ = 0.87) and current (κ = 0.91) PTSD diagnosis. SCID PTSD diagnostic status was the standard to which PTSD SC status was compared.

Functional impairment. Functional impairment was assessed online using the Inventory of Psychosocial Functioning (IPF),10 an 80-item self-report measure designed to assess multiple dimensions of functional impairment in the past 30 days. Items are rated on a 7-point scale (0 = never and 6 = always). The IPF has excellent psychometric properties.11

Service connection status. Trained research assistants collected PTSD SC status information by accessing the disabilities section of participants’ EMRs. Documentation of the occurrence of PTSD C&P examinations was collected by accessing the C&P section, the progress notes section, and the health summaries section of the EMRs.

Statistical Analysis Plan

We conducted 3 types of analyses to examine concordance between PTSD SC status and diagnostic status. First, analyses were conducted to examine both the overall association between PTSD SC status and SCID PTSD diagnostic status (both current and lifetime) and the directionality of concordance/discordance. Specifically, we calculated 2 ×— 2 contingency tables and classified the 4 possible outcomes in these concordance analyses as (1) true positives (PTSD SC is Yes and SCID PTSD diagnosis is Yes), (2) false negatives (PTSD SC is No and SCID PTSD diagnosis is Yes), (3) false positives (PTSD SC is Yes and SCID PTSD diagnosis is No), and (4) true negatives (PTSD SC is No and SCID PTSD diagnosis is No). Overall concordance was calculated by summing the true positives and true negatives. The association between PTSD SC status and PTSD diagnostic status was further examined by calculating odds ratios for any given individual with PTSD SC status meeting criteria for current and lifetime PTSD based on the SCID.

Second, we calculated a series of diagnostic accuracy statistics to examine the performance of PTSD SC status in comparison to both current and lifetime SCID PTSD diagnostic status. Third, Cohen d effect sizes with 95% confidence intervals (CI) were calculated to examine whether there were significant differences in self-reported functional impairment and PTSD symptom count between individuals classified as (1) true positive versus false negative, (2) true negative versus false positive, (3) true negative versus false negative, and (4) true positive versus false positive. We also calculated Cohen d to determine whether true positives and false positives differed in the percentage of PTSD SC received. When CIs for Cohen d did not include 0, the effect was considered statistically significant.12

RESULTS

The overall concordance between PTSD SC status and current and lifetime SCID diagnosis was 70.2% and 77.2%, respectively. Individuals with PTSD SC were twice as likely as those without PTSD SC to have a current SCID PTSD diagnosis (OR = 2.11 [95% CI, 1.47-3.04]; P < .001) and almost 3 times as likely to have a lifetime SCID PTSD diagnosis (OR = 2.72 [95% CI, 1.67-4.41]; P < .001). For both current and lifetime PTSD, true positive was the most frequent outcome and true negative was the least frequent outcome. There were more false positives than false negatives for current PTSD. In contrast, there were more false negatives than false positives for lifetime PTSD (Table 1).

Click figure to enlarge

We next calculated diagnostic accuracy statistics for PTSD SC status in reference to both current and lifetime SCID PTSD diagnostic status. Sensitivity, specificity, positive predictive value, and negative predictive values for both current and lifetime PTSD diagnosis are presented in Table 2. Positive predictive values were greater than negative predictive values for both current and lifetime PTSD. The ϕ and κ coefficients were nearly identical for both current and lifetime PTSD and were relatively poor for both.

Click figure to enlarge

Analyses comparing true positives (individuals for whom their PTSD SC status accurately reflected the presence of PTSD according to the SCID assessment) with the false negatives (those who were not service connected for PTSD despite the presence of PTSD according to the SCID assessment) on functional impairment and PTSD symptom count showed that true positives reported significantly more PTSD symptoms than false negatives for both current and lifetime PTSD, but no differences on functional impairment (Table 3). Analyses comparing true negatives (individuals for whom their PTSD SC status accurately reflected the absence of PTSD according to the SCID assessment) with false positives (those who were service connected for PTSD despite the absence of PTSD according to the SCID assessment) on these same outcomes showed no differences on either outcome (Table 3). Analyses comparing true negatives with false negatives on these outcomes revealed that false negatives reported significantly greater functional impairment and PTSD symptoms than true negatives for both current and lifetime PTSD (Table 3). Finally, analyses comparing true positives with false positives on these variables and SC percentage revealed that true positives reported significantly greater functional impairment and more PTSD symptoms and received a greater PTSD SC percentage than false positives for both current and lifetime PTSD (Table 4).

Click figure to enlarge

Click figure to enlarge

As a post hoc analysis, we examined whether false negatives were service connected for a non-PTSD mental health condition. Slightly more than half of the false negatives for both current and lifetime PTSD were service connected for another mental health disorder (Table 5). These individuals did not significantly differ from either the false negatives with no other mental health disorder service connection or the true positives on either functional impairment severity or PTSD symptom count (all t values < 1.55; all P values > .11).

Click figure to enlarge

DISCUSSION

This is the first study to examine the extent to which veterans’ PTSD SC status matches with their PTSD diagnostic status at the time of an independent, semistructured diagnostic interview. We found that PTSD SC status was concordant with current and lifetime PTSD diagnostic status about 70% and 77% of the time, respectively. For current PTSD status, our results showed a slightly higher proportion of false positives—individuals who did not meet SCID criteria but who had a PTSD chart diagnosis—than false negatives—individuals who met SCID criteria but did not have a diagnosis in their chart. For lifetime PTSD, we observed the opposite pattern: the proportion of false negatives was approximately twice the proportion of false positives.

Although these results suggest that in most cases SC status is consistent with PTSD diagnostic status, the numbers of false positives and false negatives do raise the possibility that, consonant with concerns about the validity of VA PTSD-related disability rating decisions, PTSD C&P outcomes may be inconsistent for a significant minority of veterans.2,4-6 Regardless of whether we use current or lifetime PTSD status for comparative purposes, results indicate that we should be concerned about both the number of veterans who may have PTSD who are not service connected for PTSD (and are therefore denied the associated benefits including recognition that their disorder is military service related, access to free health care, and potential monetary compensation) and the number of veterans who are service connected for PTSD and receiving associated benefits when they may not have the disorder. Although questions and concerns about the latter have been discussed at great length, much less attention has been paid to the former even though research has shown that veterans receiving PTSD disability benefits report greater reductions in PTSD symptoms as well as less poverty and less homelessness than those who are denied these benefits.13 Because PTSD prevalence in this sample is considerably higher than in the overall VA population, the diagnostic accuracy results cannot help us determine whether current or lifetime PTSD status is most useful when making comparisons with PTSD SC status. However, because PTSD symptoms naturally wax and wane over time14 or remit or change as a function of other factors (eg, receiving treatment15 or disability benefits13), lifetime PTSD diagnostic status might be the more appropriate variable with which to make comparisons.

Other explanations for PTSD SC misclassification include insufficient knowledge or inadequate C&P examination practices, patient or institutional pressures, atypical symptom presentation, and examiner biases.16 It is also worth mentioning that PTSD SC status is not solely dependent upon the results of the C&P examination. The examination is only one piece of evidence used by the Veterans Benefits Administration to determine veterans’ SC status.

Interestingly, although approximately half of our false negatives are service connected for other mental health conditions, these participants did not differ from participants who were not service connected for another mental health disorder or the true positives on functional impairment severity or PTSD symptom count. These findings suggest that mental health SC is not a proxy for symptom severity and that service connection for another mental disorder cannot explain why the false negatives were not given SC for PTSD.

Our categorization of veterans into true positives, true negatives, false positives, and false negatives was generally supported by our examination of group differences in functional impairment, PTSD symptom count, and percentage of SC received. Findings showing that PTSD false negatives are as impaired and almost as symptomatic as true positives and are more impaired and symptomatic than true negatives provide additional evidence that many false negatives are deserving of disability benefits. Findings showing that false positives are similar to true negatives, but different than true positives, similarly suggest that many false positives should not be receiving benefits.

Past studies have shown that a very small percentage of VA disability examiners regularly use standardized testing in their C&P examinations17 and that the quality and accuracy of PTSD disability examinations are greatly improved by incorporating evidence-based assessment methods.18 In concert with these findings, our results provide additional evidence that a thorough, multimethod assessment is necessary in PTSD C&P examinations to determine PTSD diagnosis and associated impairment level in a reliable and valid fashion. A multimethod approach combines data derived from various sources and takes advantage of each measure’s relative strengths, overcoming the psychometric limitations of any single instrument and maximizing correct diagnostic decisions.19

Study limitations include a restriction of our sample to combat-exposed veterans who had undergone a mental health assessment. Furthermore, participants were selected on the basis of 2 visits with a health care professional; this may have biased the sample by excluding veterans who used VA mental health services on only 1 occasion. Because our sample is not representative of all veterans who use VA health care, this study cannot establish the prevalence of invalid PTSD disability claims. It also cannot definitively explain why PTSD SC status may disagree with SCID outcomes. Strengths of this study include the use of a national sample with equal numbers of male and female veterans and the use of standard clinical diagnostic instruments in the assessment of PTSD.

Overall, our findings indicate that PTSD SC status and the results of a PTSD diagnostic interview are discordant for a significant minority of veterans. Although the number of veterans who are service connected without meeting criteria for PTSD is concerning, the current study revealed an equally concerning disparity: the substantial number of veterans who meet PTSD criteria but have not been granted SC status. Thorough, multimethod assessments may help to reduce this discordance and increase the likelihood that individuals who do not meet criteria for PTSD do not receive benefits, but also that disability benefits are granted to those individuals worthy of receiving them. Additional research is needed to explore the scope of this problem in other settings and with the new PTSD diagnostic criteria. Further, research is needed to examine the reasons for existing discrepancies as well as how to eliminate them.

Submitted: November 18, 2014; accepted March 12, 2015.

Online first: January 19, 2016.

Author contributions: Dr Marx had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Drs Bovin, Gallagher, Holowka, Keane, Marx, Rosen, Schnurr, and Szafranski and Ms Engel-Rebitzer. Acquisition of data: Drs Holowka, Keane, Marx, and Rosen and Ms Engel-Rebitzer. Analysis and interpretation of data: Drs Bovin, Gallagher, Marx, and Szafranski and Ms Engel-Rebitzer. Drafting of the manuscript: Drs Bovin, Gallagher, Holowka, Keane, Marx, Rosen, Szafranski, and Schnurr and Ms Engel-Rebitzer. Critical revision of the manuscript for important intellectual content: Drs Holowka, Keane, Marx, Rosen, and Schnurr. Statistical analysis: Drs Bovin, Gallagher, Marx, and Szafranski and Ms Engel-Rebitzer. Obtained funding: Drs Keane, Marx, and Rosen. Administrative, technical, or material support: Ms Engel-Rebitzer.

Potential conflicts of interest: There are no conflicts of interest to report.

Funding/support: This work was funded by US Department of Defense Awards W81XWH-08-2-0100 and W81XWH-08-2-0102.

Role of the sponsor: The funding organization had no role in the design or conduct of the study; in the collection, management, analysis, and interpretation of the data; or in preparation, review, or approval of the manuscript.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the Department of Defense or Department of Veterans Affairs.

Previous presentation: Data included in this manuscript were presented at the 30th Annual Meeting of the International Society for Traumatic Stress Studies, November 6-8, 2014, Miami, Florida.

REFERENCES

1. Department of Veterans Affairs. Compensation. Washington, DC: Department of Veterans Affairs; 2013.

2. Veterans Benefits Administration. Annual Benefits Report. Washington, DC: Veterans Benefits Administration; 2013.

3. Department of Veterans Affairs Office of the Inspector General. Review of State Variances in VA Disability Compensation Payments. Washington, DC: VA Office of Inspector General; 2005.

4. McNally RJ, Frueh BC. Why we should worry about malingering in the VA system: comment on Jackson et al (2011). J Trauma Stress. 2012;25(4):454-460. PubMed doi:10.1002/jts.21713

5. Frueh BC, Elhai JD, Grubaugh AL, et al. Documented combat exposure of US veterans seeking treatment for combat-related post-traumatic stress disorder. Br J Psychiatry. 2005;186(6):467-475. PubMed doi:10.1192/bjp.186.6.467

6. Marx BP, Jackson JC, Schnurr PP, et al. The reality of malingered PTSD among veterans: reply to McNally and Frueh (2012). J Trauma Stress. 2012;25(4):457-460. doi:10.1002/jts.21714

7. Rosen RC, Marx BP, Maserejian NN, et al. Project VALOR: design and methods of a longitudinal registry of post-traumatic stress disorder (PTSD) in combat-exposed veterans in the Afghanistan and Iraqi military theaters of operations. Int J Methods Psychiatr Res. 2012;21(1):5-16. PubMed doi:10.1002/mpr.355

8. First M, Spitzer R, Williams J, et al. Structured Clinical Interview for DSM-IV Axis 1 Disorders (SCID-I). Handbook of Psychiatric Measures. Washington, DC: American Psychiatric Association; 2000:49-53.

9. Bovin MJ, Weathers F. Assessing PTSD Symptoms. The Oxford Handbook of Traumatic Stress Disorders. New York, NY: Oxford University Press; 2012:235.

10. Marx BP, Schnurr PP, Rodriguez P, et al. Development of a Functional Impairment Scale for Active Duty Service Members and Veterans. Symposium presented at the Annual Meeting of the International Society for Traumatic Stress Studies; November 4-7, 2009; Atlanta, GA.

11. Holowka DW, Marx BP. Assessing PTSD-Related Functional Impairment and Quality of Life. The Oxford Handbook of Traumatic Stress Disorders. New York, NY: Oxford University Press; 2012:315-330.

12. Lipsey M, Wilson D. Practical Meta-Analysis. Thousand Oaks, CA: SAGE Publications; 2001.

13. Murdoch M, Sayer NA, Spoont MR, et al. Long-term outcomes of disability benefits in US veterans with posttraumatic stress disorder. Arch Gen Psychiatry. 2011;68(10):1072-1080. doi:10.1001/archgenpsychiatry.2011.105 PubMed

14. Grieger TA, Cozza SJ, Ursano RJ, et al. Posttraumatic stress disorder and depression in battle-injured soldiers. Am J Psychiatry. 2006;163(10):1777-1783, quiz 1860. PubMed doi:10.1176/ajp.2006.163.10.1777

15. Foa EB, Keane TM, Friedman MJ, et al. Effective Treatments for PTSD: Practice Guidelines from the International Society for Traumatic Stress Studies. New York, NY: Guilford Press; 2008.

16. Norman GR, Eva KW. Diagnostic error and clinical reasoning. Med Educ. 2010;44(1):94-100. PubMed doi:10.1111/j.1365-2923.2009.03507.x

17. Jackson JC, Sinnott PL, Marx BP, et al. Variation in practices and attitudes of clinicians assessing PTSD-related disability among veterans. J Trauma Stress. 2011;24(5):609-613. PubMed doi:10.1002/jts.20688

18. Speroff T, Sinnott PL, Marx B, et al. Impact of evidence-based standardized assessment on the disability clinical interview for diagnosis of service-connected PTSD: a cluster-randomized trial. J Trauma Stress. 2012;25(6):607-615. PubMed doi:10.1002/jts.21759

19. Keane TM, Wolfe J, Taylor KL. Post-traumatic stress disorder: evidence for diagnostic validity and methods of psychological assessment. J Clin Psychol. 1987;43(1):32-43. PubMed doi:10.1002/1097-4679(198701)43:1<32::AID-JCLP2270430106>3.0.CO;2-X