Advertisement

Retrospective identification of infection in the emergency department: A significant challenge in sepsis clinical trials

  • Catherine S. O'Neal
    Correspondence
    Corresponding author at: Catherine S. O'Neal, Louisiana State University Health Sciences Center, Internal Medicine Residency Program, 5246 Brittany Dr., Baton Rouge, LA 70808, USA.
    Affiliations
    Louisiana State University Health Sciences Center, Internal Medicine Residency Program, Baton Rouge, LA, USA

    Our Lady of the Lake Regional Medical Center, Baton Rouge, LA, USA
    Search for articles by this author
  • Diana Hamer
    Affiliations
    Louisiana State University Health Sciences Center, Internal Medicine Residency Program, Baton Rouge, LA, USA

    Our Lady of the Lake Regional Medical Center, Baton Rouge, LA, USA
    Search for articles by this author
  • Mandi W. Musso
    Affiliations
    Our Lady of the Lake Regional Medical Center, Baton Rouge, LA, USA

    Louisiana State University Health Sciences Center, Emergency Medicine Residency Program, Baton Rouge, LA, USA
    Search for articles by this author
  • Terrell S. Caffery
    Affiliations
    Our Lady of the Lake Regional Medical Center, Baton Rouge, LA, USA

    Louisiana State University Health Sciences Center, Emergency Medicine Residency Program, Baton Rouge, LA, USA
    Search for articles by this author
  • Morgan K. Walker
    Affiliations
    National Institutes of Health, Bethesda, MD, USA
    Search for articles by this author
  • Katherine W. Lavie
    Affiliations
    Louisiana State University Health Sciences Center, Internal Medicine Residency Program, Baton Rouge, LA, USA

    Our Lady of the Lake Regional Medical Center, Baton Rouge, LA, USA
    Search for articles by this author
  • Matthew S. Berlinger
    Affiliations
    Department of Medicine, Johns Hopkins University, Baltimore, MD, USA
    Search for articles by this author
  • Christopher B. Thomas
    Affiliations
    Louisiana State University Health Sciences Center, Internal Medicine Residency Program, Baton Rouge, LA, USA

    Our Lady of the Lake Regional Medical Center, Baton Rouge, LA, USA

    Baton Rouge General Medical Center, Baton Rouge, LA, USA
    Search for articles by this author
  • Shannon M. Alwood
    Affiliations
    Our Lady of the Lake Regional Medical Center, Baton Rouge, LA, USA

    Louisiana State University Health Sciences Center, Emergency Medicine Residency Program, Baton Rouge, LA, USA
    Search for articles by this author
  • Tonya Jagneaux
    Affiliations
    Louisiana State University Health Sciences Center, Internal Medicine Residency Program, Baton Rouge, LA, USA

    Our Lady of the Lake Regional Medical Center, Baton Rouge, LA, USA

    Baton Rouge General Medical Center, Baton Rouge, LA, USA
    Search for articles by this author
  • Michael A. Sanchez
    Affiliations
    Louisiana State University Health Sciences Center, Internal Medicine Residency Program, Baton Rouge, LA, USA

    Baton Rouge General Medical Center, Baton Rouge, LA, USA
    Search for articles by this author
  • Hollis R. O'Neal
    Affiliations
    Louisiana State University Health Sciences Center, Internal Medicine Residency Program, Baton Rouge, LA, USA

    Our Lady of the Lake Regional Medical Center, Baton Rouge, LA, USA

    Baton Rouge General Medical Center, Baton Rouge, LA, USA
    Search for articles by this author
Open AccessPublished:March 13, 2022DOI:https://doi.org/10.1016/j.amjms.2022.02.008

      Abstract

      Background

      This study examined three methods for retrospectively identifying infection in emergency department (ED) patients: modified objective definitions of infection (MODI) from the CDC/NHSN, physician adjudication determination of infection, and ED treating physician behavior.

      Methods

      This study used a subset of data from a prospective sepsis trial. We used Fleiss's Kappa to compare agreement between two physicians retrospectively adjudicating infection based on the patient's medical record, modified infection definition from the CDC/NHSN, and ED treating physician behavior.

      Results

      Overall, there was similar agreement between physician adjudication of infection and MODI criteria (Kappa=0.59) compared to having two physicians independently identify infection through retrospective chart review (Kappa=0.58). ED treating physician behavior was a poorer proxy for infection when compared to the MODI criteria (0.41) and physician adjudication (Kappa = 0.50).

      Conclusions

      Retrospective identification of infection poses a significant challenge in sepsis clinical trials. Using modified definitions of infection provides a standardized, less time consuming, and equally effective means of identifying infection compared to having multiple physicians adjudicate a patient's chart.

      Key Indexing Terms

      Introduction

      In each iteration of the consensus definitions of sepsis, infection is a requisite condition.
      • Bone R.C.
      • Sprung C.L.
      • Sibbald WJ.
      Definitions for sepsis and organ failure.
      • Singer M.
      • Deutschman C.S.
      • Seymour C.W.
      • et al.
      The third international consensus definitions for sepsis and septic shock (Sepsis-3).
      • Levy M.M.
      • Fink M.P.
      • Marshall J.C.
      • et al.
      2001 SCCM/ESICM/ACCP/ATS/SIS international sepsis definitions conference.
      Unfortunately, there are no universally accepted definitions of infection and no reliable mechanism to differentiate patients with infectious versus noninfectious etiologies. Cultures are, at times, insensitive, and at other times, nonspecific, leading to a subjective clinical diagnosis. This subjectivity is underscored in the validation study of the new sepsis definition
      • Seymour C.W.
      • Liu V.X.
      • Iwashyna T.J.
      • et al.
      Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (Sepsis-3).
      that used physician behavior, defined as the combination of orders for body fluid cultures and antimicrobials, as a proxy for suspected infection. Many sepsis clinical trials use retrospective physician adjudication of the patients’ medical records to identify infection, a costly and time-consuming task with inherent subjectivity.
      Since 1998, the National Healthcare Safety Network (NHSN) and the CDC have provided annually updated criteria for common infections for surveillance of hospital-acquired infections.
      • Horan T.C.
      • Andrus M.
      • Dudeck M.A.
      CDC/NHSN surveillance definition of health care-associated infection and criteria for specific types of infections in the acute care setting.
      • Garner J.S.
      • Jarvis W.R.
      • Emori T.G.
      • Horan T.C.
      • Hughes J.M.
      Cdc definitions for nosocomial infections, 1988.

      Centers for Disease Contol and Prevention (CDC). CDC/NHSN surveillance definitions for specific types of infections. 2017; Available at: https://www.cdc.gov/nhsn/pdfs/pscmanual/17pscnosinfdef_current.pdf. Accessed March 20, 2022.

      However, there is no literature regarding the application of these definitions for the retrospective identification of infection in emergency department (ED) patients for clinical trials research. We modified the NHSN definitions for infection to include only the information typically available for these infection types at the time of an ED visit and within the first 48 hours of admission with the intention to provide a standardized approach that may be used by all healthcare facilities to determine the presence of an infection.
      The purpose of the current study is to determine whether modified objective definitions of infection (MODI) from the CDC/NHSN

      Centers for Disease Contol and Prevention (CDC). CDC/NHSN surveillance definitions for specific types of infections. 2017; Available at: https://www.cdc.gov/nhsn/pdfs/pscmanual/17pscnosinfdef_current.pdf. Accessed March 20, 2022.

      performed comparably to ED treating physician behavior (e.g., combination of culture orders and ED antibiotic administration) and physician adjudication of the patient record. We hypothesized that there would be low concordance rates between physicians’ identification of infection and that MODI would provide a standardized means of identifying infected patients.

      Methods

      Patient population

      This is a subset of data comparing the MODI criteria to physician adjudication and ED treating physician behavior. This data was retrospectively analyzed from a prospective, observational cohort of high acuity patients with suspected or confirmed infection. Data from this trial have previously been published.
      • Guillou L.
      • Sheybani R.
      • Jensen A.E.
      • et al.
      Development and validation of a cellular host response test as an early diagnostic for sepsis.
      The study was performed in the EDs of two academic medical centers between February and December 2016. Inclusion criteria for the prospective, observational trial were adults who provided informed consent with signs of infection defined as two of four SIRS criteria and at least one indication of organ dysfunction (e.g., elevated serum lactate (> 2mmol/L), hypoxia, hypotension, acute kidney injury, elevated total bilirubin, decreased platelet count, or elevated INR). Exclusion criteria were death expected within 24 hours, unable or unwilling to consent, hematologic malignancy or myelodysplastic, myeloproliferative disorder, or neutropenia. Data from all of the patients enrolled in prospective trial was retrospectively analyzed for the purposes of the current study. Both institutions’ Institutional Review Boards approved this study.

      Data abstraction

      Data abstracted from the medical record included demographic, physiologic, radiologic, laboratory, diagnostic, and discharge information as well as physician's orders for cultures and medication administered from enrollment to the third hospital day. Patients were identified as having physician suspected infection if the ED physician ordered cultures and antibiotics in the ED.

      Adjudication

      Two physicians from an adjudication committee retrospectively adjudicated patient charts through independent review of each patient's medical record to determine the presence of infection. If both adjudicators agreed, their answer was used as the final adjudication. In cases of disagreement, a third physician served as a tiebreaker with the majority decision, or consensus, determining the endpoint. Cases in which no physicians agreed were labeled as non-consensus. Whether the individual physician determined that the patient was infected or not, they were required to specify the organ system involved in the infection, insult or injury, respectively.

      Modified objective definitions of Infection (MODI) from the CDC/NHSN

      We derived the criteria for MODI from the CDC/NHSN surveillance definitions and criteria,
      • Garner J.S.
      • Jarvis W.R.
      • Emori T.G.
      • Horan T.C.
      • Hughes J.M.
      Cdc definitions for nosocomial infections, 1988.
      ,

      Centers for Disease Contol and Prevention (CDC). CDC/NHSN surveillance definitions for specific types of infections. 2017; Available at: https://www.cdc.gov/nhsn/pdfs/pscmanual/17pscnosinfdef_current.pdf. Accessed March 20, 2022.

      specifically, the Tennessee Department of Health's implementation of these definitions through checklists.

      Tennessee Department of Health. About healthcare associated infections (HAI). 2017; Available at: https://tn.gov/health/cedep/hai.html. Accessed March 20, 2022.

      Modifications included removing specific window-periods and requirements of hospital stays and interventions, so the criteria included in the MODI would reflect the data available during an ED visit, and, if the patient was hospitalized, the data collected during the first 48 hours of hospital admission. Additionally, we removed certain criteria specific to physician behavior, such as treatment for infection with antibiotics. We included only the most common sites of infection: respiratory, gastrointestinal (GI), urinary, cardiovascular, central nervous system (CNS), skin and soft tissue, and bone and joint in the current study. Authors created forms for abstracting MODI data for each of the major sites (online supplement 1). The definitions and modification for each site of infection are available in the online supplement 2. Finally, these criteria were operationalized in REDCap,
      • Harris P.A.
      • Taylor R.
      • Minor B.L.
      • et al.
      The REDCap consortium: building an international community of software platform partners.
      so that coordinators entered data directly into the electronic data capture system.
      Trained research coordinators retrospectively abstracted MODI criteria using patients’ medical record. Data pertaining to the ED intake history and physical and review of symptoms was abstracted only for the duration of the ED encounter. All other data was collected for the first 48 hours from study enrollment in accordance to the MODI. Adjudicators were blinded to the MODI, and coordinators were blinded to adjudicators’ decisions.

      Statistical analyses

      Chi-square analyses were used to compare rates of agreement. Variation in diagnosing the presence or absence of infection was quantified using Fleiss’ Kappa (Fleiss et al 2003). Kappa values greater than .75 indicated strong agreement, between .40-.75 indicated fair to good agreement, and values less than .40 indicated poor agreement. We examined agreement between physician adjudicators’ diagnoses of infected, not infected, and indeterminate. Next, we included all indeterminate cases in the infected category in order to assess agreement between final physician adjudication (infected and not infected) and MODI (infected and not infected). Finally, we compared both retrospective means of identifying infection to ED treating physician behavior (e.g., orders for body fluid cultures and antibiotics). Analyses were run in SPSS Version 26.

      Results

      Three hundred and ninety-three subjects met inclusion criteria. (Table 1). The two initial adjudicators agreed in 301 (76.6%) of cases, indicating fair to good agreement, with Fleiss Kappa = 0.59. The 92 (23.4% of total) remaining cases required a third adjudicator, and in 71 of these cases, consensus was reached, with two of three adjudicators in agreement. The 21 remaining cases, in which one physician chose infected, one chose not infected, and the third chose indeterminate, were ultimately classified as infected as per the study protocol. In all, there were 173 patients adjudicated as infected and 220 patients adjudicated as not infected (Fig. 1).
      Table 1Patient demographic information and infection classification.
      Physician AdjudicationMODI ClassificationPhysician Behavior
      InfectedNot InfectedInfectedNot infectedCx - onlyAbx - onlyCx & Abx
      N1732201622317221187
      Age, m (SD)62.8 (18.6)62.5 (17.6)65.0 (17.4)61.0 (18.3)60.8 (18.5)62.4 (14.9)64.1 (17.5)
      Gender, n (%)
       Male87 (50.3)111 (50.5)82 (50.6)116 (50.2)39 (54.9)12 (57.1)89 (47.8)
       Female86 (49.7)109 (49.5)80 (49.4)115 (49.8)32 (45.1)9 (42.9)97 (52.2)
       Race, n (%)
       White100 (59.5)115 (52.3)91 (56.2)124 (53.7)39 (54.2)10 (47.6)107 (57.2)
       Black68 (40.5)100 (45.5)69 (42.6)99 (42.9)31 (43.1)9 (42.9)76 (40.6)
       Other5 (2.9)5 (2.3)2 (1.2)8 (3.5)2 (2.8)2 (9.5)4 (2.1)
      Physician Adjudication, n (%)
       Infected--127 (78.4)46 (19.9)21 (29.2)8 (38.1)132 (70.6)
       Not infected--35 (21.6)185 (80.1)51 (70.8)13 (61.9)55 (29.4)
       MODI Classification, n (%)
       Infected----23 (31.9)7 (33.3)117 (62.6)
       Not Infected----49 (68.1)14 (66.7)70 (37.4)
      Source of involvement, n (%)
       Bone & Joint8 (4.6)4 (1.8)6 (3.7)
       CNS1 (0.6)11 (5)0 (-)
       Gastrointestinal27 (15.6)33 (15)19 (11.7)
       Genitourinary27 (15.6)7 (3.1)18 (11.1)
       Primary Blood11 (6.3)5 (2.2)17 (10.5)
       Respiratory56 (32.4)51 (23.2)87 (53.7)
       Skin-Skin / Wound21 (12.1)0 (-)15 (9.3)
       Multiple Sources19 (26.0)45 (20.4)0 (-)
       Other2 (1.2)64 (29)0 (-)
      Note: Cx only = Culture Orders Only; Abx Only = Antibiotic Orders Only; Cx & Abx = Culture and Antibiotic Orders.
      Fig. 1
      Fig. 1Agreement between physicians using physician adjudication.
      Of the 393 patients included, 162 (41.2%) met MODI classification of infection. Fleiss's Kappa indicated fair to good agreement between the MODI and final physician adjudication (Kappa = 0.58; PA = 79.4%). There were 35 patients identified as infected by MODI criteria but adjudicated as not infected by physician adjudication. Of those, 32 met MODI criteria for a Respiratory infection and three for other sources for infection. Of the 46 patients adjudicated as infected by physicians but not classified as infected by MODI, source of infection listed by physician consensus was Respiratory for 14 cases, GI for 10, Skin for 10, 9 other cases had other sources listed. Three cases did not have a physician consensus for source of infection.
      We compared physician adjudication to treating ED physician behavior and MODI classification to ED physician behavior. Agreement was fair to good (PA = 75.6%, Kappa = 0.50) when ED physician behavior was compared to physician adjudication. When MODI classification and treating ED physician behavior were compared, PA and kappa were slightly lower (70.7%, 0.41).

      Discussion

      The primary objective of the current investigation was to compare three methods of identifying infection for sepsis clinical trials: modified CDC/NHSN definitions of infection (MODI) for identification of infection, physician adjudication, and ED treating physician behavior. The current study highlights the complexity of diagnosing infection both in real time (ED treating physician behavior) and retrospectively (using the entire medical record). Over one in five patients required tiebreak. Further, three physicians could not reach consensus about the presence of infection in 21 cases. These findings are consistent with the current literature suggesting that disagreement between physicians on the diagnosis of infection or identification of the source of infection is high.
      • Klein Klouwenberg P.M.
      • Ong D.S.
      • Bos L.D.
      • et al.
      Interobserver agreement of centers for disease control and prevention criteria for classifying infections in critically ill patients.
      ,
      • Lopansri B.K.
      • Miller Iii R.R.
      • Burke J.P.
      • et al.
      Physician agreement on the diagnosis of sepsis in the intensive care unit: estimation of concordance and analysis of underlying factors in a multicenter cohort.
      MODI definitions are based on NHSN/CDC definitions of infection used widely and in a standardized fashion by infection preventionists to identify hospital acquired infections. Benefits of using a modified NHSN definition include that they are easily adopted across different facilities and can be followed by both clinical and data teams for abstraction. Unfortunately for identifying sepsis on presentation, some of the definitions require modification as not all clinical data is accounted for in the first 48 hours of the hospital visit. NHSN definitions also sometimes include physician behavior as part of definition. For the purpose of adjudication, physician behavior was removed so that criteria for infection included objective data. The MODI criteria for infection demonstrated comparable rates of concordance with final physician adjudication. The MODI criteria offer a standardized means of diagnosing infection for clinical trials that removes variability inherent in physician adjudication and saves physicians significant time and effort. The MODI criteria may also provide an objective and standardized means for the comparison of patient populations from different sites of a multi-centered study.
      One of the Infectious Diseases Society of America (IDSA's) primary challenges of the Sepsis-3 guidelines was the Task Force's failure to acknowledge the inherent difficulty in identifying infection and differentiating sepsis from non-infectious syndromes.
      • Force I.S.T.
      Infectious diseases society of America (IDSA) POSITION STATEMENT: why IDSA did not endorse the surviving sepsis campaign guidelines.
      Infection was defined using a cohort consisting of patients who received a combination of blood cultures and antibiotics.
      • Seymour C.W.
      • Liu V.X.
      • Iwashyna T.J.
      • et al.
      Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (Sepsis-3).
      The results of our current study demonstrate that treating ED physician behavior had a slightly lower percent agreement and kappa compared to MODI classification and physician adjudication.
      Limitations include the lack of a gold standard for diagnosing sepsis or infection. As a result of the variability among physician adjudicators, it may be difficult to assess the exact utility of the MODI criteria. Since we relied on retrospective review of clinical data in the medical record, lack of specific documentation and key words may lead to misclassification. In addition, there is some degree of inter-relation among the three standards, as adjudicators and study personnel reviewing for MODI criteria were bound by data obtained during routine clinical treatment of individual cases. Finally, the data was collected at two institutions in a large metropolitan area in the Southeast United States, and both sites were adjudicated by the same research team. Additional studies will be needed to assess the generalizability of these results.
      In summary, the current data validates the IDSA's concerns regarding the variability in identifying patients who are infected from those with noninfectious etiologies. Using MODI criteria was as good as other means of classifying objectively and has some benefits such as standardizing the definition of infection by removing some of the subjectivity inherent in clinical decision-making, as well as relieving physicians of the time and effort required to make similar decisions through retrospective assessments. Finally, these criteria are easily implemented in an electronic data capturing system, improving efficiency and making remote assessment for infection possible. The MODI criteria offer an alternative to using physician adjudication in sepsis clinical trials by standardizing definitions of infection and allowing for an objective comparison of subjects across sites.

      Autor Contributions

      CO and HO developed the idea for creating modified definitions of infection. MM and DH were primarily responsible for statistical analyses. All authors were responsible for making intellectual contributions regarding refining the definitions, collecting data, and editing the manuscript.

      Source of funding

      This is a subset of data from a prospective, observational cohort that was funded by CytoVale, Inc, San Francisco, CA.

      Declaration of Competing Interest

      The authors have no conflicts of interest to declare.

      Acknowledgements

      The authors would also like to express their gratitude to the Office of Research at Our Lady of the Lake Regional Medical Center and the personnel who worked tirelessly to screen and enroll patients: Jennifer Daigle, BSN; Brittany Rishell, BSN; Katie Vance, PhD; JoAnne Hollingsworth, BSN, Allie Piehet, BS, and Emily Perdue, RN. Lastly, the authors would like to recognize the contributions of the team at Cytovale for their work in the set-up and completion of the prospective study from which the data from this project was collected: Roya Sheybani, Henry Tse, Ajay Shah, Anne Jensen, Audun Johnson, Bliss Lambert, Chris Dahlberg, Hon Seng, Kylie Ortego, Lionel Guillou, Maddy Hem and Nicholas Martinez.

      Appendix. SUPPLEMENTARY MATERIALS

      References

        • Bone R.C.
        • Sprung C.L.
        • Sibbald WJ.
        Definitions for sepsis and organ failure.
        Crit Care Med. 1992; 20: 724-726
        • Singer M.
        • Deutschman C.S.
        • Seymour C.W.
        • et al.
        The third international consensus definitions for sepsis and septic shock (Sepsis-3).
        JAMA. 2016; 315: 801-810
        • Levy M.M.
        • Fink M.P.
        • Marshall J.C.
        • et al.
        2001 SCCM/ESICM/ACCP/ATS/SIS international sepsis definitions conference.
        Crit Care Med. 2003; 31: 1250-1256
        • Seymour C.W.
        • Liu V.X.
        • Iwashyna T.J.
        • et al.
        Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (Sepsis-3).
        JAMA. 2016; 315: 762-774
        • Horan T.C.
        • Andrus M.
        • Dudeck M.A.
        CDC/NHSN surveillance definition of health care-associated infection and criteria for specific types of infections in the acute care setting.
        Am J Infect Control. 2008; 36: 309-332
        • Garner J.S.
        • Jarvis W.R.
        • Emori T.G.
        • Horan T.C.
        • Hughes J.M.
        Cdc definitions for nosocomial infections, 1988.
        Am J Infection Control. 1988; 16: 128-140
      1. Centers for Disease Contol and Prevention (CDC). CDC/NHSN surveillance definitions for specific types of infections. 2017; Available at: https://www.cdc.gov/nhsn/pdfs/pscmanual/17pscnosinfdef_current.pdf. Accessed March 20, 2022.

        • Guillou L.
        • Sheybani R.
        • Jensen A.E.
        • et al.
        Development and validation of a cellular host response test as an early diagnostic for sepsis.
        PLoS One. 2021; 16e0246980
      2. Tennessee Department of Health. About healthcare associated infections (HAI). 2017; Available at: https://tn.gov/health/cedep/hai.html. Accessed March 20, 2022.

        • Harris P.A.
        • Taylor R.
        • Minor B.L.
        • et al.
        The REDCap consortium: building an international community of software platform partners.
        J Biomed Inform. 2019; 95103208
        • Klein Klouwenberg P.M.
        • Ong D.S.
        • Bos L.D.
        • et al.
        Interobserver agreement of centers for disease control and prevention criteria for classifying infections in critically ill patients.
        Crit Care Med. 2013; 41: 2373-2378
        • Lopansri B.K.
        • Miller Iii R.R.
        • Burke J.P.
        • et al.
        Physician agreement on the diagnosis of sepsis in the intensive care unit: estimation of concordance and analysis of underlying factors in a multicenter cohort.
        J Intensive Care. 2019; 7: 13
        • Force I.S.T.
        Infectious diseases society of America (IDSA) POSITION STATEMENT: why IDSA did not endorse the surviving sepsis campaign guidelines.
        Clin Infect Dis. 2018; 66: 1631-1635