Development of quality indicators for antimicrobial treatment in adults with sepsis
BMC Infectious Diseases volume 14, Article number: 345 (2014)
Outcomes in patients with sepsis are better if initial empirical antimicrobial use is appropriate. Several studies have shown that adherence to guidelines dictating appropriate antimicrobial use positively influences clinical outcome, shortens length of hospital stay and contributes to the containment of antibiotic resistance.
Quality indicators (QIs) can be systematically developed from these guidelines to define and measure appropriate antimicrobial use. We describe the development of a concise set of QIs to assess the appropriateness of antimicrobial use in adult patients with sepsis on a general medical ward or Intensive Care Unit (ICU).
A RAND-modified, five step Delphi procedure was used. A multidisciplinary panel of 14 experts appraised and prioritized 40 key recommendations from within the Dutch national guideline on antimicrobial use for adult hospitalized patients with sepsis (http://www.swab.nl/guidelines). A procedure to select QIs relevant to clinical outcome, antimicrobial resistance and costs was performed using two rounds of questionnaires with a face-to-face consensus meeting between the rounds over a period of three months.
The procedure resulted in the selection of a final set of five QIs, namely: obtain cultures; prescribe empirical antimicrobial therapy according to the national guideline; start intravenous drug therapy; start antimicrobial treatment within one hour; and streamline antimicrobial therapy.
This systematic, stepwise method, which combined evidence and expert opinion, led to a concise and therefore feasible set of QIs for optimal antimicrobial use in hospitalized adult patients with sepsis. The next step will entail subjecting these quality indicators to an applicability test for their clinimetric properties and ultimately, using these QIs in quality-improvement projects. This information is crucial for antimicrobial stewardship teams to help set priorities and to focus improvement.
Severe sepsis and septic shock are a substantial burden to health care, affecting millions of patients around the world each year . The average cost of care for a patient with severe sepsis is about 22.000 USD . It is often thought that severe sepsis is primarily seen on the intensive care unit (ICU). However, studies point out that the majority (50 – 68%) of patients with severe sepsis are admitted to a general medical ward , with a mortality rate around 26 – 29.5% [4, 5]. Since the diagnosis ‘severe sepsis’ is often poorly documented in the medical records by the treating clinicians, specific sepsis-targeted measures may not have been performed and antimicrobial use may have been inappropriate .
As the necessary first step in the improvement of appropriate use in patients with sepsis, guidelines have been developed that describe appropriate antimicrobial use in patients with sepsis admitted to a general medical ward or an Intensive Care Unit (ICU). Despite the availability of these guidelines, antimicrobials are used inappropriately: several studies show that inappropriate initial antimicrobial use in patients with severe sepsis or septic shock is associated with a reduction in survival [6–9]. As a second important step towards change and improvement of daily clinical care, the guideline-based development of quality indicators has been suggested [10, 11]. Quality indicators (QIs) are measurable elements that can be used to gain insight into the appropriateness of the given antimicrobial treatment, which is important to set priorities and to focus improvement. The aim of our study was to develop a concise and therefore feasible set of QIs to measure and monitor the appropriateness of antimicrobial use in adults with sepsis admitted to a general medical ward and/or ICU.
The Dutch Working Party on Antibiotic Policy (SWAB) publishes evidence-based guidelines for antimicrobial use. We used the guideline for antimicrobial use in hospitalized patients with sepsis (published online in 2010) as a starting point for the development of a set of QIs . This guideline covers antimicrobial use in all hospitalized adult patients with sepsis, except antimicrobial use in sepsis associated with indwelling intravascular devices that are not removed (tunnelled catheter or totally implantable vascular access devices) and therefore requires different therapy.
We used a systematic approach -the RAND-modified Delphi method [13, 14]- to develop a set of QIs in order to measure the appropriateness of antimicrobial treatment in adult patients with sepsis admitted to a general ward and/or ICU (Figure 1). For developing QIs by means of the Delphi method medical ethical approval was not required.
Extraction of guideline recommendations
One infectious diseases physician and one quality-of-care specialist independently extracted key recommendations from the national guideline for antimicrobial use in hospitalized patients with sepsis. Disagreements were resolved by consensus. The selected recommendations were translated into potential QIs and each indicator was graded to determine its scientific soundness (level of evidence), using Tables 1 and 2.
First questionnaire round
The list of the potentially relevant QIs was converted into a written questionnaire and used for the RAND-modified Delphi method to achieve expert consensus on these QIs. The consensus procedure was performed between February and April 2011. All authors of the above mentioned national guideline, and an additional intensive care specialist and a hospital pharmacist were approached, and they all agreed to participate in the multidisciplinary expert panel. Our final expert panel consisted of four infectious diseases physicians (all working primarily outside the ICU), two medical microbiologists, two hospital pharmacists, three intensive care specialists, two haematologists and one general surgeon (14 experts).
The questionnaire was sent by email to the experts, asking them to rate the QIs using the following criteria:
The potential QI leads to health gain for the patient, less bacterial resistance or promotes efficiency of care;
The potential QI is generalizable to all adult patients treated for sepsis with antimicrobial use;
There is enough scientific evidence or expert consensus to justify the recommended care.
The expert panel was asked to rate the potential QIs using a 9-point Likert scale (with 1 denoting “definitely not appropriate care” and 9 denoting “definitely appropriate care”). The answer category ‘cannot assess’ was also available. The panel members (experts) were asked to add suggestions and comments regarding the potential QIs, and also to add additional potential QIs or topics for consideration.
The results from the first questionnaire were analysed using a standardized Microsoft Office Access-based consensus tool. Potential QIs rated with an overall median score of 8 or 9 without disagreement were considered to be face valid and reliable , and were accepted as preliminary indicators. Disagreement was defined as the case in which less than 70% of the scores were in the top tertile (scores 7, 8, or 9) . If there was disagreement and the median score was below 8, the potential indicator was rejected and not discussed during the consensus meeting. QIs with a median score of 7 without disagreement or a median score of 8 or 9 with disagreement were discussed during the consensus meeting.
Expert panel meeting
All panel members were invited for a consensus meeting during which an overview of the first round ratings was provided. Goals of the consensus meeting were to achieve consensus on the QIs with a median score of 7 without disagreement or a median score of 8 or 9 with disagreement, and to rephrase accepted indicators using the comments from the panel in the first round of questionnaires. These QIs and the suggested new QIs by the panel members were discussed and reformulated when necessary.
Second questionnaire round, ranking procedure
After the consensus meeting, all discussed, reformulated and added potential indicators were included in a second questionnaire. First, the panel was asked whether they agreed (yes or no) with the proposed indicators and their definitions. Redefined indicators were accepted if at least 70% of the experts agreed with the new formulation. Second, the panel was asked to prioritize the potential indicators by selecting the ‘top 5’ of most important indicators. For each number-one ranking by a panel member, we granted a potential QI five points, for each number-two ranking, we granted four points and so on. QIs receiving more than 15% of the maximum possible ranking points were considered to be the most relevant indicators.
Extraction of guideline recommendations
Key recommendations were extracted from the national guideline independently by one infectious diseases physician (S.E.G.) and one quality-of-care specialist (M.E.L.J.H.). In consensus, 40 key recommendations were extracted from the national guideline and translated into potential QIs.Figure 1 shows the entire Delphi method as performed in the next steps.
First questionnaire round
The 40 potential QIs were scored by 12 of the 14 panel members during the first questionnaire round (86% response rate). Twenty-two potential QIs had a high score (8 or 9) without disagreement and were accepted, and nine potential indicators were rejected, see Figure 1 and Table 3. The panel did not agree on nine potential QIs, they either had a median score of 7 without disagreement or a median score of 8 or 9 with disagreement. One new potential indicator regarding adapting the antimicrobial dose to renal function was proposed by one panel member. Results are shown in Table 3.
Expert panel meeting
Six panel members (four infectious diseases specialists, one medical microbiologist and one intensive care specialist) were present during the consensus meeting (43%). The 22 accepted QIs with comments from the panel, the nine potential QIs with disagreement and the new indicator were discussed during the meeting.
From these 32 QIs, this smaller panel accepted 13 potential QIs; two QIs were rephrased and one new potential QI was added. The other 17 QIs were rejected and merged into four new composite QIs. These potential QIs had a more generic formulation and contained: starting empirical antimicrobial therapy; duration of therapy; changing empirical therapy to pathogen-directed therapy; and harmonizing local guidelines with the national guideline (indicator 42 – 45, Table 3). This resulted in a total set of 20 potential QIs.
Second questionnaire round, ranking procedure
After the consensus meeting, the resulting 20 potential indicators were sent to the entire panel for comments and approval. This questionnaire was scored by 13 of the 14 panel members (93% response rate). All 20 indicators were accepted, because in all cases 70% or more of the panel members agreed with the new content and rephrasing. Indicator 42 and 43 (local guidelines should correspond to the national guideline and prescribe according to the national guideline) were merged into one indicator because four out of the 13 panel members (31%) found them to be overlapping indicators. In the same questionnaire the panel members were also asked to rank the QIs. Out of this set of 19 indicators, five indicators received more than 15% of the maximum possible ranking points and were prioritized. They were found to be the most important QIs for antimicrobial care in adult patients with sepsis. The results of the second questionnaire are shown in Table 3 and Table 4 shows the final set of QIs. For the attendance list of the participation of the Delphi procedure and the development of the sepsis guideline, see Additional file 1: Table S1.
This systematic, stepwise method combining evidence and expert opinion, generated a valid, concise and therefore feasible set of five QIs to measure appropriate antimicrobial use in adult patients with sepsis admitted to general medical wards and/or ICUs. One QI specifically applies to patients with severe sepsis or septic shock (Table 4, indicator 2).
The issue of local resistance patterns and national versus local guideline recommendations for empirical treatment choices was extensively discussed during the Delphi consensus meeting. The national sepsis guideline underlines that hospitals can and should deviate from the recommendations based on local resistance patterns. We therefore favored to follow the national guidelines, but to guarantee the generalizability of the QIs another QI was added during the consensus meeting; local guidelines should correspond to the national guideline, but should deviate based on local resistance patterns (QI number 42).
In a systematic review performed by McGregor et al. empiric or definitive antibiotic therapy was considered to be appropriate if the regimen exhibited in vitro activity against the isolated pathogen(s) . We derived our key recommendations for appropriate antibiotic therapy from the national, evidence-based guideline for antimicrobial use in hospitalized patients with sepsis. This, according to the national and international experts, implies more than correct (empirical or definitive) antibiotic therapy alone: it defines correct antimicrobial use at patient level along the entire antibiotic pathway, from start (including appropriate diagnostics) to streamlining and discontinuing of antimicrobial therapy.
The final set consists of independent QIs, that can be used to provide insight into the appropriateness of current antimicrobial use, to identify where there is room for improvement . In a recent paper by our group  we found that patients with urinary tract infections who in particular adherence to the total set of QIs, showed a significant dose–response relationship with a shorter length of hospital stay. This argues for application of the QIs in a bundle approach.
This is the first study that specifically describes the development of QIs for the entire antimicrobial treatment of sepsis patients, also outside the ICU, via the modified Delphi technique. Several studies have described the use of quality measures for antimicrobial sepsis treatment [20–25]. However, some indicators were not systematically developed using a Delphi method , and some only focused on optimal sepsis care on the ICU [20, 24] or focused on the start of treatment (first 24 hours) and not on the entire clinical course [22, 23, 25].
The results of our study show resemblance with the concise Surviving Sepsis Campaign Care Bundle originated from the guideline [1, 26]. They also defined the optimal start of antimicrobial treatment and taking cultures as important parameters, only our panel members also defined streamlining as an important QI.
Our study has several strengths. We used the systematic modified Delphi method, a common and validated technique in which scientific evidence is combined with expert opinion [13, 14, 27, 28]. Boulkedid and colleagues recently reviewed its use and reporting, and formulated a practical guideline for using this RAND modified Delphi technique. Our procedure is consistent with their guideline . Our panel was multidisciplinary, with 14 experts from 6 different specialties. Furthermore, the response rate of the first and second questionnaire (86% and 93%) was high, which increases the validity of the results.
A limitation of this study is the national setting in which the QIs were developed, with a Dutch national expert panel and a Dutch guideline. This leads to the question of whether the results can be generalized to a wider international population. However, the guideline reviewed and graded the recent international literature and the QI development was performed by a multidisciplinary panel, in which several members have international experience and expertise on the topic.
Another potential limitation was the attendance at the expert panel meeting, which was 43%. However an extensive summary concerning the results from the consensus meeting was sent to all panel members, as they were asked to give their final remarks and approval for the added and rephrased potential QIs. Since 93% returned the second questionnaire, we believe that an incomplete attendance did not undermine the validity of the results.
We describe the complete and precise development of a concise set of quality indicators for optimal antimicrobial use in hospitalized adult sepsis patients, by means of the Delphi method. This paper can be used as manual for others, since transparency of healthcare becomes more important worldwide, and QIs give insight into the appropriateness of daily clinical care. At this moment, we are testing the applicability of this set of QIs in practice in 22 hospitals. After establishing their clinimetric properties we will analyze the association between adherence to the QIs and outcomes like duration of hospital stay. In the future, our guideline-based indicators can be used for national monitoring of antimicrobial use in hospitalized adults with sepsis or for quality improvement projects. This information is crucial for antimicrobial stewardship teams to help set priorities and to focus improvement.
Dutch working party on antibiotic policy
Intensive care unit
United States dollar.
Dellinger RP, Levy MM, Rhodes A, Annane D, Gerlach H, Opal SM, Sevransky JE, Sprung CL, Douglas IS, Jaeschke R, Osborn TM, Nunnally ME, Townsend SR, Reinhart K, Kleinpell RM, Angus DC, Deutschman CS, Machado FR, Rubenfeld GD, Webb SA, Beale RJ, Vincent JL, Moreno R, Surviving Sepsis Campaign Guidelines Committee including the Pediatric Subgroup: Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013, 41: 580-637.
Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR: Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med. 2001, 29: 1303-1310.
Rohde JM, Odden AJ, Bonham C, Kuhn L, Malani PN, Chen LM, Flanders SA, Iwashyna TJ: The epidemiology of acute organ system dysfunction from severe sepsis outside of the intensive care unit. J Hosp Med. 2013, 8: 243-247.
Esteban A, Frutos-Vivar F, Ferguson ND, Penuelas O, Lorente JA, Gordo F, Honrubia T, Algora A, Bustos A, Garcia G, Diaz-Regañón IR, de Luna RR: Sepsis incidence and outcome: contrasting the intensive care unit with the hospital ward. Crit Care Med. 2007, 35: 1284-1289.
Sundararajan V, Macisaac CM, Presneill JJ, Cade JF, Visvanathan K: Epidemiology of sepsis in Victoria, Australia. Crit Care Med. 2005, 33: 71-80.
Garnacho-Montero J, Garcia-Garmendia JL, Barrero-Almodovar A, Jimenez-Jimenez FJ, Perez-Paredes C, Ortiz-Leyba C: Impact of adequate empirical antibiotic therapy on the outcome of patients admitted to the intensive care unit with sepsis. Crit Care Med. 2003, 31: 2742-2751.
Kollef MH, Sherman G, Ward S, Fraser VJ: Inadequate antimicrobial treatment of infections: a risk factor for hospital mortality among critically ill patients. Chest. 1999, 115: 462-474.
Kumar A, Ellis P, Arabi Y, Roberts D, Light B, Parrillo JE, Dodek P, Wood G, Kumar A, Simon D, Peters C, Ahsan M, Chateau D, Cooperative Antimicrobial Therapy of Septic Shock Database Research Group: Initiation of inappropriate antimicrobial therapy results in a fivefold reduction of survival in human septic shock. Chest. 2009, 136: 1237-1248.
Menendez R, Torres A, Reyes S, Zalacain R, Capelastegui A, Aspa J, Borderias L, Martin-Villasclaras JJ, Bello S, Alfageme I, de Castro FR, Rello J, Molinos L, Ruiz-Manzano J: Initial management of pneumonia and sepsis: factors associated with improved outcome. Eur Respir J. 2012, 39: 156-162.
Grol R, Grimshaw J: From best evidence to best practice: effective implementation of change in patients' care. Lancet. 2003, 362: 1225-1230.
Campbell SM, Braspenning J, Hutchinson A, Marshall MN: Research methods used in developing and applying quality indicators in primary care. BMJ. 2003, 326: 816-819.
SWAB guidelines for Antibacterial therapy of adult patients with sepsis. 2010, http://www.swab.nl/swab/cms3.nsf/uploads/65FB380648516FF2C125780F002C39E2/$FILE/swab_sepsis_guideline_december_2010.pdf,
Mourad SM, Hermens RP, Nelen WL, Braat DD, Grol RP, Kremer JA: Guideline-based development of quality indicators for subfertility care. Hum Reprod. 2007, 22: 2665-2672.
Stienen JJ, Tabbers MM, Benninga MA, Harmsen M, Ouwens MM: Development of quality indicators based on a multidisciplinary, evidence-based guideline on pediatric constipation. Eur J Pediatr. 2011, 170 (12): 1513-1519.
CBO: Kwaliteitsinstituut voor de Gezondheidszorg CBO, handleiding voor werkgroepleden. [http://www.cbo.nl/themas/evidence-based-werken-richtlijnen-/projecten/richtlijnen]
Campbell SM, Cantrill JA, Roberts D: Prescribing indicators for UK general practice: Delphi consultation study. BMJ. 2000, 321: 425-428.
McGregor JC, Rich SE, Harris AD, Perencevich EN, Osih R, Lodise TP, Miller RR, Furuno JP: A systematic review of the methods used to assess the association between appropriate antibiotic therapy and mortality in bacteremic patients. Clin Infect Dis. 2007, 45: 329-337.
Marwick C, Watts E, Evans J, Davey P: Quality of care in sepsis management: development and testing of measures for improvement. J Antimicrob Chemother. 2007, 60: 694-697.
Spoorenberg V, Hulscher ME, Akkermans RP, Prins JM, Geerlings SE: Appropriate antibiotic use for patients with urinary tract infections reduces length of hospital stay. Clin Infect Dis. 2014, 58: 164-169.
Berenholtz SM, Pronovost PJ, Ngo K, Barie PS, Hitt J, Kuti JL, Septimus E, Lawler N, Schilling L, Dorman T: Developing quality measures for sepsis care in the ICU. Jt Comm J Qual Patient Saf. 2007, 33: 559-568.
Diaz-Martin A, Martinez-Gonzalez ML, Ferrer R, Ortiz-Leyba C, Piacentini E, Lopez-Pueyo MJ, Martin-Loeches I, Levy MM, Artigas A, Garnacho-Montero J: Antibiotic prescription patterns in the empiric therapy of severe sepsis: combination of antimicrobials with different mechanisms of action reduces mortality. Crit Care. 2012, 16: R223-
Levy MM, Dellinger RP, Townsend SR, Linde-Zwirble WT, Marshall JC, Bion J, Schorr C, Artigas A, Ramsay G, Beale R, Parker MM, Gerlach H, Reinhart K, Silva E, Harvey M, Regan S, Angus DC: The surviving sepsis campaign: results of an international guideline-based performance improvement program targeting severe sepsis. Intensive Care Med. 2010, 36: 222-231.
Nguyen HB, Corbett SW, Steele R, Banta J, Clark RT, Hayes SR, Edwards J, Cho TW, Wittlake WA: Implementation of a bundle of quality indicators for the early management of severe sepsis and septic shock is associated with decreased mortality. Crit Care Med. 2007, 35: 1105-1112.
Pestana D, Espinosa E, Sanguesa-Molina JR, Ramos R, Perez-Fernandez E, Duque M, Martinez-Casanova E: Compliance with a sepsis bundle and its effect on intensive care unit mortality in surgical septic shock patients. J Trauma. 2010, 69: 1282-1287.
Schull MJ, Guttmann A, Leaver CA, Vermeulen M, Hatcher CM, Rowe BH, Zwarenstein M, Anderson GM: Prioritizing performance measurement for emergency department care: consensus on evidence-based quality of care indicators. CJEM. 2011, 13: 300-343.
Bochud PY, Bonten M, Marchetti O, Calandra T: Antimicrobial therapy for patients with severe sepsis and septic shock: an evidence-based review. Crit Care Med. 2004, 32: S495-S512.
Hermanides HS, Hulscher ME, Schouten JA, Prins JM, Geerlings SE: Development of quality indicators for the antibiotic treatment of complicated urinary tract infections: a first step to measure and improve care. Clin Infect Dis. 2008, 46: 703-711.
Schouten JA, Hulscher ME, Wollersheim H, Braspennning J, Kullberg BJ, van der Meer JW, Grol RP: Quality of antibiotic use for lower respiratory tract infections at hospitals: (how) can we measure it?. Clin Infect Dis. 2005, 41: 450-460.
Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C: Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011, 6: e20476-
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2334/14/345/prepub
Collaborators: Dutch Sepsis QI expert panel. H.I. Bax, MD. E.F. Schippers, MD, PhD. S. van Assen, MD, PhD. C.W. Ang, MD, PhD. P. Sturm, MD, PhD. Y.G. van der Meer, PhD. Prof. M.A. Boermeester, MD. J.A. Schouten, MD, PhD. Prof. P. Pickkers, MD. J.J.W.M. Janssen, MD, PhD. Prof. N.M.A. Blijlevens, MD. N.P. Juffermans, MD, PhD.
This work was supported by Zon/MW, the Netherlands Organisation for Health Research and Development, project number 205100003.
The authors declare that they have no competing interests.
CB conducted and organized the entire Delphi procedure, analyzed the results, prepared the consensus meeting and drafted the manuscript. MH carried out the extraction of the guideline recommendations, participated in organizing the Delphi procedure and analyzing the results, and rewrote/revised the manuscript. SN participated in the Delphi procedure and rewrote/revised the manuscript. IG participated in the Delphi procedure and rewrote/revised the manuscript. JP participated in the consensus meeting, helped organizing the Delphi procedure and rewrote/revised the manuscript. SE carried out the extraction of the guideline recommendations, participated in organizing the Delphi procedure and analyzing the results, and rewrote/revised the manuscript. All authors read and approved the final manuscript.
Electronic supplementary material
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
About this article
Cite this article
van den Bosch, C.M., Hulscher, M.E., Natsch, S. et al. Development of quality indicators for antimicrobial treatment in adults with sepsis. BMC Infect Dis 14, 345 (2014). https://doi.org/10.1186/1471-2334-14-345