A cluster randomized trial for the implementation of an antibiotic checklist based on validated quality indicators: the AB-checklist
BMC Infectious Diseases volume 15, Article number: 134 (2015)
Recently we developed and validated generic quality indicators that define ‘appropriate antibiotic use’ in hospitalized adults treated for a (suspected) bacterial infection. Previous studies have shown that with appropriate antibiotic use a reduction of 13% of length of hospital stay can be achieved. Our main objective in this project is to provide hospitals with an antibiotic checklist based on these quality indicators, and to evaluate the introduction of this checklist in terms of (cost-) effectiveness.
The checklist applies to hospitalized adults with a suspected bacterial infection for whom antibiotic therapy is initiated, at first via the intravenous route. A stepped wedge study design will be used, comparing outcomes before and after introduction of the checklist in nine hospitals in the Netherlands. At least 810 patients will be included in both the control and the intervention group. The primary endpoint is length of hospital stay. Secondary endpoints are appropriate antibiotic use measured by the quality indicators, admission to and duration of intensive care unit stay, readmission within 30 days, mortality, total antibiotic use, and costs associated with implementation and hospital stay. Differences in numerical endpoints between the two periods will be evaluated with mixed linear models; for dichotomous outcomes generalized estimating equation models will be used. A process evaluation will be performed to evaluate the professionals’ compliance with use of the checklist. The key question for the economic evaluation is whether the benefits of the checklist, which include reduced antibiotic use, reduced length of stay and associated costs, justify the costs associated with implementation activities as well as daily use of the checklist.
If (cost-) effective, the AB-checklist will provide physicians with a tool to support appropriate antibiotic use in adult hospitalized patients who start with intravenous antibiotics.
Dutch trial registry: NTR4872
The need to improve antibiotic use
The increasing antimicrobial resistance rate is one of the most important health care problems at this moment. The total consumption of antibiotics is the main driving force [1,2]. The World Health Organization signalled the emergence of antimicrobial resistance (AMR), along with the steady decline in the discovery of new antimicrobials, as a major health threat for the coming decade. To help control AMR, a better use of the current agents is necessary . Recent studies have shown considerable room for improvement in the two most common bacterial infections: respiratory and urinary tract infections [4,5]. An important question is how to achieve such an improvement.
Previous studies have shown that appropriate antibiotic use is not only of great importance to curb antimicrobial resistance, but also has a short-term consequence. The evaluation of antibiotic treatment for urinary tract infections in hospitals showed an inverse relationship between the proportion of appropriate antibiotic use in a patient and length of stay (LOS). Prescribing therapy in accordance with local hospital guidelines was associated with a shorter LOS (7.3 days vs. 8.7 days; P = 0.024) . Recently a similar inverse relationship has been shown between appropriate antibiotic use in treatment of all bacterial infections in the hospital and LOS, with a reduction of 13% in LOS with appropriate use. [Abstract poster: Quality Indicators for Monitoring Appropriate Antibiotic Use in Hospitals: an Important Tool for Antibiotic Stewardship. K-325. 54th Interscience Conference on Antimicrobial Agents and Chemotherapy, September 5 -9, 2014, Washington DC].
Guidelines defining appropriate antimicrobial treatment are available. To measure whether guideline recommendations are implemented in daily practice, an instrument is needed that validly measures the quality of antibiotic use in daily clinical practice. Guideline-based quality indicators (QIs) are such tools. Recently, generic QIs for the antimicrobial treatment of bacterial infections in hospitals have been developed and validated . The challenge is the embedding of appropriate antibiotic use, as defined by these QIs, in daily practice.
A checklist can be a tool for implementation; since it has been shown in other high-risk disciplines that adherence to checklists increases safety . This also holds true for the field of medicine. For example, in surgery, implementation of a comprehensive multidisciplinary SURgical PAtient Safety System (SURPASS) is associated with an absolute risk reduction in surgical complications of 10.6% . Similarly, the introduction of a checklist to improve patient care among gynaecologic oncology patients has resulted in a decreased length of stay of one day (4.5 days in the pre- and 3.5 days in the post-implementation period (P = 0.007)) . In the field of infectious diseases, Pronovost et al. showed that introduction of a five-points checklist to reduce infections when inserting a central venous catheter, resulted in a decrease of the median rate of infections from 2.7 per 1,000 patients to zero after three months .
We hypothesize that an antibiotic checklist, to be used at the start of antibiotic treatment and after 48-72 hours of treatment, will improve clinical patient care, reduce costs and will ultimately contribute to the containment of antimicrobial resistance.
This study is set to implement an antibiotic checklist based on the validated generic quality indicators measuring the appropriateness of intravenously initiated antibiotic use in hospitalized adults treated for a suspected bacterial infection, and to analyse the effect of the introduction of the checklist on patient care in terms of shorter duration of intensive care unit (ICU)- and hospital stay, adequate treatment, decreased mortality rates, decreased total antibiotic use and lower costs.
Methods and design
Phase 1: The AB-checklist and barriers for use
The ‘Development of reliable generic quality indicators for the optimization of antibiotic use in the hospital’ (RIANT)-study used a RAND modified Delphi method to develop a set of generic QIs for appropriate antibiotic use in the treatment of all bacterial infections in hospitalized adult patients . The clinimetric properties of these QIs have been tested in 1,890 hospitalized patients, in 22 Dutch hospitals. This analysis resulted in eight useful QIs, of which seven were process indicators that focus on the actual care provided. The antibiotic checklist includes these seven QIs.
The antibiotic checklist is intended as a reminding tool for the physician who starts intravenous (IV) antibiotic treatment. It is actually divided into two bundles (Figure 1). The first bundle has to be completed at the moment of prescribing IV antibiotics, and comprises five items. The second bundle is used during the course of treatment, at the latest after 72 hours of treatment. This part consists of two items.
As a preparation, we identified barriers of implementing this checklist, using a questionnaire based on an instrument of the Netherlands Organization for Applied Scientific Research (TNO), which is designed to survey determinants that influence the uptake of an innovation in health care organizations [12,13]. The questionnaire was sent to specialists and residents who work at the participating hospitals. Based on the identified barriers, the checklist design and its implementation format were adapted.
Phase 2: cluster randomized clinical trial to evaluate the AB-checklist
Study design and setting
The (cost-) effectiveness of introducing the antibiotic checklist will be studied in a controlled, multicentre, prospective study using a stepped wedge design, comparing outcomes before and after implementation of the checklist while accounting for potential confounders .
The checklist will be introduced in nine hospitals over four time periods, during a total study period of eleven months (Figure 2). The aim is to include at least 810 patients in the period before- (the baseline group), and at least 810 patients with a completed checklist in the period after checklist introduction (intervention group). During the transition period implementation activities will be started and no patients will be included. This period runs from one month before introduction of the checklist to one week after introduction. Since compliance to the checklist will not be 100%, we will also collect data concerning the LOS from the patients in whom the checklist should have been but was not completed during the period after checklist introduction (the spill over group).
To promote selection of a patient population representative for every day clinical practice in the Netherlands, two university hospitals and seven teaching hospitals will participate, and in each hospital at least one surgical and one non-surgical ward will be included. The ICU and the paediatric ward will be excluded, as the quality indicators do not apply to these populations. Patients with a suspected bacterial infection presenting on the emergency department (ED) will be included if they are going to be hospitalized on one of the participating wards.
The study coordinator will design a project plan together with the local project leader. This plan will focus on physicians, nurses and medical students who are involved in patient care. The plan will include standard items (see Table 1), but additional items can be added, if locally required, to cover practical issues as for example: How do we arrange that for each patient starting with antibiotics intravenously a checklist is filled in? How do we arrange that the checklist does not get lost?
The physician will use the checklist in all hospitalized adults (≥18 years old), or adults at the ED who will be admitted at a participating ward, with a suspected community-acquired and/or hospital-acquired bacterial infection, and will be treated with intravenous antibiotics. The treating physician identifies these patients upon prescription of antibiotic therapy. The patients included in the study are all patients that meet the above mentioned inclusion criteria, whether they have a completed checklist or not (see Figure 3 for the flowchart). They will be included in each hospital using an overview of the patients who started with intravenous antibiotics. This overview will be generated by the local departments of hospital pharmacy from the computerized physician order entry systems. Patients are excluded from the study in case of anticipated hospital stay of less than 24 hours, antibiotics used as prophylaxis or intended for less than 24 hours, hospitalization at the ICU at the start of the treatment, transfer from another hospital, or antibiotic treatment in the ambulance before presentation at the ED.
Primary and secondary endpoints
The primary endpoint is length of hospital stay in days. LOS in community-acquired infections will be defined as the number of days between admission and discharge. In hospital-acquired infections LOS will be defined as the number of days between start of antibiotic treatment and discharge. Secondary endpoints are appropriate antibiotic treatment according to the generic QIs (yes/no per QI and sum score of all QIs) , admission to and duration of ICU stay in days, readmission within 30 days, mortality in the hospital and in the first 30 days after discharge, total antibiotic use, costs of hospital stay and costs associated with implementation.
For patient outcome we will collect data concerning the LOS, admission to and duration of ICU stay, readmission within 30 days and mortality in hospital or within 30 days after discharge for the baseline group and the intervention group. Furthermore we will collect data for possible confounders of patient’s outcome, namely age, sex, community-or hospital acquired infection, Charlson comorbidity index [15,16], use of antibiotics during the previous 30 days, and clinical condition as assessed by the Modified Early Warning System . To calculate QI performance we will collect data concerning the number of taken blood cultures including its results, the number of taken cultures of suspected sides of infection including its results, antibiotic use including dose, interval and duration of treatment in days, the (suspected) type of infection and relevant laboratory parameters. For the spill over group, we will only collect data concerning the LOS and the possible confounders.
Antibiotics included in the study belong to group J01 (antimicrobials for systemic use) of the Anatomic Therapeutic Chemical (ATC) classification system. When multiple antibiotics are prescribed to one patient, all antibiotics will be registered. Antiviral therapy, medication for tuberculosis, antibiotics used for non-systemic selective intestinal decontamination and antibiotics used as pro-kinetics will be excluded.
The power analysis is based on previous results of appropriate antibiotic use on length of stay as described at ‘background’ [6,7]. In a simulation study based on the stepped wedge design, with nine hospitals each contributing 180 patients with a suspected infection, we estimated the power to approximate 80% to demonstrate a significant reduction of 13% in length of stay, assuming that the intraclass correlation coefficient does not exceed 0.20. With each hospital contributing 180 patients with a suspected infection, the total sample size will be 1620 patients. For each hospital we have determined a minimum number of patients in the baseline group and intervention group, depending on the randomization order of the hospital in the stepped wedge.
We will use mixed linear models to analyse the overall effect of the introduction of the checklist on length of stay. Therefore we will evaluate differences between baseline (before introduction of the checklist) and intervention plus spill over (after introduction of the checklist) (intention to treat analysis). We will also determine the theoretical effect of the checklist on LOS by evaluating differences between baseline and intervention data without the spill over data (per protocol analysis). Differences between the baseline data and the intervention data for secondary outcomes will also be evaluated with mixed linear models. For dichotomous outcomes (mortality), the comparison will be evaluated with a similar approach, using generalized estimating equation models. These mixed models account for within-cluster dependencies, and allow adjustment for possible confounders. To assess whether differences between these groups may be explained by confounding patient characteristics, we will first test with multivariate multilevel (linear mixed) analyses if these variables have a significant effect (p < 0.05) on the outcomes. Hereafter, we will compute the estimate of the total causal effect of checklist implementation on the outcomes before and after adjusting for a significant variable. If the estimate varies 10% or more, possible confounding is present and in that case we will adjust for this variable.
Validation and process evaluation
We will evaluate the physicians’ compliance with the checklist. Firstly we will check whether the ticks in the completed checklists are in accordance with the performance in clinical practice. Secondly we will determine the percentage of eligible patients for whom the checklist has been used after checklist introduction. The number of patients in whom the checklist was actually used (the intervention group) will be divided by the number of patients in whom it should have been used (the intervention plus the spill over group).
The key question for the economic evaluation is whether the benefits of using the checklist, which will likely include reduced antibiotic use, reduced length of stay and associated costs, justify the costs associated with implementation activities as well as daily use of the checklist. For reasons of feasibility, the cost analysis will only include a few main cost drivers of health care utilization, namely LOS on a general or ICU ward and antibiotic use. Unit costs for health care utilization will be estimated according to the Dutch guideline on (unit) costing in health care . LOS and ICU stay will be valued based on the guideline prices. Furthermore, we will compare the total antibiotic use in days per antibiotic class in the baseline and the intervention group. Antibiotic costs will not be valued because the benefits become more evident in terms of reduced quantities rather than their particular costs. Implementation costs will be divided in non-recurring and recurring costs. Non-recurring costs are related to our study, such as costs of the development of the materials, the implementation strategy and costs of evaluation of the implementation. Recurring costs are the costs of accomplishment of the implementation strategy. The primary analysis is a cost-effectiveness analysis, with the incremental cost-effectiveness ratio expressed as the implementation related costs per reduction of length of stay in days. As we hypothesize that an effective implementation of the checklist will result in reduced health care use and thus lower costs, we will also perform a cost-benefit analysis, evaluating whether the implementation costs are offset by this cost reduction (costs of implementation < health care cost reduction). Robustness of the results for uncertainty in the assumptions will be evaluated in sensitivity analyses, including estimates of unit costs and volume of implementation activities (considering less activities will be needed if checklists become part of standard care). In a budget impact analysis results from the economic evaluation will be extrapolated to the national level to estimate the total clinical and budget impact in terms of health benefits, reduction in antibiotic use, and associated economic impact on the health care budget per annum for the Netherlands. The budget impact analysis will be performed according to the criteria provided by Mauskopf et al. , from a societal perspective as well as from the health insurance and/or national health care budgeting framework perspective. We will evaluate the impact of varying implementation rates, comparing care as usual (0% implementation), potential impact (100% implementation), and the probable rates of 85% and 50% implementation.
The project proposal was discussed with the Medical Ethics Research Committee of the Academic Medical Centre. They confirmed that the Medical Research Involving Human Subjects Acts does not apply to this study and that an official approval by the committee was not required, because patients will receive treatment according to standard care, and will have no burden of checklist use.
This study protocol describes the design, implementation and evaluation of the AB-checklist study. To our knowledge, no other published studies or ongoing trials (http://apps.who.int/trialsearch/) investigate the implementation of a generic antibiotic checklist to be used for suspected bacterial infections in the hospital. If (cost) effective, the AB-checklist will provide physicians with a tool to support appropriate antibiotic use in adult hospitalized patients who start intravenous antimicrobials.
Intensive care unit
Length of hospital stay
Goossens H, Ferech M, Vander Stichele R, Elseviers M, ESAC Project Group. Outpatient antibiotic use in Europe and association with resistance: a cross-national database study. Lancet. 2005;365:579–87.
Tacconelli E. Antimicrobial use: risk driver of multidrug resistant microorganisms in healthcare settings. Curr Opin Infect Dis. 2009;22:352–8.
WHO. Antimicrobial resistance: global report on surveillance 2014. http://www.who.int/drugresistance/documents/surveillancereport/en/. Accessed April 2014.
Schouten JA, Hulscher MEJL, Wollersheim H, Braspenning J, Kullberg BJ, van der Meer JWM, et al. Quality of antibiotic use for lower respiratory tract infections at hospitals: (How) can we measure it? Clin Infect Dis. 2005;41:450–60.
Hermanides HS, Hulscher MEJL, Schouten JA, Prins JM, Geerlings SE. Development of quality indicators for the antibiotic treatment of complicated urinary tract infections in the hospital. Clin Infect Dis. 2008;46:703–11.
Spoorenberg V, Hulscher ME, Akkermans RP, Prins JM, Geerlings SE. Appropriate antibiotic use for patients with urinary tract infections reduces length of hospital stay. Clin Infect Dis. 2014;58:164–9.
van den Bosch CM, Geerlings SE, Natsch S, Wille J, Prins JM, Hulscher ME. Quality indicators to measure appropriate antibiotic use in hospitalized adults. Clin Infect Dis. 2015;60:281–91.
Gawande A. The Checklist Manifesto. How to Get Things Right. Metropolitan Books of Henry Holt and Company LLC; 2009.
de Vries EN, Prins HA, Crolla RM, den Outer AJ, van Andel G, van Helden SH, et al. Effect of a comprehensive surgical safety system on patient outcomes. N Engl J Med. 2010;363:1928–37.
Diaz-Montes TP, Cobb L, Ibeanu OA, Njoku P, Gerardi MA. Introduction of checklists at daily progress notes improves patient care among the gynecological oncology service. J Patient Saf. 2012;8:189–93.
Pronovost P, Needham D, Berenholtz S, Sinopoli D, Chu H, Cosgrove S, et al. An intervention to decrease catheter-related bloodstream infections in the ICU. N Engl J Med. 2006;355:2725–32.
Fleuren M, Paulussen TGWM, van Dommelen P, van Buuren S. Meetinstrument voor Determinanten van Innovaties (MIDI). Leiden: TNO; 2012. https://www.tno.nl/.
Fleuren M, Wiefferink K, Paulussen T. Determinants of innovation within health care organizations: literature review and Delphi study. Int J Qual Health Care. 2004;16:107–23.
Brown. The stepped wedge trial design: a systematic review. BMC Med Res Methodol. 2006;6:54.
Sundararajan V, Quan H, Halfon P, Fushimi K, Luthi JC, Burnand B, et al. for the International Methodology Consortium for Coded Health Information (IMECCHI). Cross-National Comparative Performance of Three Versions of the ICD-10 Charlson Index. Med Care. 2007;45:1210–5.
Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi JC, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005;43:1130–9.
Subbe CP, Kruger M, Rutherford P, Gemmel L. Validation of a modified early warning score in medical admissions. Q J Med. 2001;94:521–6.
Hakkaart- van Roijen L, Tan SS, Bouwmans CAM. Medical Technology Assessment Institute. Handleiding voor kostenonderzoek - Methoden en standaard kostprijzen voor economische evaluaties in de gezondheidszorg. College voor zorgverzekering. 2010. http://www.zinl.nl.
Mauskopf JA, Sullivan SD, Annemans L, Caro J, Mullins CD, Nuijten M, et al. Principles of good practice for budget impact analysis: report of the ISPOR Task Force on good research practices-budget impact analysis. Value Health. 2007;10:336–47.
This study is funded by the Netherlands Organisation for Health Research and Development (ZonMw) grant 836021001.
The authors declare that they have no competing interests.
SEG, JMP and MEJLH were involved in the conception of the study and coordinated the application for funding. BCO outlined the statistical and economic analysis. FVD, SEG and JMP drafted the manuscript. All authors are members of the ‘Implementation of an antibiotic checklist’ study group. All authors read and approved the final version of the manuscript.
About this article
Cite this article
van Daalen, F.V., Prins, J.M., Opmeer, B.C. et al. A cluster randomized trial for the implementation of an antibiotic checklist based on validated quality indicators: the AB-checklist. BMC Infect Dis 15, 134 (2015). https://doi.org/10.1186/s12879-015-0867-2
- Quality indicators
- Stepped-wedge design