Identifying heterogeneity in the Hawthorne effect on hand hygiene observation: a cohort study of overtly and covertly observed results

Hand hygiene (HH) is an effective measure to prevent healthcare-associated infection [1–4]. Observation and feedback make up one of the five main strategies to improve HH [3]. Direct observation remains the gold standard method of evaluating HH compliance [3, 4]. However, direct observation has several limitations, including the ability to observe only a small percentage of HH opportunities in real-world scenarios, the need for a large number of observers and a considerable amount of time, the lack of a standardized training/validation process, and observation bias [5, 6].

Observation bias, also known as the Hawthorne effect, plays a critical role in evaluating HH compliance. When the Hawthorne effect was estimated by the difference between direct overt and covert observation results, the magnitude of the effect on HH compliance ranged from 7 to 16 percentage points (PPs) before 2009 [7–9] to 30–34 PPs after Five Moments for Hand Hygiene became a standard in 2009 [10, 11]. Srigley et al. even noted a surprising 3-fold increase in HH compliance under direct observation vs under an electronic monitoring system (EMS) [12]. Hagel et al. reported a similar outcome by using electronic handrub dispensers to record HH events and captured a 2.6-fold higher HH density when healthcare workers (HCWs) were under observation than when they were not (21 vs 8 HH events per hour) [13]. McLaws and Kwok demonstrated direct observation rates were inflated by an average of 55–64 PPs (2.8–3.1 times higher) than automated surveillance rates by an EMS in a medical ward [14]. The Hawthorne effect may alter HCWs’ usual behaviour and often leads to overestimation of HH compliance [5, 6, 9, 15]. At least three methods have been proposed to help avoid or decrease the Hawthorne effect besides the use of EMS. These include 1) letting HCWs habituate to the presence of observers, although there is still no standard method for clinical application [3, 6]; 2) using indirect methods to monitor HH compliance, such as monitoring consumption of alcohol handrubs [15]; and 3) covertly observing HH compliance [3, 15].

Few studies have provided a detailed description and subgroup analysis of the Hawthorne effect [9, 11]. Kohli et al. conducted an observational study comparing the observation results of three infection control members and one student intern in three inpatient care units and found that the Hawthorne effect was more pronounced in high-performing than low-performing units. The authors speculated that the staff in high-performing units take pride in their work and may wish to show their good performance to recognized observers [9]. Kovacs-Litman and colleagues described a much lower difference between covertly and overtly observed compliance rates in physicians than in nurses (19 PPs vs 41 PPs, p < 0.0001) [11]. However, the two studies had several study design flaws, including the presence of selection bias [9, 11], a small number of observers [9], only HH opportunities occurred before and after contact with the patient or patient’s environment being observed [9], and a relatively small number of HH opportunities [9]. Since healthcare facilities around the world spend so much time and utilize so many people for HH compliance observation (mostly overt), further research to investigate the Hawthorne effect is necessary [6].

The main purpose of the present study is to explore the heterogeneity of the Hawthorne effect in different HCW subgroups and settings. We analysed covert and overt observation data in the same hospital during the same period after controlling for observation place, time, department, and occupation category. The study results may clarify the impact of the Hawthorne effect under various HH circumstances.

This study comprehensively investigated the extent of the Hawthorne effect on HH compliance observation and revealed that the Hawthorne effect differed significantly for HCW occupations and observation locations but not for departments, observation times, or HH indications. In this study, several attempts were made to reduce potential bias. In order to minimize observer bias, both overt and covert observers received the same training and underwent the same certification process; moreover, a large group of observers were recruited to observe a massive number of HH opportunities. To minimize selection bias, we used matched analysis to balance 4 important variables between the overt and covert observation groups. The overall Hawthorne effect was comparable between our study (24 PPs) and previous studies of direct HH compliance observation (7–34 PPs) [7–11]. The study results may influence how we interpret direct observer audits of HH compliance in the future.

The heterogeneity of the Hawthorne effect on HH observation has several impacts on HH monitoring. First, Hawthorne effect heterogeneity should be taken into consideration when identifying risk factors for not washing one’s hands. The identification of risk factors for HH noncompliance was often based on analyses of overt observation data in previous studies [20, 21]. For example, HH compliance was considered lower in physicians than in nurses [7, 20]. Is this proposition true, or do the observations reflect a greater Hawthorne effect in nurses than physicians [11]? The answer may even vary among health care facilities. Second, some experts may take advantage of the Hawthorne effect on HH observation to improve HH compliance, that is, using the strategy of frequent HH observation to stimulate HCWs to wash their hands more often. Implementing the strategy in the scenario with a higher Hawthorne effect will maximize the efficiency of HH improvement programmes. Third, using HH compliance by overt observation as a performance indicator to compare different HCW occupations, wards, and hospitals is not adequate. The major purpose of overt observation should remain the provision of feedback to HCWs.

Our finding that the Hawthorne effect impacted physicians less than nurses (11 PPs vs. 30 PPs) was comparable to the finding of Kovacs-Litman and colleagues (19 PPs vs. 41 PPs) [11]. Our study did not show the previously reported more pronounced Hawthorne effect on HH in higher-performance units and lack of a significant Hawthorne effect on HH in low-performance units [9]. Kwok et al. noted that a socially cohesive ward is more likely than a socially isolated ward to adopt an intervention for HH improvement [22]. This difference may partially explain why units with high HH performance, which are presumed to be more social cohesive, have a larger response to overt observation, and vice versa. However, in our study, HH compliance was higher and the Hawthorne effect on compliance was lower in intensive care units than in wards or the outpatient department; moreover, HH compliance was lowest and the Hawthorne effect was highest in the surgical department. We consider the association between HH performance and the Hawthorne effect to be undetermined. Further studies are needed to address this issue.

The use of EMS to monitor HH compliance has attracted increasing attention in recent years [12–14, 23, 24]. An interesting point is that the estimated Hawthorne effect is often larger when HH compliance from overt observation is compared to EMS measures (1.6 to 3.1 times higher) [12–14] than to covertly observed measures (7–34 PPs or 12–69% higher) [7–11], including this study (24 PPs or 43% higher). This difference may imply that the Hawthorne effect can only be reduced, rather than eliminated, by covert observation. As long as the HCWs are not alone, whether the persons nearby tend to observe HH or not, the Hawthorne effect exists. To go one step further, the EMS may also produce a Hawthorne effect to a certain extent if the HCWs are aware of the existence of EMS. The differences in results between various observation methods may cause another form of Hawthorne effect heterogeneity.

There are several limitations to the study. A major limitation is that the Hawthorne effect was measured by calculating the difference between overt and covert observation results, as in previous studies [7–9, 11]. Although the Hawthorne effect may contribute the most to the difference, selection and observer bias will certainly exist despite statistical correction. For example, the covert observers may tend to observe HH at busier times to see more HH opportunities in less time due to the monetary incentive. Busier times were associated with lower HH compliance; thus, a form of selection bias may be present. In addition, the professions of the observers in overt and covert observation were considerably different, and we did not analyse the impact of observer profession (e.g., nurse vs physician observers) on the Hawthorne effect in the study. Observer bias may occur as a result. We believe that an accurate measurement of the Hawthorne effect can be obtained only through well-designed controlled trials, for example, the same group of observers conducting both overt and covert observations in designated areas during the same period of time. Until perfect results are available, the results retrieved from a large cohort with careful stratification and matching may provide useful information. Second, this investigation is a single-centre study that was conducted in a hospital with good HH implementation. The impact of the Hawthorne effect may vary from hospital to hospital, and we recommend caution in applying the study results to other hospitals. Third, covert observation may not be generally acceptable in other hospitals or countries, which would prevent the generalization of the study method.

In this study, we demonstrated not only the existence of the Hawthorne effect but also a significant difference in the Hawthorne effect among different HCW occupations and observation locations. Due to the heterogeneity of the Hawthorne effect, overt observation should not be used as the sole method of comparing performance among different clinical settings, HCWs, or hospitals. Overt observation continues to be the main method of HH observation because it is easy to perform and provides more information for feedback than EMS provides. However, better methods of benchmarking performance are needed.