Description of economic evaluations identified

The interventions evaluated by the studies included N95 respirators [12, 20, 22], HEPA, DM and DMF respirators [19, 23], and surgical masks [19–21, 23]. Amongst the studies that focused on influenza, two of the five, by Dan et al. [12] and Chen and Liao [21], integrated mask/respirator use in a scenario in conjunction with other non-pharmaceutical control measures e.g. natural ventilation [21] or isolation measures [12], as opposed to evaluating the use of masks/respirators exclusively. This makes it difficult to assess to what degree the mask/respirator aspect of the intervention is driving the cost-effectiveness, or if the results are more attributable to the other control measures included.

The settings of the evaluated studies varied more broadly for the influenza studies than for those on tuberculosis. Three of the five influenza studies were in a community setting [11, 20, 22]. Of the others, one considered control measures for both the community and hospital setting [12] and the last was set in a primary school [21]. All of the seven identified studies were set in the USA or other developed countries, limiting the generalisability of the findings, e.g. for decision making in low resource settings. The lack of studies in low resource settings has been cited as one of the major impediments to the use of cost-effectiveness analysis for pandemic influenza preparedness planning in this context [24].

Intervention outcome measures

Four of the five influenza studies included the number of infection cases as one of the outcome measures of intervention effectiveness (Table 1) [11, 12, 21, 22]. The remaining influenza study quantified and reported the consequences of infection on other outcomes e.g. prevented deaths, hospitalisations, outpatient visits and absenteeism [20].

Both tuberculosis studies measured conversion rates of newly positive protein derivative tests in HCWs at study hospitals [19, 23]. A positive test indicates a 5–10 % chance that the individual will develop active tuberculosis in their lifetime [25]. One of the two studies also estimated the number of patients isolated for known tuberculosis per month, confirmed pulmonary tuberculosis cases in patients in the past year and active cases of pulmonary tuberculosis in HCWs in the last 5 years [23]. This additional information provided a more complete measure of the impact of the intervention.

Cost-effectiveness outcomes

Only three [12, 19, 23] of the seven studies reported results as cost per unit of effect. The evaluation by Dan et al. [12] of influenza and SARS reported an incremental cost per death averted [12]. The other two were the tuberculosis studies, with one reporting the cost to prevent a single occupational case of tuberculosis in the next 41 years [19] and the other reporting the cost per tuberculosis case prevented and the cost per life saved [23]. Two of the pandemic influenza studies evaluating combinations of interventions in the community concluded that mask/respirator interventions were cost saving in a pandemic setting [11, 22]. The first estimated substantial net savings if 10 % of the population had worn N95 respirators during influenza A(H1N1)pdm09, assuming an effectiveness of 20 % [11]. The second estimated that surgical mask use at 60 % compliance would yield savings of $US100-250 million, far exceeding the estimated intervention cost of $US20 million [20]. The influenza modelling studies by Jones and Adida [22] focused on demonstrating the feasibility of the methods rather than reporting key economic results. The influenza study by Chen and Liao [21] calculated a cost per individual to implement control measures, but limited the inclusion of costs to intervention and outpatient treatments costs.

Mask and respirator effectiveness or “strategy impact”

Approaches to incorporate intervention effectiveness amongst the seven studies varied (Table 1). Both of the tuberculosis evaluations by Adal et al. [19] and Nettleman et al. [23] reported that no efficacy studies were available and thus no specific measures of mask or respirator effectiveness were included in their evaluations. Nettleman et al. [23] assumed that in a best case scenario, HEPA respirators would prevent 25 % of HCW exposure to tuberculosis, but no data was cited to support this estimate. Adal et al. [19] implicitly assumed that all HCW cases of tuberculosis that may have been acquired from patients in isolation with active tuberculosis would be prevented by the intervention.

Of the five influenza studies, Tracht et al. [11] was the only study that attempted to utilise clinical efficacy data. They obtained this from a randomised trial in a university residence setting [26] that found a significant decrease (35–51 %) in influenza-like illness if hand hygiene and medical masks were used together compared to a control, but no difference for medical mask use alone. Tracht et al. [11] uses this result to estimate N95 respirator use provides a 50 % decrease in susceptibility to influenza A(H1N1)pdm09. This may not be robust as the clinical trial measured different outcomes (i.e. influenza-like illness, not influenza A(H1N1)pdm09 cases) and it assumes there is no difference between the efficacy of medical masks and N95 respirators based on the findings of Loeb et al. [27], although other trials have found evidence that there is a difference [28]. The study by Jones and Adida [22] derived efficacy estimates from the assigned protective factor (APF) measure for N95 respirators released by the National Institute for Occupational Safety and Health [29]. The APF is estimated from laboratory testing completed by this organisation, including laboratory testing of the quantitative measure of fit when the respirators are fit tested to ensure a protective seal around the wearer’s face. Cahill et al. [20] also used efficacy estimates from laboratory investigation data [30]. These estimates may have limited applicability to measuring real-world effectiveness [31]. Dan et al. [12] estimated baseline intervention effectiveness rates of 50, 80 and 90 % as protective equipment (including masks) and isolation precautions were increased. They did not cite data to support these estimates, however a 5 % failure rate where transmission still occurs despite the use of protective equipment and isolation measures, was obtained from a hospital simulation study [32]. Chen and Liao [21] applied a relative efficacy estimate of masks in combination with other control measures which was developed in a previous modelling study that did not cite supporting data for efficacy input assumptions [33]. Other than Tracht et al. [11], none of the analyses used intervention effectiveness estimates from clinical trials such as the mask/respirator randomised trials that have been published to date on interventions in community and healthcare settings [26–28, 34–38].

Costs

Only two of the seven evaluations, by Dan et al. [12] and Jones and Adida et al. [22], explicitly stated the economic perspective from which they had conducted their evaluation, these being a policy developer view [22] and the healthcare institution perspective [12]. On the basis of costs included and excluded, four of the seven studies are most aligned with healthcare payer perspectives [12, 19, 21, 23] and the remaining three with a societal perspective [11, 20, 22] (e.g. they included some form of productivity costs). Both tuberculosis evaluations by Adal et al. [19] and Nettleman et al. [23], considered costs from a healthcare provider perspective, reflecting the objective of preventing the burden of nosocomial cases in HCWs and patients. The influenza studies [11, 20, 22] more frequently applied a societal perspective, consistent with the substantial economic impact of productivity losses due to influenza absenteeism [39].

The reporting of costs included in the evaluations was often not transparent. One evaluation by Dan et al. [12] identified the omission of the indirect costs of lost revenue from the cancellation of elective surgeries, decreased outpatient attendance, lost clinical teaching time and administration costs of senior staff meetings. These costs were not discussed in other influenza modelling studies where they may have been relevant [11, 20, 22]. The tuberculosis studies by Adal et al. [19] and Nettleman et al. [23] relied on hospital databases for the number of masks/respirators used, and multiplied this by unit costs to calculate total costs. Administration costs were most thoroughly reported in the tuberculosis study by Adal et al. [19], which included staff program planning time and staff medical evaluation time.

All of the studies that estimated a cost per case, included the cost of infected individuals [11, 12, 20–22]. For example, one influenza modelling study by Jones and Adida [22] assumed a fixed average cost for each infection. This may not always be an ideal method to cost resources used as the cost of an average case may vary amongst patients [40]. Tracht et al. [11] allocated a separate average hospital cost per day due to influenza A(H1N1)pdm09 infections for three age groups.

Approaches to model adherence to interventions

Three of the studies evaluating the intervention in the community included the impact of compliance [11, 20, 22]. These included simple approaches, as used by Cahill et al. [20] of assessing 60 and 100 % adherence scenarios, as well as by Jones and Adida [22] where an estimate of 50 % compliance was assumed. Tracht et al. [11] conducted a more thorough sensitivity analysis by dividing the population into three age groups and testing three scenarios with different proportions of compliance in each group. No study attempted to use compliance rates from real-world investigations such as clinical trials [28, 36].

What clinical evidence is there on comparative mask and respirator effectiveness?

Only one of the reviewed studies [11] cited clinical evidence to support their mask/respirator intervention effectiveness parameters. However, there is a growing body of research in this area. An initial systematic review on the topic reported pooled efficacy measures from case control studies for N95 respirators and medical masks of 91 and 68 % respectively for the prevention of respiratory viruses [49]. A weakness of this initial review was that the case control studies included in the meta-analysis focused exclusively on SARS [50]. As a consequence, it should not be assumed that the conclusions of this meta-analysis are necessarily true for influenza or other respiratory infections [51].

The updated edition of this review concluded that there was no evidence of a significant difference in effectiveness between N95 respirators and medical masks [50]. However, this conclusion was heavily influenced by a single cluster randomised trial by Loeb et al. [27] that compared N95 respirators and medical masks, both used selectively in high-risk situations. This trial contained only 446 HCWs and may have been underpowered to detect a difference between the arms [28]. Two larger trials not included in this review have been published [28, 36]. One of these by Macintyre et al. [28], involving 1669 HCWs, found significantly greater protective efficacy for continuous N95 use when compared to continuous medical mask use, but no evidence that selective N95 use was superior to continuous medical mask use for the prevention of clinical respiratory illness [28].

Limitations

The limitations of the review include the restriction to English language articles used in the search of the Scopus database. Further potential limitations are that the identification and screening process was carried out predominantly by one of the authors (SM). Finally, the limited number of studies identified for inclusion made it difficult to make general conclusions about cost-effectiveness.

Recommendations

One key recommendation is that future economic analyses should attempt to apply clinical mask/respirator efficacy data, preferably from clinical trials, where this data is deemed applicable to the infectious agent, intervention and setting, rather than using estimations derived from expert opinion or laboratory testing studies [31]. In some cases this may not be possible, for example, it is not appropriate to extrapolate clinical trial results for respiratory viruses to tuberculosis and for ethical reasons, randomised trial data cannot be collected on the effectiveness of a respirator intervention in HCWs for tuberculosis prevention. While the use of other types of efficacy data applied in the studies identified by this review have limitations, the results of the studies may still have validity. However, the underlying assumptions made in the analyses do need to be carefully considered.

Although economic evaluations have been conducted for the use of mask/respirator strategies in HCWs for tuberculosis, the way these evaluations should be used to inform policy needs to be carefully considered. Cost-effectiveness criteria should only be one part of the decision making process for any intervention and there may be specific cases where this factor should be given less weight. Economic evaluations would be of less relevance to a decision maker in situations and settings where a respirator would be used regardless of its cost-effectiveness, but they may be more useful in evaluating situations where these devices are not currently recommended in a given setting.

Future evaluations should report their results in such a way that would allow them to be easily compared to other non-pharmaceutical or pharmaceutical interventions (e.g. cost per quality-adjusted life year (QALY) gained). If an evaluation fails to report results in an appropriate format, this limits its use for decision making as it means that the value of mask/respirator interventions cannot be compared with spending on healthcare interventions targeted at other diseases [40]. However, in some cases the use of intermediate outcomes (e.g. cases prevented) can still be useful for comparing alternative ways of reducing specific infection events. This may provide useful information to decision makers.

Future analysts may also want to consider whether masks/respirators are cost-effective for use by HCWs in the seasonal influenza context, or if they are only cost-effective in extreme pandemic scenarios which may necessitate a stockpiled supply of masks/respirators.