The realistic performance achievable with mycobacterial automated culture systems in high and low prevalence settings

Background Diagnostic tests are generally used in situations with similar pre-test probability of disease to where they were developed. When these tests are applied in situations with very different pre-test probabilities of disease, it is informative to model the likely implications of known characteristics of test performance in the new situation. This is the case for automated Mycobacterium tuberculosis (MTB) liquid culture systems for tuberculosis case detection which were developed and are widely used in low burden settings but are only beginning to be applied on a large scale in high burden settings. Methods Here we model the performance of MTB liquid culture systems in high and low tuberculosis (TB) prevalence settings using detailed published data concentrating on the likely frequency of cross-contamination events. Results Our model predicts that as the TB prevalence in the suspect population increases there is an exponential increase in the risk of MTB cross-contamination events expected in otherwise negative samples, even with equivalent technical performance of the laboratories. Quality control and strict cross-contamination measures become increasingly critical as the burden of MTB infection among TB suspects increases. Even under optimal conditions the realistically achievable specificity of these systems in high burden settings will likely be significantly below that obtained in low TB burden laboratories. Conclusions Liquid culture systems can play a valuable role in TB case detection in laboratories in high burden settings, but laboratory workers, policy makers and clinicians should be aware of the increased risks, independent of laboratory proficiency, of cross-contamination events in high burden settings.


Background
Automated liquid culture for M. tuberculosis (MTB) is widely used in industrialized countries and increases the sensitivity and reduces the time required for culture based tuberculosis (TB) diagnosis. The use of this highly sensitive technique increases the importance of robust protocols and quality control to prevent and if this occurs detect laboratory cross-contamination. Protocols and quality control systems have been developed over the past years that are adequate to ensure excellent performance in settings with a low burden of TB. With the recent trends in TB diagnostics for high burden countries moving towards more modern technologies and scale-up of laboratories for the implementation of new methods, we and others in the field have expressed concern about the feasibility and viability of these changes in certain locations [1,2]. Laboratories in high burden TB areas that perform diagnosis based on conventional microscopy are now for example recommended to move towards innovative liquid culturing systems for case detection and molecular line probe assays for drug susceptibility testing. These new technologies are more sensitive and rapid, but the concern is that they are also more prone to laboratory cross-contamination potentially resulting in misdiagnosis. This is especially problematic when these tests are performed in laboratories with inadequate infrastructure and maintenance, or laboratories that lack the possibility of confirmatory tests. This is often the case in countries with a high TB burden. The consequences of a high contamination risk are numerous, but essentially results in bacterial culture results from true negative samples becoming positive. This issue forms the focus of our study. As the treatment of TB patients is long and complex with the potential for side effects, a false-positive MTB test is not trivial and places a significant burden on the TB suspect, the suspect's family and the health system in general [3,4].
Many studies have investigated the risks, causes and consequences of laboratory cross-contamination both in low-and high burden TB settings. However, the effect of the prevalence of positive MTB samples in the laboratory -which is strongly associated with the TB prevalence in the area -is generally not explicitly considered with respect to the absolute risk of cross-contamination of a true negative sample. The aim of this study is to model how cross-contamination risk, in laboratories with different performance, is influenced by the rate of samples positive for MTB processed in the laboratory for liquid culture.

Methods
First, a mathematical model is used to calculate the risk of a negative sample being tested false-positive based on various assumptions. It is assumed that the false positive rate (FPR) and false negative rate (FNR) are constant within a laboratory and depend only on the laboratory's performance. The variable factor under investigation is the prevalence of positive MTB samples (PSP) in the tested batch of samples, which influences the chance that a cross-contamination event will result in sample contamination with mycobacteria.
To determine the variables at stake a diagnostic 2 × 2 table is used (Table 1).
This theoretical framework provides us with the following variables: -False positive rate (number of false positive samples divided by total number of positive samples) is F = FP/p -False negative rate (number of false negative samples divided by total number of negative sample) is G = FN/q The risk of a negative sample being tested falsely positive (called R) is calculated by dividing the false-positive samples (the false-positive rate times the positive TB samples) by the sum of the false positive and true negative samples; in a formula this is: The number of true negative samples can be derived from the negative samples minus the false-negative samples (the false-negative rate times the negative samples). The new formula then becomes: This formula can be applied to a wide array of different laboratory settings, ranging from laboratories with a very low FPR, FNR and PSP to laboratories with a very high FPR, FNR and PSP. Regarding the variables required, it should be noted that data on FPR and PSP are readily available from various field studies. In contrast, the FNR can not easily be measured in practice, since the gold standard is liquid culture. Thus we are in fact modelling the performance of the gold standard test. If we assume an increased FNR (G) for liquid culture, the effect will be an increase of R in the formula. Here we prefer to utilize the most conservative model minimizing R. Therefore, we assume that the FNR is zero (G = 0), creating the following simplified formula: In reality of course there will be a small portion of falsenegatives, meaning that we slightly underestimate R with our formula.

Results
We used published study data from real-life laboratory settings to explore the implications of this formula. A study on culture cross-contamination rates in Dutch laboratories provides data representing situations with a low prevalence of TB and low levels of cross contamination [5]. In that study, all culture results of the TB national reference laboratory (NRL) were investigated over a seven-year period (1993 until 2000). The Dutch laboratories sent on average 1570 culture samples per year to the NRL, of which 21 were tested culture-positive by the NRL, leading to a mean PSP of 21/1570 = 0.0134 = 1.34%.
Since the formula reflects rates, the number of positive and negative samples (p and q) can be replaced by the rate of positive and negative samples compared to the total number of samples in a batch (PSP and 1-PSP). The NRL also typed all positive TB cultures using DNA fingerprinting. It was estimated, that the FPR in this time period was 2.4% on average, ranging from 3.9% at the start to 1.1% at the end of the investigation. It should be noted, that the very low FPR of 1.1% (F = 0.011) was reached after cross-contamination investigations had been performed in the Dutch laboratories and advice had been given about how to minimize the FPR over a number of years. We assume this extremely low FPR to represent the maximum practically achievable specificity of the test under ideal conditions. Using the above-mentioned data from the Dutch study, the risk of a negative sample becoming falsely positive can be calculated as followed: This is the risk of a negative sample being cross-contaminated with a positive sample for a given FPR and PSP (1.1% and 1.34%, respectively). When the FPR in the laboratory is kept constant, but the prevalence of positive MTB samples increases, the relation between positive sample prevalence and cross-contamination risk can be visualized as shown in figure 1 (dashed line). It can be seen that the risk rises exponentially when the positive sample prevalence rises (upper graph).
In another published study, a South-African laboratory reported data representing a laboratory with a very high FPR and PSP [6]. In that study an average of 1186 culture samples per year were tested in the laboratory of which 55% were culture positive (PSP = 0.55, p = 1186 * 0.55 = 652). The FPR was reported to be 7.3% at the beginning of the study and after implementation of specific interventions it decreased to 2.1%. In order to model a broad range of realistic situation in this case we use the initial high FPR (F = 0.073) reported in the formula, the risk of contamination of a true negative culture can be calculated as follows: With a constant FPR and increasing or decreasing the prevalence of positive TB samples in the South African laboratory, the relating contamination risks are depicted in figure 1 (solid line). Again, an exponential relation is seen between PSP and contamination risk.
Considering that a PSP of up to 55% does occur in high burden labs such as the South African laboratory, the part of the graph that is considered to represent a realistic situation is enlarged in figure 1 (lower graph). It is clear that when the FPR in a laboratory is kept constant, the risk of a sample getting cross-contaminated rises exponentially when the positive sample rate in the tested batch rises. Critically with a slightly raised FPR and high PSP this effect can quickly become a serious issue.

Discussion
When applying diagnostic tests developed for use in situations with a very different pre-test probability of disease, it is informative to model the likely implications on specific aspects of test performance in the new situation. In this report we model the likely performance of M. tuberculosis automated liquid culture systems in high-burden countries.
Our calculations of the expected performance of liquid culture in different settings show that a laboratory with optimized procedures that achieves a low FPR (e.g. 1.1%as for the Dutch laboratories) constitutes a low risk for a MTB negative TB suspect sample becoming contaminated, even when a high proportion of true positives are tested. When a contamination risk of 2.5% is taken as cutoff value (meaning that a higher contamination risk for a true negative sample is unacceptable, equivalent to a greater than 1 in 40 chance of a negative sample being falsely called positive), the PSP in a laboratory with a FPR of 1.1% due to cross-contamination needs to increase to over 60% in order for the contamination risk to rise above the cut-off value. Fortunately, a PSP of over 60% is unlikely in most countries. However, if the FPR is 7.3%as for the South African laboratory when liquid culture was initially introduced -, the 2.5% cut-off value for contamination risk of a negative sample is reached when the PSP gets to 26%. With a FPR of 2.1% the PSP must reach 55% before the contamination risk of a true negative sample reaches 2.5%.
Over the previous years detailed and careful work has accurately defined the primary concern with automated liquid culture systems for the detection of M. tuberculosis infection. Namely, as a consequence of the comparatively rapid growth achieved and their excellent sensitivity they are more sensitive to sample cross-contamination than previous methods. We focused on this issue in this report. Once this problem is recognized the impact can be assessed and minimized. This results in a marked increase in performance in the years following adoption of the methods as well as the need for sufficient throughput and investment to maintain laboratory skills. Of course even in the best laboratories some level of crosscontamination is inevitable and is this report we aimed to be quite conservative regarding the likely cross contamination rates. In industrialized countries with functioning quality systems and active reference laboratory support this level appears to be around 1% of all positive samples [5].
The cross-contamination risk for true negative samples is not only dependent on a laboratories performance (mostly measured by FPR), but is also strongly influenced by the MTB PSP in the laboratory. Unfortunately, as our calculations demonstrate, the implications for even relatively low levels of cross-contamination are much more serious in locations with high TB prevalence in suspects. These laboratories are ironically at present often the least equipped to fully implement all the required protocols. These precautions include: separating the processing of non-sterile specimens and specimen with a high risk of contamination (e.g. specimen of known positive cultures, proficiency-test samples or contaminated culture vials); using individual aliquots of buffer solution; uncapping only one tube at a time; leaving samples at rest for five minutes after centrifugation or mixing; decreasing the maximum number of samples processed in one batch; reenforcing hygiene protocols and clean laboratory practice among staff; and raising awareness and cautiousness of the possibility of false-positive results in general [7][8][9]. In order to monitor the performance of TB culture based diagnostic programs it is also useful to have the facility to genetically type a proportion of isolated strains, a service which is also frequently unavailable were it is most needed [10].
It has to be noted that in this study only the risk of cross-contamination leading to false-positive case detection is discussed. Like in most studies, the research articles used here focus on calculating the false-positive culture rate, i.e. the proportion of positive samples that is false-positive for the presence of MTB. However, they do not take into account the possible risk of a positive MTB culture sample being cross-contaminated with another positive sample, or the risk of a positive sample being contaminated with a (multi-)drug resistant MTB sample. The first event could potentially alter the Mycobacterium tuberculosis family type and lead to errors in determining epidemiological patterns. The latter could lead to a misdiagnosis of drug susceptibility of a sample and consequently to a patient receiving an incorrect treatment regimen. Thus, laboratory cross-contamination can also lead to erroneous detection of (multi-)drug resistance outbreaks or other epidemiological outbreaks. Since this study does not measure the risks of these other crosscontamination events, nor the effect that MTB positive sample rate might have on these risks, it is suggested that future studies also focus on these factors.

Conclusions
It can be concluded that even if future global laboratory interventions are able to optimize the culture procedures leading to similar low false-positive culture rates in all laboratories around the world, the mathematical model described in this study predicts that the risk of contamination of negative cultures will inevitably remain higher in areas with a higher TB prevalence. It is likely that alter-native liquid culture systems will have similar characteristics. Therefore, it is of critical importance that clinicians and policy makers involved in TB diagnostics remain aware of the possibility of laboratory cross-contamination, and take into account the regional differences when interpreting results and potential of these highly sensitive culture methods.