Performance of BioFire array or QuickVue influenza A + B test versus a validation qPCR assay for detection of influenza A during a volunteer A/California/2009/H1N1 challenge study

Influenza places a significant burden on global health and economics. Individual case management and public health efforts to mitigate the spread of influenza are both strongly impacted by our ability to accurately and efficiently detect influenza viruses in clinical samples. Therefore, it is important to understand the performance characteristics of available assays to detect influenza in a variety of settings. We provide the first report of relative performance between two products marketed to streamline detection of influenza virus in the context of a highly controlled volunteer influenza challenge study. Nasopharyngeal swab samples were collected during a controlled A/California/2009/H1N1 influenza challenge study and analyzed for detection of virus shedding using a validated qRT-PCR (qPCR) assay, a sample-to-answer qRT-PCR device (BioMerieux BioFire FilmArray RP), and an immunoassay based rapid test kit (Quidel QuickVue Influenza A + B Test). Relative to qPCR, the sensitivity and specificity of the BioFire assay was 72.1% [63.7–79.5%, 95% confidence interval (CI)] and 93.5% (89.3–96.4%, 95% CI) respectively. For the QuickVue rapid test the sensitivity was 8.5% (4.8–13.7%, 95% CI) and specificity was 99.2% (95.6–100%, 95% CI). Relative to qPCR, the BioFire assay had superior performance compared to rapid test in the context of a controlled influenza challenge study.


Background
Influenza remains a major global health concern with significant morbidity and mortality from seasonal infections and poses the potential for catastrophic pandemics [1]. In addition to the human cost, influenza infection also results in a tremendous economic burden with more than 20 million days of lost annual Open Access *Correspondence: mcilwain@stanford.edu 1 Department of Pathology, Stanford University School of Medicine, Stanford, CA, USA Full list of author information is available at the end of the article productivity and an estimated $11.2 billion in annual direct or indirect costs in the United States alone [2].
Distinguishing influenza infection from other acute respiratory conditions based on symptoms alone is difficult [3], but timely and accurate detection of influenza infection is a key component of both global disease surveillance monitoring, individual clinical case management, and clinical studies. A gold-standard for definitive diagnosis of influenza infections is a quantitative real-time polymerase chain reaction (qPCR) assay where the copies of viral nucleic acid in clinical samples are quantified and compared to a standard curve produced using the same strain [4,5]. However, standard qPCR approaches are time-consuming and require trained operators in a laboratory setting [5]. Other options for more rapid diagnosis are designed to overcome some of these limitations, sample-to-answer PCR-based systems and point-of-care immunoassay based rapid influenza diagnostic tests (RIDT) [6]. Sample-to-answer qPCR-based systems typically simplify workflow by automating sample nucleic acid extraction, using reagent cassettes that include pre-loaded positive and negative controls, and by generating easy to interpret outcome reports in about 1 h. RIDTs on the other hand are primarily lateral flow immunoassays that rely on antigen detection in biological samples with colorimetric readouts. Such tests provide results in 15 min or less, are simple to perform, and are generally at a lower cost than PCR-based methods [7]. However, the performance characteristics of each assay should always be a major consideration when choosing an influenza test.
Controlled human infection studies where healthy volunteers are challenged with influenza virus are important tools for the evaluation of novel influenza therapeutics [8][9][10]. Such studies also provide a rare opportunity to examine the performance characteristics of influenza diagnostics in a highly controlled setting. While multiple studies have reported the relative performance of different influenza detection methods [11], such assays are rarely (if ever) compared using samples obtained from a cohort of individuals all exposed to the same dose of a genetically identical virus at a known point in time. Samples obtained in a volunteer A/California/2009/H1N1 influenza challenge study were separately compared to test outcomes of a validated qPCR assay versus a sample-to-answer qPCR device (BioMerieux BioFire FilmArray RP) and a lateral flow immunoassay based RIDT (Quidel QuickVue Influenza A + B Test). Understanding the relative performance of these methods in this context has important implications for the design of future influenza challenge studies.

Influenza challenge study design and sample collection
During a volunteer influenza challenge study (Clinical-Trials.gov Identifier: NCT02918006) [12], 143 volunteers with low levels of pre-existing antibodies (HAI titers of less than 1:20) were challenged intranasally with an identical dose of A/California/2009/H1N1 challenge virus stock (a post-hoc qPCR assay determined each dose to be between 5.50 × 10 5 to 3.5 × 10 6 TCID 50 units [12]. This same dose of this same virus stock was sufficient to result in virus shedding in 67% of healthy volunteers in a prior dose-ranging study [13]. The presence of virus shedding was strongly correlated with the appearance of volunteer reported symptoms. Please see Liebowitz et al. [12] for additional details on symptomology, inclusion criteria, demographics of volunteers, and other information. Nasopharyngeal (NP) swabs (Quidel, San Diego, CA) were collected up to twice daily by inserting the swab into the nasopharynx and turning before placement in a supplied tube containing universal virus transport medium. NP swab specimens were used for the detection of virus shedding by a validated qPCR assay (qPCR) throughout the study. Starting on day 4 post virus challenge, matched NP swabs were collected from individual volunteers at each time point and the presence of virus was evaluated using a qPCR assay and either the BioMerieux BioFire FilmArray RP (BioFire) or Quidel QuickVue Influenza A + B Test (rapid test).  Table S1). To evaluate the quality of the NP swab samples, a separate PCR reaction was performed to detect the Human RNPase P gene. Detection of this gene confirms that the swab sample is of sufficient quality that cell-associated virus can be detected and quantified and acts as an internal control for any possible PCR inhibitors in the swab sample. A one-step quantitative RT-probe Hydrolysis kit, Ambion AgPath-ID ™ One-Step kit (Thermo Fisher, Waltham, MA) was used in the PCR reaction following the manufacturer's instructions. The final concentration of primers was 0.8 µM and 0.2 µM for the probe. 5µL of the extracted material was used in each reaction. PCR conditions using an Applied Biosystems ABI 7500 PCR system (Thermo Fisher, Waltham, MA) were as follows: 50.0 °C for 30 min; 95.0 °C for 10 min; 45 cycles of 95 °C, 15 s followed by 55 °C for 34 s.

Quantitative real-time PCR (qPCR) assay and qualification
To develop a standard curve for quantitation, the HA gene sequence was obtained from GenBank (KU933485.1) for A/California/07/2009. A forward primer at positions 1-25 (ATG AAG GCA ATA CTA GTA GTT CTG C) with a 5′ T7 promoter and a reverse primer at positions 1702-1673 (TTA AAT ACA TAT TCT ACA CTG TAG AGA CCC ) were used to generate a transcript of 1702 base pairs in a one-step RT PCR reaction. The product was run on a 1% gel and the band was purified with the Zymoclean Gel DNA Recovery Kit (Zymo Research, Irvine, CA). A Megascript T7 transcription kit (Thermo Fisher, Waltham, MA) was used to generate an RNA transcript. The transcript was cleaned up using the Qiagen RNeasy Mini Kit (Qiagen, Hilden, Germany), run on a 1% agarose gel to confirm the size, and then quantified by multiple readings on a Nanodrop. The concentration and copy number were calculated from the OD readings. Standard curves were generated by freshly diluting transcripts tenfold from 4.0 × 10 6 to 4.0 copies/ µL (2.0 × 10 7 to 20.0 copies/reaction) before each run. Standard curves were shown to have an average efficiency of 100% based on the slope of the curves. A positive control of extracted A/California/04/2009 H1N1 virus was run in the reaction over 20 times by two technicians over a 5-week period to obtain data to set an acceptance range based on 2 standard deviations of the average quantity obtained in the assay.
To confirm specificity towards A/California H1N1, eight different influenza A and B viruses (Additional file 2: Table S2) were tested in the assay. Only A/California H1N1 specific isolates were detected in the assay. Additionally, to demonstrate the specificity of the primers and probe, the PCR product of the positive virus control was run on a 2% agarose gel and assessed for a band at 177 bp to confirm that only the targeted portion of the gene was amplified (Additional file 3: Figure S1). Intraassay precision and intermediate precision was determined to have a coefficient of variance (CV) of 11% and 25%, respectively.
The limit of detection (LOD) for the assay was determined from running the standard reference in two-fold dilutions surrounding the lower end of the standard curve in replicates of 20. The LOD was then calculated as the concentration where 95% of the reference standard dilutions gave a positive response (Ct ≤ 40). The LOD was calculated to be 16 copies/reaction. For purposes of comparison with BioFire or rapid test, qPCR samples above the LOD were considered positive, samples below the LOD were considered to be negative.

BioFire FilmArray
NP swab samples were loaded into the Biofire FilmArray respiratory panel cassette according to the manufacturer's instructions and analyzed using the BioFire FilmArray Multiplex PCR System (BioMerieux, Marcyl'Étoile, France) [14]. This device uses a fully automated procedure for nucleic acid purification, amplification, multiplexed PCR, and melting analysis, and generates a report with binary outcomes for various respiratory pathogens. Within assay cassettes are two positive quality controls, the first is an RNA process control to verify successful extraction and reverse transcription, the second is an independent DNA control to verify a successful PCR reaction occurred [14]. All samples analyzed in this study passed both quality controls, and those that were positive for influenza A were considered positive in this report. Samples that were positive for targets other than influenza A were excluded from analysis while all other samples were considered negative.

Rapid influenza test
NP swab samples were applied to the QuickVue Influenza A + B Test (Quidel, San Diego, CA) using the manufacturer's instructions [15]. Colorimetric tests were read by eye to determine positive or negative results as per protocol in the test kit insert. All samples reported in this analysis properly displayed positive procedural control lines, indicating successful execution of the test kit protocol [15].

Statistical analysis
Sensitivity (true positive rate) was calculated as the ratio of true positive results divided by the sum of true positive and false negative results [(true positive)/(true positive + false negative)]. The specificity (true negative rate) was calculated by dividing the number of true negative results by the total number of true negative plus falsepositive results [(true negative)/(true negative + false positive)]. The positive and negative predictive values were calculated as true positive divided by all positive results and true negative divided by all negative results, respectively. The confidence interval was calculated using the epiR package [16]. Differences in qPCR copy number values between samples determined to be true positive and false negative by BioFire or rapid test were calculated using the Wilcoxon signed-rank test on log 10 values.

Results
During a recent volunteer influenza challenge study, 143 healthy volunteers were challenged intranasally with A/California/2009/H1N1 virus [12]. Nasopharyngeal (NP) swabs were routinely collected and tested for the presence of influenza virus as part of the clinical conduct of this study. Multiple NP swabs were collected and used for analysis with either the BioMerieux Bio-Fire FilmArray RP (BioFire) or Quidel QuickVue Influenza A + B Test (rapid test) in addition to the validated qPCR assay, thus providing an opportunity to compare the relative performance of these tests.
To facilitate the analysis described here, the goldstandard qPCR assay was considered to be diagnostically accurate for the detection of influenza A virus in all samples. All values above the LOD by qPCR were recorded as positive, and all samples below the LOD were recorded as negative.
Sensitivity, or true positive rate (TPR), was calculated as the number of TP results divided by the sum of TP and FN results. Specificity, or true negative rate (TNR), was calculated as the number of true negatives (TN) divided by the sum of TN and false positives (FP). The 95% confidence interval was calculated using the epiR package in R.

Discussion
Rapid and simplified methods for influenza virus detection can be valuable tools, but it is important to understand how these tests compare to other assays in different settings. One important setting is during volunteer influenza challenge studies, where detection of virus shedding may be used as a trigger for treatment or other protocol steps. Here the performance of BioFire, a simplified sample-to-answer qPCR-based detection method, and QuickVue Influenza A + B Test rapid test, a colorimetric immunoassay, were assessed by comparing their diagnostic results to a validated qPCR assay using matched  BioFire is an arrayed multiplexed qPCR-based device delivering binary results (positive or negative) for targets included in its assay cassettes. Although BioFire had a high specificity of 93.5%, sensitivity of BioFire was lower at 72.1%. This lower sensitivity may reflect a higher LOD by the BioFire assay for the A/California/2009/H1N1 influenza challenge strain compared to qPCR in the context of this study. A higher LOD could result from differences in sample prep, primer sets, instrumentation, or programmed reporting threshold. Our calculated sensitivity of 72.1% for BioFire was only slightly lower than the 73-89% sensitivity range reported in other studies [17][18][19]. Taken together, while such assays are simpler to use and offer a quicker turnaround compared to standard qPCR, this may come at the expense of slightly decreased sensitivity.
Rapid influenza diagnostic tests are well known to have decreased sensitivity compared to qPCR [7]. While some studies have reported sensitivities as high as 63-71% [20][21][22], others report much lower values in the range of 26-33% [23,24]. In our analysis, the rapid test sensitivity was very low, at only 8.5% relative to qPCR. This wide range of sensitivities reported across studies may be attributed to differences in virus strain or magnitude of antigen load, at least in part, by the day post-infection in which samples are collected. Additionally, as a colorimetric test, the qualitative results are susceptible to greater variation due to biases in operator readings compared to the quantitative driven results of qPCR and BioFire. Nevertheless, the specificity of the rapid test is very high. Our study found a specificity of 99.2% for this test, in line with other reports of 96-100% specificity [20,21,23,24]. Therefore, despite poor sensitivity, a positive rapid test, when present, is a strong indicator that a sample will also be positive by qPCR.
For operational reasons, this study was limited to examining samples starting at 4 days post-viral challenge, corresponding to approximately three days postsymptom onset in challenge models [8]. It is important to understand that cohorts of community-acquired infection self-identifying as symptomatic are fundamentally different from those of volunteer influenza challenge studies, and that the results of this study may not be extensible to community-acquired infection settings. For example, in other studies that examine the performance of influenza detection assays during community-acquired infection [17,20,21,23,24], NP samples were collected immediately when patients met a set of subjective symptom criteria. Symptom-based self-selection may bias for the inclusion of individuals in those studies with higher magnitudes of viral titer and antigen load compared to a controlled challenge study in which all study participants are automatically assessed at predetermined time points. These differences may explain the reduced overall performance of the rapid test in this study compared to prior studies. The requirement of higher viral loads to trigger a positive rapid test in this study is suggested by significantly higher qPCR copy numbers for rapid test TP samples versus rapid test FN samples, an effect that was less pronounced for BioFire. Nevertheless, the data reported here is important for our broader understanding of the performance of these diagnostic tools, especially for those considering their use in future influenza challenge studies. If available from future influenza challenge studies, an examination of samples taken over the entire course of virus shedding period, including from earlier time points post-inoculation would be worthwhile to determine the relationship between day post-infection, viral load, and performance of both rapid tests and sample-to-answer molecular tests for influenza A.

Conclusion
In summary, we have compared the performance of two simplified influenza A detection methods to a validated qPCR assay in a controlled A/California/2009/H1N1 challenge study. In this setting, BioFire closely reflected virus shedding detected by qPCR, while the rapid test did not. While not without limitations, integration of sample-to-answer influenza tests such as BioFire into environments where standard qPCR assays are impractical stands to greatly improve detection of influenza over antigen-based rapid tests alone.