Real-world comparison of two molecular methods for detection of respiratory viruses
Virology Journalvolume 8, Article number: 332 (2011)
Molecular polymerase chain reaction (PCR) based assays are increasingly used to diagnose viral respiratory infections and conduct epidemiology studies. Molecular assays have generally been evaluated by comparing them to conventional direct fluorescent antibody (DFA) or viral culture techniques, with few published direct comparisons between molecular methods or between institutions. We sought to perform a real-world comparison of two molecular respiratory viral diagnostic methods between two experienced respiratory virus research laboratories.
We tested nasal and throat swab specimens obtained from 225 infants with respiratory illness for 11 common respiratory viruses using both a multiplex assay (Respiratory MultiCode-PLx Assay [RMA]) and individual real-time RT-PCR (RT-rtPCR).
Both assays detected viruses in more than 70% of specimens, but there was discordance. The RMA assay detected significantly more human metapneumovirus (HMPV) and respiratory syncytial virus (RSV), while RT-rtPCR detected significantly more influenza A. We speculated that primer differences accounted for these discrepancies and redesigned the primers and probes for influenza A in the RMA assay, and for HMPV and RSV in the RT-rtPCR assay. The tests were then repeated and again compared. The new primers led to improved detection of HMPV and RSV by RT-rtPCR assay, but the RMA assay remained similar in terms of influenza detection.
Given the absence of a gold standard, clinical and research laboratories should regularly correlate the results of molecular assays with other PCR based assays, other laboratories, and with standard virologic methods to ensure consistency and accuracy.
Respiratory viruses are common causes of human disease. Molecular detection techniques have allowed previously known viruses to be more reliably identified and new viruses to be discovered[1–10]. Molecular techniques such as real-time RT-PCR (RT-rtPCR) can be performed for each individual virus, or they can be combined into a multiplex RT-rtPCR assay. Other molecular methods that can target a single analyte or multiple viruses use a variety of nucleic acid detection based strategies, including microarrays, mass spectrometry , spectrally distinct bead microarrays, capillary electrophoresis , or microsphere flow cytometry. The performance of molecular diagnostic methods, either RT-rtPCR or multiplex assays, typically has been evaluated by comparing them with conventional direct fluorescent antibody (DFA) or culture, with few direct comparisons reported between molecular assays. Moreover, there are few reports of laboratories collaboratively comparing methods between institutions. In this study, we compared a multiplex respiratory virus panel (Respiratory MultiCode Assay [RMA], EraGen Biosciences and U. Wisconsin-Madison) with RT-rtPCR (Vanderbilt University) for the detection of 11 common respiratory viruses.
Nasal and throat swabs were collected from children ≤ 12 months old who were admitted to the Monroe Carell Jr. Children's Hospital at Vanderbilt from September 1, 2005 through May 31, 2006 with acute respiratory illness, as part of the Tennessee Children's Respiratory Initiative . All specimens collected during this period were included in the study. Within four hours of collection, the samples were divided into aliquots, lysis buffer added, and specimens were frozen at -80°C. Separate unthawed aliquots were sent to the research laboratories at Vanderbilt University and University of Wisconsin-Madison for RT-rtPCR and RMA testing, respectively.
Real-time RT-PCR testing
Single RT-rtPCR experiments were performed for the following 11 viruses: respiratory syncytial virus (RSV), influenza A and B, human metapneumovirus (HMPV), human rhinovirus (HRV), parainfluenza 1, 2 and 3 (PIV1, PIV2, PIV3), and human coronaviruses Netherlands (NL-63), OC43, and 229E. RNA was extracted on an automated instrument (MagNApure Total Nucleic Acid extraction kit; Roche Applied Science), and refrozen at -80°C until each individual RT-rtPCR was performed. Primers and probes for HCoV-NL63, OC43, 229E, RSV, HMPV and HRV were obtained from the published literature[17–22]. Influenza primer and probe sequences were provided by Steve Lindstrom, Centers for Disease Control and Prevention . Primers and probes for PIV1, 2 and 3 were designed using Primer Express version 2.0 (Applied Biosystems). Primer and probe sequences, concentration, and annealing temperatures are listed in Table 1. Twenty-five-μL reaction mixtures containing 5 μL of specimen RNA were tested using the QuantiTect Probe RT-PCR kit (QIAGEN) on a Smart Cycler II (Cepheid). Cycling conditions were 30 min at 50°C, 15 min at 94°C, and 45 cycles of 8 seconds at 94°C and 60 seconds at 55-66°C (Table 1). All RT-rtPCR assays were optimized and characterized using RNA runoff transcripts and were capable of detecting < 50 RNA transcript copies/reaction (data not shown). All samples were tested by a commercial RT-rtPCR assay for human beta-actin mRNA (Applied Biosystems) to ensure RNA integrity.
RMA is a multitarget, high throughput technology that integrates multiplex PCR and microsphere flow cytometry [15, 24, 25]. While the Luminex technology can detect up to 80 targets in one reaction, this version of the assay included 19 viral targets. We report results for the same 11 viruses tested by the RT-rtPCR assays. Nucleic acid was extracted with Trizol reagents. The RMA assay was performed with primers and conditions as previously described, including the addition of an external insect virus spiked into each specimen to control for RNA extraction, RT, and PCR .
Retesting with RT-rtPCR and RMA
Both sites were blinded to the results from the other laboratory until all data were analyzed. In order to address potential weaknesses indicated by discordant results between the two assays in the first round of comparison, both RT-rtPCR and RMA testing were modified for a second round of testing. The first round primers and probes for RSV and HMPV [19, 21] were changed for the RT-rtPCR assay and (Klemenc et al, unpublished data) and all samples were retested for these two viruses. In the RMA assay, the primers and probe for influenza A were modified and the cDNA synthesis protocol was changed from Promega reagents to the Applied Biosystems High-Capacity cDNA Kit.
Demographic and clinical variables of the study sample are described as frequencies and percent or median and interquartile ranges [IQR] as appropriate. The RMA and RT-rtPCR assays were compared using the test for inter-rater agreement (kappa coefficient). Kappa coefficient was calculated for any comparison in which the value of all data cells was ≥ 3, with a kappa ≥ 0.75 considered to indicate good agreement. Bootstrapping was used to estimate 95% confidence limits of kappa. Cycle threshold (Ct) numbers for influenza and RSV were compared between groups of specimens using the t-test.
A total of 225 infants hospitalized with respiratory symptoms were enrolled from September 2005 through May 2006. Samples from 222 children were available for study in both sites; three specimens were only available for RMA testing. Study children were 57% male with a median age of 13 weeks [IQR 7-33 weeks]. The admitting diagnoses of these infants were bronchiolitis (58%), upper respiratory infection (33%), and other (9%).
Rates of viral detection were compared using both assay methods. Restricting the analysis to the 11 viruses tested with both methods, at least one virus was detected in 174 (78.3%) samples by RMA and in 163 (73.4%) samples by RT-rtPCR. A total of 194 and 196 viruses were identified in these samples by RMA and RT-rtPCR, respectively (Table 2).
In the first round of testing, RMA detected more RSV (n = 98, 44%) than RT-rtPCR (n = 78, 35%). The kappa coefficient between the two tests was 0.66. RMA also differentiated between RSV sub-types A and B. Of the RSV positive samples detected by RMA, 67 were RSV A and 32 were RSV B (one sample had both RSV A and RSV B). Of the 67 RSV A identified through RMA, RT-rtPCR detected RSV in 65 (97%) of them. However, of the 42 RSV B detected by RMA, RT-rtPCR only detected RSV B in 6 (14.2%) samples.
In the second round, the primers and probes for the RSV RT-rtPCR assay were redesigned based on further analysis of published viral sequences . Using these new primers and probes, RSV was identified in 107 (49%) samples with RT-rtPCR. The inter rater agreement for RSV between the repeat RT-rtPCR and the first RMA was 0.86. Of the 67 RSV A and 32 RSV B detected by the RMA assay, the repeat sample analysis detected RSV in 66 and 28 respectively. 12 samples were positive for RSV with repeat RT-rtPCR testing but negative for RSV with RMA testing; the mean Ct for these specimens was higher than the mean Ct for specimens detected by both assays (34.54 ± 3.18 vs. 28.58 ± 3.71, p < 0.001).
In the first round, the RT-rtPCR detected influenza A in 25 (11.2%) samples while the RMA detected influenza A in 7 (3.1%) samples. RT-rtPCR detected influenza B in 1 sample while RMA detected influenza B in a different sample. In the second round of testing, the influenza A RMA primers and probes were redesigned. Compared to 25 samples positive for influenza A by rRT-PCR, the repeat RMA assay detected 5 positive samples; all 5 were also detected by the RT-rtPCR assay. The mean Ct of the 5 influenza A specimens detected by the RMA assay was lower than the mean Ct of all influenza A specimens (30.02 ± 1.98 vs. 34.99 ± 1.88, p < 0.001).
RT-rtPCR detected HRV in 63 (28.2%) samples as compared to the RMA, which detected HRV in 56 (25.1%) samples (kappa coefficient 0.78). In two of the discordant samples, RT-rtPCR detected HRV while the RMA detected enterovirus.
During the first round of testing, RMA detected HMPV in 8 (3.5%) samples while RT-rtPCR detected HMPV in 2 (0.9%) samples. In the second round using redesigned RT-rtPCR HMPV primers and probes, HMPV was detected in 7 samples, 5 of which were concordant with the results of the first RMA assay. The first RMA assay detected HMPV in 2 additional samples that were not detected by the repeat RT-rtPCR assay, while one sample positive for HMPV in the first RMA was negative in the second set of tests.
NL63 was detected in 8 (3.5%) samples through RMA and in 6 (2.7%) samples through RT-rtPCR. HCoV OC43 was detected in 5 (2.2%) samples through RMA and in 5 (2.3%) samples through RT-rtPCR. HCoV 229E was not detected in any sample by either assay.
PIV1 was detected in 10 (4.4%) samples through RMA and in 13 (5.8%) samples through RT-rtPCR. PIV 2 was detected in 1 (0.5%) sample through RMA and in 1 (0.5%) sample through RT-rtPCR. No PIV3 was detected by either method.
Test/retest of respiratory viruses using the RMA assay
The comparison of the first and second run of RMA assay is summarized in Table 2. The most notable discrepancies between runs were for HRV and RSV.
The sensitivity and specificity of multiplex viral diagnostic assays have generally been compared to viral culture and DFA, instead of more sensitive RT-rtPCR methods [2, 26–28]. Comparison of molecular methods with other sensitive molecular techniques, such as RMA with RT-rtPCR, is appropriate to determine whether the sensitivity for each analyte is sufficient for clinical or research use. Although both assays performed well for selected viruses in this study, the first round of comparison revealed differences in the rates of detection for specific viruses (Table 2). The RMA assay detected significantly more HMPV and RSV, while the RT-rtPCR assay detected more influenza A and HRV. Redesigning primers led to substantial improvements in the detection of HMPV and RSV by rRT-PCR, while there was little improvement in the performance of RMA for influenza virus with the change in primers. This discordance could have been due to sequence variation; there was insufficient residual specimen to sequence the viruses. However, both influenza A assays targeted regions of the matrix gene that are highly conserved among GenBank sequences (data not shown). The mean Ct of the influenza and RSV specimens detected only by the RT-rtPCR assay was significantly higher than those detected by the RMA assay, suggesting that the discordance may have been due to the sensitivity of the two assays for specimens with a lower viral load.
One recent study compared a commercial version of the RMA known as PLx-Respiratory Viral Assay (PLx-RVP) to RT-rtPCR and found good concordance, but only tested a portion of the PLx-RVP negative samples . The RMA and PLx-RVP differ both in primer sequences, and in the method used to extract RNA and perform reverse transcription. Both of these factors are important determinants of assay sensitivity. Given that RNA viruses are prone to mutation, currently circulating field strains of these viruses should be regularly monitored for sequence divergences that would affect primers and probes used in PCR-based assays. This is particularly true for influenza, which undergoes considerable variation every year. Molecular assays are often targeted to conserved internal genes, but this is based on known sequences. Further, due to segment reassortment and antigenic shift, new internal gene segments may arise among influenza viruses infecting humans. For multiplex assays such as the RMA, inclusion of two probe sets that target two different parts of the viral genome might eliminate the risk that a single mutation would lead to nondetection.
Our study has some limitations. First, we did not use identical extraction methods and primers in the RT-rtPCR and RMA assays, and therefore discrepant results could have been due to differences in these processes or primer design. However, this was an intentional choice, since the specific goal of this study was to compare two "real-world" methods. Both laboratories have published extensively in the area of respiratory virus diagnostics and epidemiology, and have evaluated their independent methods. This study shows that collaboration between groups offers a valuable tool to assess assay performance impartially. Second, we did not use a gold standard test to confirm presence or absence of virus in cases of discrepancy between the two assays. We explored the option of sequencing of the discordant samples to detect sequence differences or false positives, but there was not enough residual specimen to allow this. Nonetheless, we repeated the testing of viruses that had major discrepancies with the assumption that if the repeats were positive, it provided reassurance that the positive samples were true positives. Third, the RT-rtPCR was only repeated for HMPV and RSV due to limited sample; there could have been discordances for other viruses. Fourth, the RMA assay was repeated with altered influenza primers while other primers were unchanged; it is possible that a change in influenza primers had an impact on the overall performance of the assay. One important consideration for any multiplex assay is the need to re-optimize for all targets when a change is made in any of the primers used. Finally, the results were not evaluated by DFA or culture, though the objective was to directly compare sequence-based methods.
In summary, for most viruses tested, there was good agreement between the two assays. However, discrepancies highlight the need for similar comparisons between molecular assays on a routine basis, so that weaknesses can be promptly identified and corrected. While investigators may be reluctant to make comparisons that might suggest deficiencies of particular methods, collaborative studies such as ours are critical to advance these diagnostic methodologies. Two other recent studies comparing molecular methods identified similar discrepancies [29, 31]. In addition to clinical specimens validated by culture, respiratory virus proficiency panels distributed by quality control programs like the European Quality Control for Molecular Diagnostics (QCMD) , CDC, or College of American Pathologists, or commercial proficiency panels (e.g., ZeptoMetrix NATtrol™ Respiratory Validation Panels) provide an external quality control. Clearly, the overall comparison between molecular methods should consider the performance of each individual analyte. The goal of this study was not to determine which is the "best" assay; the choice of assay for a given purpose depends on many factors, including cost, available equipment and expertise, sample volume, and others. Multiplex assays may offer substantial savings in time and cost due to multiple analytes, and are generally preferred for clinical testing. Comparison of molecular diagnostic methods to culture and DFA is useful to establish validity, and ensure that cultivable viruses are correctly identified by sequence-based technology. However, our data show that molecular diagnostic methods also must be compared to other molecular techniques to assess performance, particularly because of the superior sensitivity of the molecular methods compared to culture and DFA.
Mahony JB: Detection of respiratory viruses by molecular methods. Clin Microbiol Rev 2008, 21: 716-747. 10.1128/CMR.00037-07
Weinberg GA, Erdman DD, Edwards KM, Hall CB, Walker FJ, Griffin MR, Schwartz B: Superiority of reverse-transcription polymerase chain reaction to conventional viral culture in the diagnosis of acute respiratory tract infections in children. J Infect Dis 2004, 189: 706-710. 10.1086/381456
Esper F, Weibel C, Ferguson D, Landry ML, Kahn JS: Evidence of a novel human coronavirus that is associated with respiratory tract disease in infants and young children. J Infect Dis 2005, 191: 492-498. 10.1086/428138
Ginocchio CC, Manji R, Lotlikar M, Zhang F: Clinical evaluation of NucliSENS magnetic extraction and NucliSENS analyte-specific reagents for real-time detection of human metapneumovirus in pediatric respiratory specimens. J Clin Microbiol 2008, 46: 1274-1280. 10.1128/JCM.01567-07
Kesebir D, Vazquez M, Weibel C, Shapiro ED, Ferguson D, Landry ML, Kahn JS: Human bocavirus infection in young children in the United States: molecular epidemiological profile and clinical characteristics of a newly emerging respiratory virus. J Infect Dis 2006, 194: 1276-1282. 10.1086/508213
Arden KE, McErlean P, Nissen MD, Sloots TP, Mackay IM: Frequent detection of human rhinoviruses, paramyxoviruses, coronaviruses, and bocavirus during acute respiratory tract infections. J Med Virol 2006, 78: 1232-1240. 10.1002/jmv.20689
Mackay IM: Human rhinoviruses: the cold wars resume. J Clin Virol 2008, 42: 297-320. 10.1016/j.jcv.2008.04.002
van der Hoek L, Pyrc K, Jebbink MF, Vermeulen-Oost W, Berkhout RJ, Wolthers KC, Wertheim-van Dillen PM, Kaandorp J, Spaargaren J, Berkhout B: Identification of a new human coronavirus. Nat Med 2004, 10: 368-373. 10.1038/nm1024
van den Hoogen BG, de Jong JC, Groen J, Kuiken T, de Groot R, Fouchier RA, Osterhaus AD: A newly discovered human pneumovirus isolated from young children with respiratory tract disease. Nat Med 2001, 7: 719-724. 10.1038/89098
Allander T, Tammi MT, Eriksson M, Bjerkner A, Tiveljung-Lindell A, Andersson B: Cloning of a human parvovirus by molecular screening of respiratory tract samples. Proc Natl Acad Sci USA 2005, 102: 12891-12896. 10.1073/pnas.0504666102
Wang D, Coscoy L, Zylberberg M, Avila PC, Boushey HA, Ganem D, DeRisi JL: Microarray-based detection and genotyping of viral pathogens. Proc Natl Acad Sci USA 2002, 99: 15687-15692. 10.1073/pnas.242579699
Briese T, Palacios G, Kokoris M, Jabado O, Liu Z, Renwick N, Kapoor V, Casas I, Pozo F, Limberger R, et al.: Diagnostic system for rapid and sensitive differential detection of pathogens. Emerg Infect Dis 2005, 11: 310-313.
Merante F, Yaghoubian S, Janeczko R: Principles of the xTAG respiratory viral panel assay (RVP Assay). J Clin Virol 2007,40(Suppl 1):S31-35.
Roh KH, Kim J, Nam MH, Yoon S, Lee CK, Lee K, Yoo Y, Kim MJ, Cho Y: Comparison of the Seeplex reverse transcription PCR assay with the R-mix viral culture and immunofluorescence techniques for detection of eight respiratory viruses. Ann Clin Lab Sci 2008, 38: 41-46.
Lee WM, Grindle K, Pappas T, Marshall DJ, Moser MJ, Beaty EL, Shult PA, Prudent JR, Gern JE: High-throughput, sensitive, and accurate multiplex PCR-microsphere flow cytometry system for large-scale comprehensive detection of respiratory viruses. J Clin Microbiol 2007, 45: 2626-2634. 10.1128/JCM.02501-06
Hartert TV, Carroll K, Gebretsadik T, Woodward K, Minton P: The Tennessee Children's Respiratory Initiative: Objectives, design and recruitment results of a prospective cohort study investigating infant viral respiratory illness and the development of asthma and allergic diseases. Respirology 15: 691-699.
Fouchier RA, Hartwig NG, Bestebroer TM, Niemeyer B, de Jong JC, Simon JH, Osterhaus AD: A previously undescribed coronavirus associated with respiratory disease in humans. Proc Natl Acad Sci USA 2004, 101: 6212-6216. 10.1073/pnas.0400762101
Fry AM, Chittaganpitch M, Baggett HC, Peret TC, Dare RK, Sawatwong P, Thamthitiwat S, Areerat P, Sanasuttipun W, Fischer J, et al.: The burden of hospitalized lower respiratory tract infection due to respiratory syncytial virus in rural Thailand. PLoS One 5: e15098.
Kuypers J, Wright N, Morrow R: Evaluation of quantitative and type-specific real-time RT-PCR assays for detection of respiratory syncytial virus in respiratory specimens from children. J Clin Virol 2004, 31: 123-129. 10.1016/j.jcv.2004.03.018
Lu X, Holloway B, Dare RK, Kuypers J, Yagi S, Williams JV, Hall CB, Erdman DD: Real-time reverse transcription-PCR assay for comprehensive detection of human rhinoviruses. J Clin Microbiol 2008, 46: 533-539. 10.1128/JCM.01739-07
Maertzdorf J, Wang CK, Brown JB, Quinto JD, Chu M, de Graaf M, van den Hoogen BG, Spaete R, Osterhaus AD, Fouchier RA: Real-time reverse transcriptase PCR assay for detection of human metapneumoviruses from all known genetic lineages. J Clin Microbiol 2004, 42: 981-986. 10.1128/JCM.42.3.981-986.2004
van Elden LJ, van Loon AM, van Alphen F, Hendriksen KA, Hoepelman AI, van Kraaij MG, Oosterheert JJ, Schipper P, Schuurman R, Nijhuis M: Frequent detection of human coronaviruses in clinical specimens from patients with respiratory tract infection by use of a novel real-time reverse-transcriptase polymerase chain reaction. J Infect Dis 2004, 189: 652-657. 10.1086/381207
Belongia EA, Irving SA, Waring SC, Coleman LA, Meece JK, Vandermause M, Lindstrom S, Kempf D, Shay DK: Clinical characteristics and 30-day outcomes for influenza A 2009 (H1N1), 2008-2009 (H1N1), and 2007-2008 (H3N2) infections. JAMA 304: 1091-1098.
Marshall DJ, Reisdorf E, Harms G, Beaty E, Moser MJ, Lee WM, Gern JE, Nolte FS, Shult P, Prudent JR: Evaluation of a multiplexed PCR assay for detection of respiratory viral pathogens in a public health laboratory setting. J Clin Microbiol 2007, 45: 3875-3882. 10.1128/JCM.00838-07
Nolte FS, Marshall DJ, Rasberry C, Schievelbein S, Banks GG, Storch GA, Arens MQ, Buller RS, Prudent JR: MultiCode-PLx system for multiplexed detection of seventeen respiratory viruses. J Clin Microbiol 2007, 45: 2779-2786. 10.1128/JCM.00669-07
Atmar RL, Baxter BD, Dominguez EA, Taber LH: Comparison of reverse transcription-PCR with tissue culture and other rapid diagnostic assays for detection of type A influenza virus. J Clin Microbiol 1996, 34: 2604-2606.
Fox JD: Nucleic acid amplification tests for detection of respiratory viruses. J Clin Virol 2007,40(Suppl 1):S15-23.
Mackay IM: Real-time PCR in the microbiology laboratory. Clin Microbiol Infect 2004, 10: 190-212. 10.1111/j.1198-743X.2004.00722.x
Arens MQ, Buller RS, Rankin A, Mason S, Whetsell A, Agapov E, Lee WM, Storch GA: Comparison of the Eragen Multi-Code Respiratory Virus Panel with conventional viral testing and real-time multiplex PCR assays for detection of respiratory viruses. J Clin Microbiol 48: 2387-2395.
Mackie PL: The classification of viruses infecting the respiratory tract. Paediatr Respir Rev 2003, 4: 84-90. 10.1016/S1526-0542(03)00031-9
Li H, McCormac MA, Estes RW, Sefers SE, Dare RK, Chappell JD, Erdman DD, Wright PF, Tang YW: Simultaneous detection and high-throughput identification of a panel of RNA viruses causing respiratory tract infections. J Clin Microbiol 2007, 45: 2105-2109. 10.1128/JCM.00210-07
Wallace PS: Linkage between the journal and Quality Control Molecular Diagnostics (QCMD). J Clin Virol 2003, 27: 211-212. 10.1016/S1386-6532(03)00159-8
This work was supported by Vanderbilt CTSA grant NIH/NCRR UL1-RR024975, Thrasher Research Fund Clinical Research Grant (TVH), NIH AI-077930 (TVH), NIH U01 HL-072471 (TVH), and NIH AI-082417 (JVW).
James Gern is on the scientific advisory board for and has stock options for 3 V Biosciences, has consulted for and has stock options for EraGen Biosciences, has consulted for Synairgen and Centocor, and has received research support from AstraZeneca and Merck. Kathryn M. Edwards receives research funding from sanofi-pasteur, Wyeth, Novartis, and CSL. Marie R Griffin has received grant support from MedImmune and Wyeth. John V. Williams has served as a consultant for MedImmune, Novartis and serves on the scientific advisory board of Quidel.
AA and EKM performed RT-rtPCR and drafted the manuscript. TP and WML performed RMA testing. AA and TG performed the statistical analysis. AA, JEG, TVH, KME, MRG, and JVW conceived of the study, participated in its design and coordination, and helped to draft the manuscript. All authors read and approved the final manuscript.