Skip to main content

Pneumonia scoring systems for severe COVID-19: which one is better

Abstract

Purpose

To investigate the predictive significance of different pneumonia scoring systems in clinical severity and mortality risk of patients with severe novel coronavirus pneumonia.

Materials and methods

A total of 53 cases of severe novel coronavirus pneumonia were confirmed. The APACHE II, MuLBSTA and CURB-65 scores of different treatment methods were calculated, and the predictive power of each score on clinical respiratory support treatment and mortality risk was compared.

Results

The APACHE II score showed the largest area under ROC curve in both noninvasive and invasive respiratory support treatment assessments, which is significantly different from that of CURB-65. Further, the MuLBSTA score had the largest area under ROC curve in terms of death risk assessment, which is also significantly different from that of CURB-65; however, no difference was noted with the APACHE II score.

Conclusion

For patients with COVID, the APACHE II score is an effective predictor of the disease severity and mortality risk. Further, the MuLBSTA score is a good predictor only in terms of mortality risk.

Introduction

A novel coronavirus pneumonia outbreak in Wuhan, China, in December 2019, has had a major impact globally. This disease was named as "Corona Virus Disease 2019" (COVID-19), and this new type of corona virus was named as SARS-CoV-2 by the World Health Organization. According to the 7th edition of the Chinese National Health Commission, such patients can be categorized into light, normal, and severe depending on their clinical symptoms and test results [1]. However, a proper methodology to reflect the degree of the disease and predict the disease development still does not exist.

The CURB-65 (confusion, urea, respiratory rate, blood pressure, and age 65) scoring system [2] is being used as a measure of the severity of community-acquired pneumonia, which can be combined with other clinical parameters to assess if the patients need to be hospitalized or transferred to the intensive care unit (ICU). The APACHE II (acute physiology and chronic health evaluation II) scoring system [3, 4] is being used to evaluate the condition of patients in ICU using 12 parameters. Currently, this method is being widely used clinically due to its capability of distinguishing the severity of the disease. The MuLBSTA (multilobular infiltration, hypo-lymphocytosis, bacterial coinfection, smoking history, hyper-tension, and age) scoring system is an easy-to-use clinical tool to predict the risk of mortality in high-risk and low-risk groups of patients with viral pneumonia. With the use of this method, hospitalized patients with viral pneumonia can be classified into relevant risk categories to acquire guidance for further clinical decision making [5]. However, the effectiveness of these three scoring systems in assessing COVID-19 has not yet been reported.

Methods

Inclusion criteria

This single-center, retrospective observational clinical study was approved by the Ethics Committee of the General Hospital of central theater command of PLA (2020–008-1). A total of 53 cases of severe novel coronavirus pneumonia were confirmed in the General Hospital of central theater command of People’s Liberation Army between 1, January 2020 and 4, March 2020 (Fig. 1).

Fig. 1
figure 1

Research process

The inclusion criteria for the study were as follows: (1) All patients were confirmed positively by SARS-CoV-2 nucleic acid RT-PCR (Ct value ≤ 38.0, BGI, Shenzhen, China) using specimens derived from oropharyngeal swabs or sputum, prior to or during the hospitalization; and (2) Patients with the severe form of the disease were categorized based on the 7th edition of the Chinese National Health Commission, which included meeting any of the following criteria: (1) shortness of breath, respiratory rate ≥ 30 beats/min; (2) oxygen saturation ≤ 93% in the resting state; (3) arterial blood oxygen partial pressure (PaO2)/oxygen concentration (FiO2) ≤ 300 mmHg (1 mmHg = 0.133 kPa); and (4) lung images showing obvious progress of lesions > 50% within 24–48 h.

The exclusion criteria for the study were as follows: (1) Age < 18 years; (2) Patients with definite diagnosis of cancer; (3) Long-term hospitalization ≥ 3 m before death; (4) Presence of unconsciousness before admission; and (5) Patients receiving renal replacement therapy.

Data acquisition

Data related to demography, underlying comorbidities, symptoms, physical and radiological findings, laboratory values, and respiratory and physiologic parameters of the subjects while receiving mechanical ventilation were collected from electronic and paper medical records. We used a positive bacterial culture of blood and sputum samples as the criteria for bacterial growth. The APACHE II, MuLBSTA, and CURB-65 scores were calculated for different treatment time points, and the predictive power of each score for treatment with clinical respiratory support and respective mortality risk was compared.

Observational indicators

High-flow oxygen inhalation, noninvasive ventilator support, and invasive ventilator support were used as the three treatment methods. The APACHE II, MuLBSTA, and CURB-65 scoring systems were used to calculate the patient scores at each time point. The area under the receiver operating characteristic (ROC) curve was used to calculate the hierarchical boundary values of each scoring model for each treatment method [6]. The sensitivity and specificity of all the values were calculated, and the difference in area under ROC curve (AUROC) of each scoring model for the same treatment was compared. The patients were divided into high-flow oxygen inhalation group, noninvasive ventilator support group, and invasive ventilator support group. They were also categorized into death and non-death groups. The categorization was based on the severity of the patient's condition and the outcome. The APACHE II, MuLBSTA, and CURB-65 scores for the high-flow oxygen inhalation, noninvasive ventilator support, and invasive ventilation support groups prior to intubation were recorded. Further, the APACHE II, MuLBSTA, and CURB-65 scores in the death group were recorded on the day of death.

The MuLBSTA score was recorded based on the following [5]: multilobular infiltration (5 points), lymphocytes ≤ 0.8*109/L (4 points), bacterial infection (4 points), acute smokers (3 points) or quitters (2 points), hypertension (2 points), age ≥ 60 years (2 points), maximum 22 points.The CURB-65 score was recorded based on the following[2]: consciousness disorder (1 point), blood urea nitrogen > 7 mmol/L (1 point), respiratory frequency ≥ 30 beats/min (1 point), systolic blood pressure < 90 mmHg or diastolic blood pressure ≤ 60 mmHg (1 point), age ≥ 65 years (1 point), maximum 4 points. APACHE II score system is now widely used in the intensive care unit (ICU). APACHE II score system includes a 12-point acute physiology score (including Temperature, Heart rate, Breathing rate, Blood pressure, Oxygen partial pressure, PH, K+, Na+, Creatinine, HCT, WBC and Consciousness), Age point, and Chronic health evaluation. The higher the score, the more serious the condition [3, 4].

Statistical methods

Software Package for Social Sciences (IBM SPSS 25.0) was used for statistical analysis. Dates were described with median and range of continuous variables as well as frequency and percentage of categorical variables. The performance of each scoring system was evaluated by measuring the AUROC. Further, the χ2 test was used to calculate sensitivity and specificity. The different scoring models used different ROC curve areas for comparison.

Results

Basic information

Out of the 53 patients, 27 patients in the high-flow nasal catheter oxygen therapy group were cured and discharged. The remaining 26 patients underwent noninvasive ventilator support. Out of these, 20 patients further underwent endotracheal intubation; however, 16 patients could not be cured and eventually died. One of the patients who died had only received noninvasive ventilator treatment but not endotracheal intubation. The median time from onset to admission was 7 days, onset to noninvasive ventilator support was 12 days, onset to invasive ventilator support was 20 days, onset to death was 25 days, and onset to discharge was 35 days. The other demographic characteristics are listed in Table 1.

Table 1 Demographic characteristics

The scores of CURB-65, APACHE II, MuLBSTA, and the frequency of each score in each group

The frequency (number of patients) of high-flow oxygen inhalation group, noninvasive ventilation support group, and invasive ventilation support group in CURB score 0, 1, 2, 3, 4, in APACHE II score 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 20, 21 and in MuLBSTA score 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18 are separately shown in Figs. 2, 3, and 4.

Fig. 2
figure 2

Distribution of different treatment methods in CURB-65 scoring system. Quantity means “number of patients”

Fig. 3
figure 3

Distribution of different treatment methods in APACHE II scoring system. Quantity means “number of patients”

Fig. 4
figure 4

Distribution of different treatment methods in MuLBSTA scoring system. Quantity means “number of patients”

Cut-off values of CURB-65, APACHE II, and MuLBSTA for predicting the risk of noninvasive ventilator support, invasive ventilator support, and mortality

In terms of the cut-off values of CURB-65, 1.5 points was used for noninvasive ventilator support, 2.5 points for invasive ventilator support and mortality. In terms of the cut-off values of APACHE II, 9.5 points was used for noninvasive ventilator support, 12.5 points for invasive ventilator support and 11.5 points for mortality. In terms of the cut-off values of MuLBSTA, 8.5 points was used for noninvasive ventilator support, 10.5 points for invasive ventilator support and 13.5 points for mortality. These have been listed in Table 2.

Table 2 Cut-off values of CURB-65, APACHE II, and MuLBSTA for predicting the risk of noninvasive ventilator support, invasive ventilator support, and mortality

Comparison of area under ROC curve of three scoring models in each group

On evaluating the three scoring models for noninvasive ventilator support, the area under the ROC curve of the APACHE II scoring model was identified to be the largest and statistically different from that of the MuLBSTA and CURB-65 models (P = 0.0046 and 0.0059, respectively). Further, no statistical difference was identified between the MuLBSTA and CURB-65 models (P = 0.9369). The assessment of the need for invasive ventilator support revealed that the AUROC of the APACHE II scoring model was the largest, statistically different from the CURB-65 scoring model (P = 0.0372), and identical with the MuLBSTA scoring model (P = 0.2708). When assessing mortality, the AUROC of the MuLBSTA scoring model was identified to be the largest, which was statistically different from CURB-65 (P = 0.0021). However, no difference was noted with APACHE II (P = 0.0549). These findings are listed in Table 3 and shown in Figs. 5, 6, and 7.

Table 3 Comparison of the area under the ROC curve between the three scoring models
Fig. 5
figure 5

Comparison of noninvasive support ventilation with different scoring methods. The AUROC of the APACHE II scoring model was identified to be the largest and statistically different from that of the MuLBSTA and CURB-65 models (P = 0.0046 and 0.0059, respectively)

Fig. 6
figure 6

Comparison of invasive support ventilation with different scoring methods. The AUROC of the APACHE II scoring model was the largest, statistically different from the CURB-65 scoring model (P = 0.0372), and identical with the MuLBSTA scoring model (P = 0.2708)

Fig. 7
figure 7

Comparison of mortality risk with different scoring methods. The AUROC of the MuLBSTA scoring model was identified to be the largest, which was statistically different from CURB-65 (P = 0.0021), and no difference was noted with APACHE II (P = 0.0549)

Multivariate analysis of individual risk factors in each model for DEATH and INTUBATION in patients with COVID-19

On evaluating the individual risk factors in each model for death in patients with COVID-19, bacterial coinfection and age ≥ 60 years from MuLBSTA scoring model, breathing rate ≥ 30/min and age ≥ 65 years from CURB-65 scoring model were considered to be statistically different (P < 0.05); On evaluating the individual risk factors in each model for intubation in patients with COVID-19, breathing rate ≥ 30/min from CURB-65 scoring model were considered to be statistically different (P < 0.05). All other individual risk factors from the three scoring models had no statistical differences. These have been listed in Tables 4 and 5.

Table 4 Multivariate analysis of individual risk factors in each model for DEATH in patients with COVID-19
Table 5 Multivariate analysis of individual risk factors in each model for INTUBATION in patients with COVID-19

Discussion

In this study, we analyzed 53 patients with a severe form of the disease in our hospital. These patients were tested positive for the nucleic acid test between January 2020 and February 2020. In terms of demographic characteristics, the patients were older, mostly male, and had underlying diseases similar to those described by other scholars [7]. However, Wang et al. identified that among the COVID patients, 54.3% were male and 45.7% were female, showing no gender difference [8]. Only severe cases were included in our study; the rate of patients on noninvasive ventilator support, patients on invasive ventilator support, and mortality was identified to be 49.1%, 37.7%, and 30%, respectively, which was similar to the results of other studies [9, 10]. The median time from onset to admission was 7 days, onset to noninvasive ventilator treatment was 12 days, and onset to invasive ventilator treatment was 20 days. The obtained data were found to be similar to that of previous studies [11, 12]. The median time from onset to discharge was 35 days and from onset to death was 25 days.

According to current studies, early respiratory support treatment can improve the condition of patients with severe COVID-19 [1]. However, such treatments are normally administered based on a single test or simple clinical experience of doctors, which has significant limitations. In our study, three scoring systems are used to calculate the approximate scores of each respiratory treatment method. This can help clinicians judge and perform reasonable and timely respiratory management. Our research suggests that high-flow oxygen inhalation can be considered when the APACHE II score < 9.5, MuLBSTA score < 8.5, or CURB-65 score < 1.5. Further, noninvasive ventilator support can be considered when the APACHE II score ranges from 9.5 to 12.5, MuLBSTA score ranges from 8.5 to 10.5, or CURB-65 score ranges from 1.5 to 2.5, and invasive ventilator support can be considered when the APACHE II score > 12.5, MuLBSTA score ≥ 10.5, or CURB-65 score ≥ 2.5. Patients may be at risk of death when the APACHE II score > 11.5, MuLBSTA score > 13.5, or CURB-65 score > 2.5.

The APACHE II score is a classic tool for assessing the severity of the disease in patients in the ICU [4, 13]. The higher the score, the more critical the situation, worse the prognosis, and higher the mortality [13, 14]. Wang et al. determined that the median APACHE II score of patients with severe novel coronavirus pneumonia was 17 (10–22) [8], which is also consistent with our research. The APACHE II score is better than the scores of the other two methods when evaluating noninvasive respiratory support treatment (P = 0.0046 and 0.0059, respectively). In terms of invasive respiratory support therapy, the APACHE II score is better than the CURB-65 score (P = 0.0372). Further, the APACHE II score is also better than that of CURB-65 (P = 0.0150) in predicting mortality risk. Therefore, with comprehensive consideration, the APACHE II score is first recommended when assessing the overall condition of patients with COVID.

The MuLBSTA score assesses the risk of death from viral pneumonia [5, 15]. Patients with MuLBSTA score > 12 are categorized as the high-risk group [7]. Further, in our study, the patients are at risk of death when MuLBSTA score > 13.5. The MuLBSTA score has a sensitivity of 0.6364 and specificity of 0.9355 when assessing the risk of death, as reported through studies [5]. We also identify the MuLBSTA score to be better compared to the CURB-65 score in assessing death risk (P = 0.0021). Therefore, we recommend MuLBSTA score as the first choice when predicting only the risk of death.

The CURB-65 score is often used to assess the severity of community-acquired pneumonia, which requires only few assessment tools [16]. Owing to its simplicity and low score, the CURB-65 score has high sensitivity and low specificity when assessing a condition [17]. It is necessary to combine other parameters of the patient with the CURB-65 score to reach a final clinical judgment. Similarly, in our study, we find that the CURB-65 score is not as efficient as the other two scoring systems in assessing the necessity of respiratory support and death risk. Therefore, based on our study results, the CURB-65 score is not recommended for assessment of patients with COVID.

Our study has few limitations. It is a single-center, retrospective study with a relatively small sample size. Further, there is a certain degree of clinical data deficiency.

Conclusion

The current study find that for patients with COVID, the APACHE II score is an effective predictor of the disease severity and mortality risk, whereas, the MuLBSTA score is only a good predictor of mortality risk.

Abbreviations

APACHE II:

Acute physiology and chronic health evaluation II

AUROC:

Area under ROC curve

SPSS:

Software Package for Social Sciences

2019-nCoV:

Novel corona virus

COVID-19:

Corona Virus Disease 2019

CURB-65:

Confusion, urea, respiratory rate, blood pressure, and age 65

ICU:

Intensive care unit

MuLBSTA:

Multilobular infiltration, hypo-lymphocytosis, bacterial coinfection, smoking history, hyper-tension, and age

ROC:

Receiver operating characteristic

References

  1. National Health and Health Commission of the People's Republic of China. Diagnosis and Treatment of Pneumonia of New Coronavirus Infection (Trial Version 7). (2020-03-03). http://www.nhc.gov.cn/yzygj/s7653p/202003/46c9294a7dfe4cef80dc7f5912eb1989.shtml.

  2. Barlow G, Nathwani D, Davey P. The CURB65 pneumonia severity score outperforms generic sepsis and early warning scores in predicting mortality in community-acquired pneumonia. Thorax. 2007;62:253–9. https://doi.org/10.1136/thx.2006.067371.

    Article  PubMed  Google Scholar 

  3. Ferrer M, Travierso C, Cilloniz C, Gabarrus A, Ranzani OT, Polverino E, Liapikou A, Blasi F, Torres A. Severe community-acquired pneumonia: characteristics and prognostic factors in ventilated and non-ventilated patients. PLoS ONE. 2018;13:e0191721. https://doi.org/10.1371/journal.pone.0191721.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Godinjak A, Iglica A, Rama A, Tančica I, Jusufović S, Ajanović A, Kukuljac A. Predictive value of SAPS II and APACHE II scoring systems for patient outcome in a medical intensive care unit. Acta Med Acad. 2016;2:97–103. https://doi.org/10.5644/ama2006-124.165.

    Article  Google Scholar 

  5. Guo L, Wei D, Zhang X, Wu Y, Li Q, Zhou M, Qu J. Clinical features predicting mortality risk in patients with viral pneumonia: the MuLBSTA Score. Front Microbiol. 2019;10:2752. https://doi.org/10.3389/fmicb.2019.02752.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Ruan L, Chen GY, Liu Z, Zhao Y, Xu GY, Li SF, Li CN, Chen LS, Tao Z. The combination of procalcitonin and C-reactive protein or presepsin alone improves the accuracy of diagnosis of neonatal sepsis: a meta-analysis and systematic review. Crit Care (Lond, Engl). 2018;22:316. https://doi.org/10.1186/s13054-018-2236-1.

    Article  Google Scholar 

  7. Chen N, Zhou M, Dong X, Qu J, Gong F, Han Y, Qiu Y, Wang J, Liu Y, Wei Y, Xia J, Yu T, Zhang X, Zhang L. Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: a descriptive study . Lancet (London, England). 2020;395(2020):507–13. https://doi.org/10.1016/S0140-6736(20)30211-7.

    Article  CAS  Google Scholar 

  8. Wang D, Hu B, Hu C, Zhu F, Liu X, Zhang J, Wang B, Xiang H, Cheng Z, Xiong Y, Zhao Y, Li Y, Wang X, Peng Z. Clinical characteristics of 138 hospitalized patients with 2019 novel coronavirus-infected pneumonia in Wuhan, China. JAMA. 2020. https://doi.org/10.1001/jama.2020.1585.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Wang Y, Wang Y, Chen Y, Qin Q. Unique epidemiological and clinical features of the emerging 2019 novel coronavirus pneumonia (COVID-19) implicate special control measures. J Med Virol. 2020. https://doi.org/10.1002/jmv.25748.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Mizumoto K, Chowell G. Estimating risk for death from 2019 novel coronavirus disease, China, January–February 2020. Emerg Infect Dis. 2020. https://doi.org/10.3201/eid2606.200233.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Huang C, Wang Y, Li X, Ren L, Zhao J, Hu Y, Zhang L, Fan G, Xu J, Gu X, Cheng Z, Yu T, Xia J, Wei Y, Wu W, Xie X, Yin W, Li H, Liu M, Xiao Y, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet (London, England). 2020;395:497–506. https://doi.org/10.1016/S0140-6736(20)30183-5.

    Article  CAS  Google Scholar 

  12. Guan WJ, Ni ZY, Hu Y, Liang WH, Ou CQ, He JX, Liu L, Shan H, Lei CL, Hui D, Du B, Li LJ, Zeng G, Yuen KY, Chen RC, Tang CL, Wang T, Chen PY, Xiang J, Li SY, et al. Clinical characteristics of coronavirus disease 2019 in China. N Engl J Med. 2020. https://doi.org/10.1056/NEJMoa2002032.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Zhou XY, Ben SQ, Chen HL, Ni SS. A comparison of APACHE II and CPIS scores for the prediction of 30-day mortality in patients with ventilator-associated pneumonia. Int J Infect Dis. 2015;30:144–7. https://doi.org/10.1016/j.ijid.2014.11.005.

    Article  PubMed  Google Scholar 

  14. Chen L, Han X, Li Y, Zhang C, Xing X. Impact of early neuraminidase inhibitor treatment on clinical outcomes in patients with influenza B-related pneumonia: a multicenter cohort study. Eur J Clin Microbiol Infect Dis. 2020. https://doi.org/10.1007/s10096-020-03835-6,10.1007/s10096-020-03835-6.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Kalil AC, Thomas PG. Influenza virus-related critical illness: pathophysiology and epidemiology. Crit Care (Lond, Engl). 2019;23:258. https://doi.org/10.1186/s13054-019-2539-x.

    Article  Google Scholar 

  16. Brabrand M, Henriksen DP. CURB-65 score is equal to NEWS for identifying mortality risk of pneumonia patients: an observational study. Lung. 2018;196:359–61. https://doi.org/10.1007/s00408-018-0105-y.

    Article  CAS  PubMed  Google Scholar 

  17. Murillo-Zamora E, Medina-González A, Zamora-Pérez L, Vázquez-Yáñez A, Guzmán-Esquivel J, Trujillo-Hernández B. Performance of the PSI and CURB-65 scoring systems in predicting 30-day mortality in healthcare-associated pneumonia. Med Clin. 2018;150:99–103. https://doi.org/10.1016/j.medcli.2017.06.044.

    Article  Google Scholar 

Download references

Acknowledgements

We thank all the medical staff of General hospital of central theater command of PLA for their hard work and great efforts in the outbreak of COVID-19.

Funding

This project was supported by the National natural science foundation of China (81901932) and Medjaden Academy & Research Foundation for Young Scientists (COVID-19-MJA20200329).

Author information

Authors and Affiliations

Authors

Contributions

ZX had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Acquisition, analysis, or interpretation of data: CPF, WH, YJZ, XMD. Drafting of the manuscript: CPF, SXY, LBX, ZJJ, QMZ. Statistical analysis: WH, XMD. Administrative, technical, or material support: ZC. Role of the funding source: ZX received Medjaden Academy & Research Foundation for Young Scientists. ZX, CPF, WH had full access to all the data, and ZX was responsible for the decision to submit for publication. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Xiang Zhou.

Ethics declarations

Ethics approval and consent to participate

This article is approved by the Ethics Committee of the General Hospital of central theater command of PLA (2020-008-1).

Consent for publication

Not applicable. The manuscript does not contain any personal data in any form (including personal details, pictures, or videos).

Availability of data and material

The datasets used or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cheng, P., Wu, H., Yang, J. et al. Pneumonia scoring systems for severe COVID-19: which one is better. Virol J 18, 33 (2021). https://doi.org/10.1186/s12985-021-01502-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12985-021-01502-6

Keywords