Next Article in Journal
Exploring the Therapeutic Potential of Ganoderma lucidum in Cancer
Previous Article in Journal
Perspectives for the Use of Umbilical Cord Blood in Transplantation and Beyond: Initiatives for an Advanced and Sustainable Public Banking Program in Greece
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Immuno-Diagnostic Interest in Monitoring CD16+CD56+ (Natural Killer) Cells and CD19+CD45+ (B Lymphocytes) in Individuals Newly Diagnosed with HIV in a Tertiary Care Center

by
Jamil Al-Mughales
1,2
1
Department of Clinical Microbiology and Immunology, Faculty of Medicine, King Abdulaziz University, Jeddah 21589, Saudi Arabia
2
Department of Clinical Laboratories, Diagnostic Immunology Division, King Abdulaziz University Hospital, Jeddah 23623, Saudi Arabia
J. Clin. Med. 2024, 13(4), 1154; https://doi.org/10.3390/jcm13041154
Submission received: 6 December 2023 / Revised: 13 February 2024 / Accepted: 14 February 2024 / Published: 18 February 2024
(This article belongs to the Section Infectious Diseases)

Abstract

:
Background/Objective: Monitoring multiple cellular markers of immune cells may provide a more accurate evaluation of the immune status of people living with human immunodeficiency virus (PLHIV). This study assessed the value of CD16+CD56+ cells (NK cells) and CD19+ lymphocytes (B cells) phenotyping in indicating viral load, AIDS status, and treatment efficacy. Method: A retrospective, laboratory-based study was conducted at the Diagnostic immunology division of a referral tertiary hospital. It involved 82 newly diagnosed HIV patients treated between 2009–2016. We explored three objectives: (1) the paired change in CD16+CD56+ and CD19+CD45+ cells counts and percentages from baseline to 2-to-6 months after treatment; (2) the association of these phenotypes with 5 gradual categories of viral load; and (3) the accuracy of CD16+CD56+ and CD19+CD45+ cells counts in indicating AIDS stage defined as CD4+ < 200 cells/mm3. The second and third objectives were tested using a pooled analysis (N = 300–373). Result: The median CD19+CD45+ and CD16+CD56+ counts increased by 1.9-fold and 1.3-fold after treatment respectively (p < 0.001). A negative correlation of viral load with both CD16+CD56+ (ρ = −0.29, p < 0.001) and CD19+CD45+ (ρ = −0.34, p < 0.001) counts was observed. CD16+CD56+ count < 73 cells/mm3 and CD19+CD45+ count < 166.5 were indicative for AIDS with 95.5% and 63.6% sensitivity respectively. Conclusions: Findings advocate for the usefulness of CD16+CD56+ and CD19+CD45+ phenotyping in characterizing the severity of HIV infection and its impact on both the humoral and cellular immunity, as well as monitoring the effectiveness of treatment.

1. Introduction

In spite of the progress in antiretroviral therapy (ART) and the global efforts to improve the therapy strategies, nearly 2 million individuals are infected with human immunodeficiency virus (HIV) each year [1]. The effective monitoring of people living with HIV (PLHIV) provides crucial data of high clinical and epidemiological value. These data enable a real-time insight into disease activity, notably the progression to the acquired immunodeficiency syndrome (AIDS) stage, patients’ infectiveness, and the efficacy of ART, which also enables the diagnosis of eventual treatment failures [2]. Additionally, monitoring ART toxicity is recommended, as delaying drug replacements after developing adverse effects may not only harm patients but also lead to nonadherence resulting in drug resistance and treatment failure. This might compromise the available ART regimens, leading to increased HIV incidence, the spread of drug-resistant HIV, and increased morbidity and mortality [3]. Thus, besides measuring the HIV viral load in patients’ sera, CD4+ T lymphocytes count testing was developed as an indicator of the patients’ immune status. This test, called immunophenotyping, uses cytometers and cell sorters enabling the identification of different lymphocyte subsets using monoclonal antibodies. Early clinical usage of CD4 immunophenotyping established a correlation between CD4 count and AIDS stage. Thus, CD4 has ever since been widely used as a strong prognostic factor and a criterial term for the definition of the AIDS stage [4]. Furthermore, the CD4+:CD8+ ratio was also reported as a marker of a great clinical usefulness in HIV [5].
However, monitoring multiple cellular markers of immune activation may provide a more accurate evaluation of the immune status of the patient and enable a better understanding of the occurrence of opportunistic infections in PLHIV with normal or subnormal CD4 count. Among the immune functions that have been reportedly altered in HIV infection are naïve CCR7, CD27, and CD45RA T cells along with memory CD57, PD1, CD160, LAG-3, Tim-3, CTLA4, and 2B4 T cells, along with other naïve and memory B cells [6]. CD16 phenotype alteration has also been reported in HIV, and may be relevant in HIV monitoring and management [7]. The CD19+ phenotype alone has not been investigated yet as a marker in HIV monitoring. It was previously demonstrated that HIV infection involves CD19+CD45+ cells, which interact with HIV virions or viral proteins by binding via complement receptor CD21, a mechanism which enhances viral dissemination and accelerates B cell depletion by apoptosis [8].
This study aimed to assess the value of CD markers including CD16+CD56+ (NK cells) and CD19+CD45+ (B cells) in assessing and monitoring PLHIV, besides CD4 and the CD4/CD8 ratio. It tested three hypotheses: (1) the effect of treatment on the counts and percentages of the two CD markers; (2) the correlation of the two CD markers with HIV viral load; and (3) their performance in predicting AIDS status indicated by CD4 count < 200 cells/mm3.

2. Materials and Methods

2.1. Design and Participants

This was a retrospective study conducted at the Immunodiagnostic Department of King Abdulaziz University Hospital (KAUH), Jeddah, Saudi Arabia. It involved all PLHIV who were diagnosed, first treated, and followed up at the Infectious Disease Department of KAUH between 2009 and 2016. After exclusion of patients who had no relevant or follow-up data, a total of 82 PLHIV were included. The institutional review board of KAUH reviewed the study protocol and provided the ethical approval for the collection and use of the relevant data, under approval no. 197-22.

2.2. Data Collection

The researcher reviewed all laboratory records of the eligible participants and used an Excel sheet to collect baseline (at diagnosis) and all follow-up data of the following parameters:
HIV RNA/mL plasma level at diagnosis, which was ranked into 5 levels: ≤500, 501–3000, 3001–10,000, 10,001–30,000, and >30,000 copies/mL. This ranking corresponds to 5 risk categories, determined by Mellors et al., and was demonstrated to be associated with differential 6-year risks of progression to AIDS of 5.4%, 16.6%, 31.7%, 55.2%, and 80.0% [9].
Blood count data including total white blood cells (WBCs), lymphocyte count, platelets, and monocyte count.
Counts and percentages of the following phenotypes: CD3+CD45+, CD3+CD4+CD45+, CD3+CD8+CD45+, CD16+CD56+ (NK cells), and CD19+CD45+ (B cells), and CD4:CD8 ratio.
Additionally, demographic data including age, gender, and nationality were collected.

2.3. Diagnostic Immunology Assays

2.3.1. Immunophenotyping of Lymphocytes

Immunophenotyping of lymphocytes was carried out at our diagnostic immunology division according to the commercial BD kits protocol (BD multitest TM Kit; Becton and Dickinson, San Jose, CA, USA). Briefly, patients’ blood samples were collected in ethylenediaminetetraacetic acid (EDTA) anticoagulant test tubes (Vacutainer Tubes with K3EDTA-Becton Dickinson, Plymouth, UK) for immunophenotyping of lymphocyte profiles. As we used BD TrucontTM tubes, we did not use the live/dead stain to check the viability of the cells. The patients’ anti-coagulated whole blood was treated with BD CD3/CD8/CD45/CD4 and CD3/CD16+CD56/CD45/CD19 antibodies (mouse IgG1 heavy chains and kappa light chains and mouse IgG2b for the CD56 antibody). Subsequently, conjugates were added to pre-incubated patient samples, including CD45-PerCP, CD3-FITC, CD8-PE, and CD4-APC-A CD16+CD56-PE (BD Bioscience, San Jose, CA, USA).
After that, lymphocytes and lymphocyte sub-populations were determined by the flow cytometry method using BD FACSCalibur™ (Becton and Dickinson, San Jose, CA, USA). Briefly, in each sample, 10,000 cells were analyzed, and the results were expressed as the percentage (proportion) of the examined cells in each subpopulation of lymphocytes. Cell subpopulations were analyzed in BD program, version 6.1.3 (DB Biosci-ence, San Jose, CA, USA). In the analyzed samples, lymphocytes were defined via the gating strategy for TBNK (Figure 1). The absolute number of individual subpopulations (the number of cells per 1 mL of blood) was determined by the formula Abs CD4 (CD8) = WBC × Lym/100 × (% positive cells Cd4 or CD8)/100.
During analysis, the absolute number (cells/µL) of positive cells in the sample was determined by comparing cellular events to bead events. In the study setting, we used BD Multiset software to determine the absolute count automatically, which can be obtained using the following equation:
A = X Y × N V
where A is the absolute count of the cell population; X is the number of positive cell events; Y is the number of bead events; N is the number of beads per test, which is found on the BD Trucount foil pouch and can vary from lot to lot; and V is the test volume. In addition, absolute counts were compared to the CBC results obtained for each patient using the hematology analyzer.

2.3.2. Nucleic Acid Amplification Test for the Quantitation of Human HIV-1 Ribonucleic Acid (RNA) in Human Plasma

The commercial COBAS® AmpliPrep/COBAS® TaqMan® HIV-1 Test, v2.0 (Roche, Switzerland) using Polymerase Chain Reaction (PCR) technology was used to achieve maximum sensitivity and dynamic range for the quantitative detection of HIV-1 group M and group O RNA in EDTA anti-coagulated plasma. The HIV-1 RNA signal was calculated in the specimens and controls by comparing their concentrations to the quantitation standard concentrations in each specimen and control. The test range was 20–10,000,000 copies/mL.

2.4. Statistical Approaches

The researcher used three different analytical approaches to test the 3 hypotheses, using two different datasets: a paired dataset and a pooled dataset.

2.4.1. Hypothesis 1

The first analytical approach consisted of a paired analysis testing the change, from baseline (pre-ART) to follow-up (post ART) times, in NK cell and B cell counts along with HIV RNA/mL plasma levels and the other tested parameters. The follow-up time considered for this analysis was 2 to 6 months, depending on the patient, corresponding to the initial assessment of ART therapeutic response. Thus, the first approach tested the effect of treatment on NK cells and B cell counts and percentages, as well as on the levels of HIV RNA/mL plasma and the other phenotypes. It was hypothesized that a decrease in HIV RNA/mL plasma levels would be concomitant with an increase in NK cells and B cells and other phenotypes. Such an increase would support the relevance of using the given parameter as an indicator for a complete response to ART.

2.4.2. Hypothesis 2

The second analytical approach tested the bivariate correlation of NK cell and B cell counts and percentages with HIV RNA/mL plasma levels. We hypothesized that a negative relationship between HIV RNA/mL plasma levels and a given phenotype (NK cells or B cells) would indicate the value of this phenotype in monitoring PLHIV and their response to antiretroviral treatment. This analysis used the pooled data of all patients at all follow-up times, subject to availability of the relevant data.

2.4.3. Hypothesis 3

The third hypothesis consisted of analyzing the sensitivity of NK cell and B cell counts in indicating AIDS status, defined by CD4 cell count < 200 cells/mm3 [10]. This analysis used the pooled database.

2.5. Statistical Methods

Both databases were edited, processed, and analyzed using the Statistical Package of Social Science (SPSS), version 21, for Windows (IBM, Chicago, IL, USA). Descriptive statistics were used for baseline data to present means and standard deviations (SD)s, medians and interquartile ranges (IQRs), or frequencies and percentages, as applicable.
To test the first hypothesis, the related-samples Wilcoxon signed-rank test was used to analyze the paired change in HIV RNA/mL plasma rank and NK cell and B cell counts and percentages, in addition to the other parameters including total WBCs; lymphocyte count; platelets; monocyte count; counts and percentages of CD3+CD45+, CD3+CD4+CD45+, CD3+CD8+CD45+; and CD4:CD8 ratio. The results are presented as medians with IQR and the p-value.
In the second analytical approach, using pooled data, Spearman’s coefficient (ρ) was used to analyze the bivariate correlation of HIV RNA/mL plasma level with NK cell and B cell counts and percentages as well as with the other parameters. Additionally, the Kruskal–Wallis test was used to analyze the variance in the different phenotypes by HIV RNA/mL plasma rank (≤500, 501–3000, 3001–10,000, 10,001–30,000, and >30,000 copies/mL); the results are presented as the median and IQR of the given parameter in the given HIV RNA/mL plasma rank.
To test the third hypothesis, Receiver Operating Characteristics (ROC) curve analysis was carried out to analyze the accuracy of NK cell and B cell counts as well as CD3+ count in indicating AIDS status, with calculation of the area under the curve (AUC). The ROC curve analysis was completed by the determination of the best cutoff of each parameter and the related sensitivity and specificity by applying the Youden’s index method to the ROC curve coordinates.
A p-value < 0.05 was considered to reject the null hypothesis.

3. Results

3.1. Baseline Characteristics

The flowchart of participants is depicted in Figure 2. Eighty-two PLHIV were included in the study, forty-four (53.7%) were male and the mean (SD) age was 49.00 (13.83) years. At diagnosis, the majority of patients (64.6%) had very high HIV RNA/mL plasma titers (>30,000 copies/mL). HCV and HbsAg were negative in all tested patients. Other serology findings and mean (SD) values of blood count parameters and immunophenotyping are presented in Table 1.

3.2. Change in HIV RNA/mL Plasma from Baseline to Outcome

There was a remarkable decrease in HIV RNA/mL plasma titers from the baseline to endpoint, with 61 (74.4%) of the patients having ≤500 copies/mL, while only 11 (13.4%) had more than 10 000 copies/mL (Table 2). The mean (SD) follow-up time from baseline to last follow-up was 30.8 (26.9) weeks and the median (IQR) was 22 (20) weeks (the results are not presented in tables).

3.3. Change in Blood Count and Phenotyping Parameters

From baseline to follow-up, the median NK cell and B lymphocyte counts increased 1.9- and 1.4-fold, respectively (p < 0.001). Additionally, there was a significant increase in total lymphocyte count (p < 0.001) as well as in the counts of other phenotypes including CD3+CD45+ (p < 0.001), CD3+CD4+CD45+ (p < 0.001), and CD3+CD8+CD45+ (p < 0.001). Furthermore, both CD3+CD4+CD45+ percentage (p < 0.001) and CD4/CD8 ratio (p = 0.001) increased significantly (Table 3).

3.4. Correlation of Immunophenotyping Parameters with HIV RNA/mL Plasma Levels

By considering pre-treatment findings only (N = 82), both NK cells (ρ = −0.26) and B cells (ρ = −0.18) were inversely correlated with HIV RNA/mL plasma; however, this was statistically significant for NK cells but not for B cells (p = 0.144). In the post-treatment period (N = 82), neither NK cells (ρ = −0.15, p = 0.187) nor B cells (ρ = −0.12, p = 0.283) showed a significant correlation with HIV RNA/mL plasma (Table 4). It should be noted that the latter correlation analyses are limited by the low sample size (N = 82) and the skewed distribution of the participants within the HIV RNA/mL plasma categories both in the pre- and post-treatment periods, as shown in Table 2.
By pooling baseline and different follow-up data of all patients, 300 to 373 observations were obtained, depending on the parameter. Bivariate correlation analysis showed that HIV RNA/mL plasma titer was negatively correlated with counts of all the studied lymphocyte phenotypes (p < 0.05); however, the relationship was weak to moderate, with Spearman’s coefficient ρ ranging between −0.17 and −0.38. The relationship of HIV RNA/mL plasma was relatively stronger with CD3+CD4+CD45+ count (ρ = −0.38, p < 0.001) and B cell count (ρ = −0.34, p < 0.001) (Table 5).

3.5. Significance of NK Cell Count, B Cell Count, and CD3+ Count in Indicating AIDS Status (Pooled Data)

The ROC curve analysis showed NK cell (AUC = 0.629, p < 0.0001) and B cell (AUC = 0.683, p < 0.0001) counts to be equally indicative of AIDS status, while CD3+ count had a higher AUC of 0.809 (p < 0.0001) (Figure 3). By applying Youden’s index, an NK cell count < 73 cells/mm3 was indicative for AIDS status with 95.5% sensitivity and 28.7% specificity, and a B cell count < 166.5 cells/mm3 was indicative for AIDS with 63.6% sensitivity and 65.4% specificity. By contrast, a CD3 count < 876.5 cells/mm3 was indicative for AIDS with 91.4% sensitivity and 63.2% specificity (Table 6).

4. Discussion

4.1. Summary of Findings

This retrospective laboratory-based study explored the significance of NK cells and B cells in monitoring newly diagnosed and treated PLHIV in a referral center in Western Saudi Arabia. It used two different databases to analyze the effect of ART response on NK cell and B cell counts, the correlation of NK cell and B cell counts and percentages with viral load, and the sensitivity of NK cell and B cells count in indicating AIDS status. A paired pre-to-post-treatment approach demonstrated a significant increase in NK cell and B cell counts, as well as CD4+ and overall CD3+ counts, as a result of ART initiation, occurring after a median follow-up period of 22 weeks. A pooled database was used to demonstrate both NK cell and B cell counts to be negatively correlated with HIV RNA/mL plasma levels, with correlation coefficients comparable to those found with CD4+ and CD3+ T cells. Additionally, NK cell and B cells count were demonstrated to be significantly predictive for AIDS status, with a sensitivity as high as 95.5% for an NK cell count < 73 cells/mm3.

4.2. Discussing Baseline Observations

It is of great importance to note that the majority (65%) of the patients had high HIV RNA/mL plasma levels at diagnosis and almost half of them (48.8%) were diagnosed at the AIDS stage. This is indicative of a significant delay in the diagnosis of HIV in Saudi Arabia, which should be considered a high-priority public health issue by the health authorities. Late HIV diagnosis not only impacts the health outcome and life expectancy of the infected individuals [11], but also increases the risk of virus dissemination in the community [12], ultimately increasing the burden of the disease. Therefore, several models have been proposed to characterize the delay in HIV diagnosis, and most of these agreed on the reliability of CD4 count in defining the late presentation, notably the concurrent diagnosis of HIV and AIDS [13]. Furthermore, the consequence of late HIV diagnosis is more detrimental in cases of inaccessible, delayed or inadequately conducted treatment. In a large Chinese cohort study of 528,234 patients newly diagnosed with HIV, where 34% had a late diagnosis, the one-month mortality rate among individuals who were diagnosed with a CD4 count < 50 cells/mm3 was 16% in the absence of ART, in comparison to <1% in whom ART was appropriately conducted. The one-year mortality rate was 64% versus 13.3% in the two groups, respectively [14]. Altogether, these observations highlight the urgent need to enhance routine testing in Saudi Arabia to reduce late HIV diagnosis and promote early initiation of ART.

4.3. Impact of HIV Infection on NK Cells

The findings from the present study support the reliability of using NK cells as a biomarker of the severity of HIV infection and the therapeutic response to ART among patients newly diagnosed with HIV. Both the count and percentage of NK cells were negatively correlated with HIV viral load and increased in response to ART initiation. Furthermore, an NK cell count < 73 cells/mm3 was indicative for AIDS status with 95.5% sensitivity. NK cells are innate lymphoid cells that ensure nonspecific immune response against tumoral and virus-infected cells [15], and contribute to the regulation of adaptive immunity [16,17]. The impact of HIV infection on this cell population has already been demonstrated. The activity of NK cells is significantly altered among PLHIV [18], and several theories have been elaborated to explain such an alteration [19,20,21]. It was even proposed that NK cell activity could be a prognostic indicator for the progression to the AIDS stage [22], which is consistent with our findings showing an association between NK cell count and AIDS status. In 2009, Mansour et al. observed a significant decline in NK cells and other CD3− phenotypes in all-stage PLHIV in comparison to non-HIV controls. By stratifying HIV into three severity stages, patients at the AIDS stage had a significantly lower NK cell count (mean ± SEM = 121 ± 11 cells/mm3) compared with asymptomatic patients (172 ± 15) and those with generalized adenopathy (187 ± 24) [23]. Other data by Vuillier et al. found a significant decrease in both NK and B cells, and the authors suggested that such a depletion “accounts for the decline in low-density CD8+ lymphocytes in HIV positive group” [24]. Interestingly, a recent review characterized the pathogenic impact of HIV infection on the frequency and functions of NK cells, which impairs their anti-viral activity notably by the upregulation of inhibitory NK cell receptors (iNKRs) and downregulation of activating NK cell receptors (aNKRs), both of which are sensitive to the viral load. On the other hand, the same review demonstrated the reversibility of these pathological changes after a well-conducted ART [25]. These observations are consistent with findings from the present study showing a negative correlation between NK cell count and viral load as well as a significant increase in NK count after treatment.
Moreover, it has been demonstrated that CMV infection induces adaptive changes to NK cells, resulting in the emergences of certain subsets such as an increase in the number of mature CD56dimNKG2A+CD57+ NK cells. This is associated with alterations in the signaling and cytokine responsiveness of NK cells [26,27]. However, in the present study, only one participant had an active CMV infection, indicated by positive IgM, which would not impact the cohort findings.

4.4. Impact of HIV Infection on CD19+ Lymphocytes

In the present study, the frequency of CD19+ lymphocytes was demonstrated to be significantly lower among newly diagnosed HIV patients before treatment, which increased after initiation of ART. Furthermore, the B cell count was negatively correlated with the viral load; however, the best B cell count cutoff was not sensitive (63.6%) or specific in indicating AIDS status. CD19+ lymphocytes encompass several subsets and have an important role, notably in T cell activation [28]. The effect of HIV infection and the plasma viremia on CD19+ and other subsets of B lymphocytes have been less characterized than the effect on CD4+ and CD8+ T cells in the early years of the HIV epidemic. However, several pathogenic mechanisms involving B cells have been discovered, contributing to the understanding of the impaired humoral immunity and its association with the development of opportunistic infections among PLHIV [29,30,31]. Clinically, Samuelsson et al. observed high levels of CD19+ cell apoptosis among PLHIV with rapidly progressing disease, which was correlated with viral load and the level of CD4+ cell apoptosis. On the other hand, the authors observed a low rate of apoptosis in CD19+ cells in slowly progressing forms of HIV [32]. These observations are consistent with our findings and support the relevance of monitoring CD19+ cells as an indicator for the impact of HIV infection on humoral immunity. Other data demonstrated other B lymphocyte subsets to be prone to apoptosis in PLHIV, such as CD10+ cells, and that there are different mechanisms of apoptosis [33]. Other observations by Moir et al. showed a generalized impairment of B lymphocyte functions in association with high HIV viral load, notably the abnormal proliferation of specific subsets of B lymphocytes, such as CD21+ cells, that are responsible for hypergammaglobulinemia [34]. Another study by Moir et al. showed a significant increase in and normalization of CD19+ cell count after a twelve-month ART, which was concomitant with the increase in CD4+ T cells and decrease in HIV viral load [35]. This is consistent with our findings and supports the responsiveness of CD19+ cells to ART and its reliability as an indicator for ART effectiveness.

4.5. Clinical Implications

This study provides insights into the role of NK cells and B cells in monitoring PLHIV, especially those newly diagnosed, which may have various clinical implications:
One of the potential applications is in the comprehensive assessment of immune status in HIV-infected individuals, especially at diagnosis.
Clinicians can further use these CD markers to track the effectiveness of ART and make informed decisions regarding treatment adjustments when necessary. This constitutes an area for research into the added value of monitoring these parameters for more effective management.
More notably, NK cell count has shown remarkable sensitivity in indicating AIDS status, with 95.5% sensitivity at a cutoff of <73 cells/mm3. The analysis of the kinetics of NK count changes during HIV infection would present a potential application in the early identification of patients progressing towards AIDS, allowing for timely intervention.
On the other hand, the inclusion of these markers in routine assessment and monitoring will entail extra costs that may require further evidence to support their added clinical value and cost-effectiveness. The complexity of the assay and the need for specialized equipment may limit its accessibility, especially in resource-limited settings where the burden of HIV is often the highest. It is to note that, in our department, the phenotyping assay systematically analyzes all lymphocyte subsets, including CD4, CD8, and other CD markers.
Furthermore, the effective implementation of these markers requires technical expertise not only in performing the assay but also in interpreting the results, which may not be readily available in all clinical settings.
The alarming rates of late HIV diagnosis, with a substantial portion of patients already in the AIDS stage at the time of diagnosis, underline the need for increased awareness and routine testing in Saudi Arabia. Early identification of HIV infection is paramount in preventing the progression to AIDS and reducing the transmission of the virus within the community.

4.6. Research Implications

The findings from this study provide various research perspectives, among which are the following:
Further research is warranted to establish the correlations between viral load and NK cell as well as B cell counts, unlike CD4+ cell counts, which have a well-established threshold for staging HIV infection. Defining standard thresholds is crucial for streamlining clinical interpretation and ensuring consistent results across different laboratories and technicians.
Secondly, the longitudinal stability of the markers in question needs to be evaluated. CD4+ cell counts are known to fluctuate with disease progression and treatment, and it is necessary to understand how NK and B cell counts may vary over time and in response to ART.
The markers’ specificity and sensitivity in detecting various stages of HIV across different populations need confirmation to ensure accurate disease staging, without interfering with other conditions.
The clinical benefits of these markers and their cost-effectiveness still need to be proven through rigorous trials before they can be adopted into standard HIV management protocols.

4.7. Limitations

The present study is limited by the retrospective design, which resulted in an unavailability of relevant data in a number of patients. Additionally, the availability of CD16+CD56+ and CD19+ phenotyping data may be contingent upon logistical and technical resources throughout the study period, potentially introducing a selection bias. Finally, the internal validity of the findings may be compromised by the use of pooled data, which may explain both the low AUC in the ROC curve results and the relatively low levels of correlation between HIV RNA/mL plasma and NK and B cells. This limitation stems from the heterogeneity of the pooled data that combined pre- and post-treatment results, thereby not accounting for the complex immunological changes that may be triggered by ART and that may interfere with the levels of the different phenotypes. Nevertheless, the pooled analysis was carried out to test whether the changes in NK and B cell counts are concomitant with the changes in CD4 and HIV RNA/mL plasma levels, and thus could reflect the immune status of the patients regardless of the treatment status and other immunological mechanisms. Still, further evidence is warranted from adequately sized, prospective paired data using fixed intervals to characterize the proportional change over time of CD markers in correlation with the change in HIV RNA/mL plasma levels after treatment initiation.

5. Conclusions

The present analysis showed that the frequencies of NK cells and B cells were significantly impacted in patients newly diagnosed with HIV proportional with the severity of the disease indicated by a high viral load and low CD4+ count. Moreover, the NK cell count was highly sensitive in indicating AIDS stage defined as a CD4+ count < 200 cells/mm3, while B cells showed a low sensitivity. Additionally, NK cell and B cell counts were responsive to the treatment and were significantly increased after the initiation of ART. This advocates for the usefulness of comprehensive lymphocytes phenotyping in characterizing the severity of HIV infection and its impact on both innate and adaptive immunity, as well as in monitoring the effectiveness of ART. However, further prospective studies are warranted to explore the cost-effectiveness and long-term relevance of these biomarkers in chronic HIV. Moreover, we noted an alarming proportion of patients with HIV diagnosed at the AIDS stage or with high viral loads, which requires urgent measures.

Funding

The author received a grant from the Deanship of Scientific Research (RG-12-140-42).

Institutional Review Board Statement

The present study was ethically approved by the Research Ethics Committee of King Abdul-Aziz University (Ref # 197-22) on 14 April 2022.

Informed Consent Statement

Patient consent was waived due to the retrospective nature of the study.

Data Availability Statement

The datasets generated and analyzed during the current study are not publicly available due to privacy and ethical considerations. However, they are available from the corresponding author upon reasonable request via email.

Acknowledgments

The author thanks the company Scientia Vallis (Paris, France), represented by Mohamed Amine HAIRECHE, for the valuable inputs in statistical analysis and editing of the present manuscript. Additionally, the author would like to acknowledge Fatima Al-Jaberi, Ola Saati, and Sultan Al-Amri for their assistance in running the patient samples in our diagnostic immunology division.

Conflicts of Interest

The author declares no conflicts of interest.

References

  1. Frank, T.D.; Carter, A.; Jahagirdar, D.; Biehl, M.H.; Douwes-Schultz, D.; Larson, S.L.; Arora, M.; Dwyer-Lindgren, L.; Steuben, K.M.; Abbastabar, H. Global, regional, and national incidence, prevalence, and mortality of HIV, 1980–2017, and forecasts to 2030, for 195 countries and territories: A systematic analysis for the global burden of diseases, injuries, and risk factors study 2017. Lancet HIV 2019, 6, e831–e859. [Google Scholar] [CrossRef]
  2. WHO. The Use of Antiretroviral Drugs for Treating and Preventing HIV Infection. 2016. Available online: https://iris.who.int/bitstream/handle/10665/208825/9789241549684_eng.pdf?sequence=1 (accessed on 15 September 2022).
  3. Pham, M.D.; Romero, L.; Parnell, B.; Anderson, D.A.; Crowe, S.M.; Luchters, S. Feasibility of antiretroviral treatment monitoring in the era of decentralized HIV care: A systematic review. AIDS Res. Ther. 2017, 14, 3. [Google Scholar] [CrossRef]
  4. Kagan, J.M.; Sanchez, A.M.; Landay, A.; Denny, T.N. A brief chronicle of CD4 as a biomarker for HIV/AIDS: A tribute to the memory of John L. Fahey. Onco Ther. 2015, 6, 55–64. [Google Scholar] [CrossRef]
  5. Vivancos-Gallego, M.J.; Okhai, H.; Perez-Elías, M.J.; Gomez-Ayerbe, C.; Moreno-Zamora, A.; Casado, J.L.; Quereda, C.; Sanz, J.M.; Sanchez-Conde, M.; Serrano-Villar, S. CD4+:CD8+ T-cell ratio changes in people with HIV receiving antiretroviral treatment. Antivir. Ther. 2020, 25, 91–100. [Google Scholar] [CrossRef]
  6. McIntyre, J. Use of antiretrovirals during pregnancy and breastfeeding in low-income and middle-income countries. Curr. Opin. HIV AIDS 2010, 5, 48–53. [Google Scholar] [CrossRef]
  7. Appay, V.; Sauce, D. Assessing immune aging in HIV-infected patients. Virulence 2017, 8, 529–538. [Google Scholar] [CrossRef]
  8. Moir, S.; Fauci, A.S. B cells in HIV infection and disease. Nat. Rev. Immunol. 2009, 9, 235–245. [Google Scholar] [CrossRef]
  9. Mellors, J.W. Plasma viral load and CD4+ lymphocytes as prognostic markers of HIV-1 infection. Ann. Intern. Med. 1997, 126, 946–954. [Google Scholar] [CrossRef] [PubMed]
  10. Yoshimura, K. Current status of HIV/AIDS in the ART Era. J. Infect. Chemother. 2017, 23, 12–16. [Google Scholar] [CrossRef] [PubMed]
  11. May, M.; Gompels, M.; Delpech, V.; Porter, K.; Post, F.; Johnson, M.; Dunn, D.; Palfreeman, A.; Gilson, R.; Gazzard, B. Impact of late diagnosis and treatment on life expectancy in people with HIV-1: UK collaborative HIV cohort (UK CHIC) study. BMJ 2011, 343, d6016. [Google Scholar] [CrossRef] [PubMed]
  12. Delpierre, C.; Cuzin, L.; Lert, F. Routine testing to reduce late HIV diagnosis in France. BMJ 2007, 334, 1354–1356. [Google Scholar] [CrossRef] [PubMed]
  13. MacCarthy, S.; Hoffmann, M.; Ferguson, L.; Nunn, A.; Irvin, R.; Bangsberg, D.; Gruskin, S.; Dourado, I. The HIV care cascade: Models, measures and moving forward. J. Int. AIDS Soc. 2015, 18, 19395. [Google Scholar] [CrossRef] [PubMed]
  14. Tang, H.; Mao, Y.; Tang, W.; Han, J.; Xu, J.; Li, J. “Late for testing, early for antiretroviral therapy, less likely to die”: Results from a large HIV cohort study in China, 2006–2014. BMC Infect. Dis. 2018, 18, 272. [Google Scholar] [CrossRef] [PubMed]
  15. Pahl, J.; Cerwenka, A. Tricking the balance: NK cells in anti-cancer immunity. Immunobiology 2017, 222, 11–20. [Google Scholar] [CrossRef] [PubMed]
  16. Moretta, A.; Marcenaro, E.; Sivori, S.; Della Chiesa, M.; Vitale, M.; Moretta, L. Early liaisons between cells of the innate immune system in inflamed peripheral tissues. Trends Immunol. 2005, 26, 668–675. [Google Scholar] [CrossRef] [PubMed]
  17. Mattiola, I.; Pesant, M.; Tentorio, P.F.; Molgora, M.; Marcenaro, E.; Lugli, E.; Locati, M.; Mavilio, D. Priming of human resting NK Cells by autologous M1 macrophages via the engagement of IL-1β, IFN-β, and IL-15 pathways. J. Immunol. 2015, 195, 2818–2828. [Google Scholar] [CrossRef] [PubMed]
  18. Rook, A.H.; Masur, H.; Lane, H.C.; Frederick, W.; Kasahara, T.; Macher, A.M.; Djeu, J.Y.; Manischewitz, J.F.; Jackson, L.; Fauci, A.S.; et al. Interleukin-2 enhances the depressed natural killer and cytomegalovirus-specific cytotoxic activities of lymphocytes from patients with the acquired immune deficiency syndrome. J. Clin. Investig. 1983, 72, 398–403. [Google Scholar] [CrossRef]
  19. Bonavida, B.; Katz, J.; Gottlieb, M. Mechanism of defective NK cell activity in patients with acquired immunodeficiency syndrome (AIDS) and AIDS-related complex. I. Defective trigger on NK cells for NKCF production by target cells, and partial restoration by IL 2. J. Immunol. 1986, 137, 1157–1163. [Google Scholar] [CrossRef]
  20. Alcocer-Varela, J.; Alarcon-Segovia, D.; Abud-Mendoza, C. Immunoregulatory circuits in the acquired immune deficiency syndrome and related complex. Production of and response to interleukins 1 and 2, NK function and its enhancement by interleukin-2 and kinetics of the autologous mixed lymphocyte reaction. Clin. Exp. Immunol. 1985, 60, 31–38. [Google Scholar] [PubMed]
  21. Poli, G.; Introna, M.; Zanaboni, F.; Peri, G.; Carbonari, M.; Aiuti, F.; Lazzarin, A.; Moroni, M.; Mantovani, A. Natural killer cells in intravenous drug abusers with lymphadenopathy syndrome. Clin. Exp. Immunol. 1985, 62, 128–135. [Google Scholar] [PubMed]
  22. Voth, R.; Rossol, S.; Gräff, E.; Laubenstein, H.P.; Schröder, H.C.; Müller, W.E.G.; zum Büschenfelde, K.H.M.; Hess, G. Natural killer cell activity as a prognostic parameter in the progression to AIDS. J. Infect. Dis. 1988, 157, 851–852. [Google Scholar] [CrossRef]
  23. Mansour, I.; Doinel, C.; Rouger, P. CD16+ NK cells decrease in all stages of HIV infection through a selective depletion of the CD16+CD8+CD3 Subset. AIDS Res. Hum. Retroviruses 1990, 6, 1451–1457. [Google Scholar] [CrossRef]
  24. Vuillier, F.; Bianco, N.E.; Montagnier, L.; Dighiero, G. Selective depletion of low-density CD8+, CD16+ lymphocytes during HIV infection. AIDS Res. Hum. Retroviruses 1988, 4, 121–129. [Google Scholar] [CrossRef]
  25. Mikulak, J.; Oriolo, F.; Zaghi, E.; Di Vito, C.; Mavilio, D. Natural killer cells in HIV-1 infection and therapy. AIDS 2017, 31, 2317–2330. [Google Scholar] [CrossRef] [PubMed]
  26. Hendricks, D.W.; Balfour, H.H.; Dunmire, S.K.; Schmeling, D.O.; Hogquist, K.A.; Lanier, L.L. Cutting edge: NKG2ChiCD57+ NK cells respond specifically to acute infection with cytomegalovirus and not Epstein–Barr virus. J. Immunol. 2014, 192, 4492–4496. [Google Scholar] [CrossRef] [PubMed]
  27. Schlums, H.; Cichocki, F.; Tesi, B.; Theorell, J.; Beziat, V.; Holmes, T.D.; Han, H.; Chiang, S.C.C.; Foley, B.; Mattsson, K.; et al. Cytomegalovirus infection drives adaptive epigenetic diversification of NK cells with altered signaling and effector function. Immunity 2015, 42, 443–456. [Google Scholar] [CrossRef] [PubMed]
  28. Flores-Borja, F.; Bosma, A.; Ng, D.; Reddy, V.; Ehrenstein, M.R.; Isenberg, D.A.; Mauri, C. CD19+CD24hiCD38hi B cells maintain regulatory T cells while limiting TH1 and TH17 differentiation. Sci. Transl. Med. 2013, 5, 173ra23. [Google Scholar] [CrossRef] [PubMed]
  29. Moir, S.; Fauci, A.S. Pathogenic mechanisms of B-lymphocyte dysfunction in HIV disease. J. Allergy Clin. Immunol. 2008, 122, 12–19. [Google Scholar] [CrossRef] [PubMed]
  30. Schnittman, S.M.; Lane, H.C.; Higgins, S.E.; Folks, T.; Fauci, A.S. Direct polyclonal activation of human B lymphocytes by the acquired immune deficiency syndrome virus. Science 1986, 233, 1084–1086. [Google Scholar] [CrossRef] [PubMed]
  31. Chirmule, N.; Oyaizu, N.; Kalyanaraman, V.S.; Pahwa, S. Inhibition of normal B-cell function by human immunodeficiency virus envelope glycoprotein, gp120. Blood 1992, 79, 1245–1254. [Google Scholar] [CrossRef] [PubMed]
  32. Samuelsson, A.; Broström, C.; van Dijk, N.; Sönnerborg, A.; Chiodi, F. Apoptosis of CD4+ and CD19+ cells during human immunodeficiency virus type 1 infection—Correlation with clinical progression, viral load, and loss of humoral immunity. Virology 1997, 238, 180–188. [Google Scholar] [CrossRef]
  33. Ho, J.; Moir, S.; Malaspina, A.; Howell, M.L.; Wang, W.; DiPoto, A.C.; O’Shea, M.A.; Roby, G.A.; Kwan, R.; Mican, J.M.; et al. Two overrepresented B cell populations in HIV-infected individuals undergo apoptosis by different mechanisms. Proc. Natl. Acad. Sci. USA 2006, 103, 19436–19441. [Google Scholar] [CrossRef] [PubMed]
  34. Moir, S.; Malaspina, A.; Ogwaro, K.M.; Donoghue, E.T.; Hallahan, C.W.; Ehler, L.A.; Liu, S.; Adelsberger, J.; Lapointe, R.; Hwu, P.; et al. HIV-1 Induces Phenotypic and Functional Perturbations of B Cells in Chronically Infected Individuals. Proc. Natl. Acad. Sci. USA 2001, 98, 10362–10367. [Google Scholar] [CrossRef] [PubMed]
  35. Moir, S.; Malaspina, A.; Ho, J.; Wang, W.; DiPoto, A.C.; O’Shea, M.A.; Roby, G.; Mican, J.M.; Kottilil, S.; Chun, T.; et al. Normalization of B Cell Counts and Subpopulations after Antiretroviral Therapy in Chronic HIV Disease. J. Infect. Dis. 2008, 197, 572–579. [Google Scholar] [CrossRef] [PubMed]
Figure 1. The flow cytometry gating strategy for TBNK lymphocytes. (a) CD3+CD4+ (APC-A) and CD3+CD8+ lymphocytes (PE-A): (i) all events; (ii) CD3+ T lymphocytes; (iii) the proportion of CD8+ cytotoxic T cells and CD4+ T helper cells. (b) CD19+ B lymphocytes (APC-A) and NK lymphocytes (PE-A): (i) all events; (ii) CD3− lymphocytes; (iii) the proportion of NK cells (CD16+CD56+) and B cells (CD19+).
Figure 1. The flow cytometry gating strategy for TBNK lymphocytes. (a) CD3+CD4+ (APC-A) and CD3+CD8+ lymphocytes (PE-A): (i) all events; (ii) CD3+ T lymphocytes; (iii) the proportion of CD8+ cytotoxic T cells and CD4+ T helper cells. (b) CD19+ B lymphocytes (APC-A) and NK lymphocytes (PE-A): (i) all events; (ii) CD3− lymphocytes; (iii) the proportion of NK cells (CD16+CD56+) and B cells (CD19+).
Jcm 13 01154 g001
Figure 2. Flowchart of the participants.
Figure 2. Flowchart of the participants.
Jcm 13 01154 g002
Figure 3. Significance of NK cell count (a), B cell count (b) and CD3+ cell count (c) in indicating AIDS status among HIV patients—Receiver Operating Characteristics (ROC) curve analysis.
Figure 3. Significance of NK cell count (a), B cell count (b) and CD3+ cell count (c) in indicating AIDS status among HIV patients—Receiver Operating Characteristics (ROC) curve analysis.
Jcm 13 01154 g003
Table 1. Baseline demographic and laboratory characteristics (N = 82).
Table 1. Baseline demographic and laboratory characteristics (N = 82).
ParameterCategoryMeanSDN%
Age (N = 82)Years 49.013.83
Gender (M = 82)Male 4453.7
Female 3846.3
Nationality (N = 82)Saudi 3441.5
Yemeni 2834.1
Other 2024.4
HCV serology (N = 82)Negative 82 100.0
Positive 00.0
HbsAg (N = 81)Negative 8198.8
Positive 00.0
Toxoplasmosis IgM (N = 31)Negative 3137.8
Positive 0.00.0
CMV IgG (N = 65)Negative 22.4
Positive 6376.8
CMV IgM (N = 6)Negative 56.1
Positive 11.2
AIDS (CD4 < 200 cells/mm3) (N = 80)No 4151.3
Yes 3948.8
Due to missing data for some patients, the values for certain variables do not sum to the total of 82.
Table 2. Progression of HIV RNA/mL plasma from baseline to outcome.
Table 2. Progression of HIV RNA/mL plasma from baseline to outcome.
Viral Load (HIV RNA/mL Plasma)BaselineAfter TreatmentStatistics * (p-Value)
N%N%
≤500 1315.96174.45.00 (<0.0001)
501–300044.989.8
3001–10 00078.522.4
10.001–30,00056.144.9
>30,0005364.678.5
* Related-samples Wilcoxon signed-rank test.
Table 3. Change in phenotype parameters (nonparametric tests).
Table 3. Change in phenotype parameters (nonparametric tests).
ParameterNBaselineAfter TreatmentChange p-Value
MedianIQR
(Q1, Q3)
MedianIQR
(Q1, Q3)
Total WBC784.273.527.284.263.726.45 0.911
Lymphocyte count691.540.943.542.221.513.58<0.001 *
Platelets 80227191.0340.0254224.00362.00 0.459
Monocyte count690.570.381.460.520.350.87 0.177
CD3+CD45+ count82908479190316308452638<0.001 *
CD3+CD45+ Percentage827566.0083.757671.0080.00 0.229
CD3+CD4+CD45+ count8020329486311146613<0.001 *
CD3+CD4+CD45+ %80154.025.0147.0030.00<0.001 *
CD3+CD8+CD45+ count6355627712769794271579<0.001 *
CD3+CD8+CD45+ %635334.2566.004835.0065.00 0.193
CD16+CD56+ (NK) count8213570237258102360<0.001 *
CD16+CD56+ (NK) %82118.0014.00108.0015.00 0.453
CD19+CD45+ (B cells) count8011160201150101390<0.001 *
CD19+CD45+ (B cells) %80106.0015.0095.0015.00 0.858
CD4/CD8 ratio590.290.060.670.370.140.640.001 *
N: number of valid data entries. Test: related-samples Wilcoxon signed-rank test; IQR: interquartile range; arrows indicate rise (⬀) or drop (⬂) of the given parameter from baseline to post-treatment times; * statistically significant result (p < 0.05).
Table 4. Correlation of immunophenotyping parameters with HIV RNA/mL plasma levels at baseline and outcome (N = 82).
Table 4. Correlation of immunophenotyping parameters with HIV RNA/mL plasma levels at baseline and outcome (N = 82).
TimeParameter Viral Load (HIV RNA/mL Plasma)NK CellsB Lymphocytes
ρp-Valueρp-Valueρp-Value
Pre-treatmentViral Load--−0.260.020 *−0.180.114
CD16+CD56+−0.260.020*--0.58<0.001 *
CD19+CD45+−0.180.1140.58<0.001 *--
Post-treatmentViral Load --−0.150.187−0.120.283
CD16+CD56+−0.150.187--0.46<0.001 *
CD19+CD45+−0.120.2830.46<0.001 *--
ρ: Spearman’s rho; viral load: HIV RNA/mL plasma; * statistically significant result (p < 0.05).
Table 5. Correlation of phenotyping parameters with HIV RNA/mL plasma (pooled data).
Table 5. Correlation of phenotyping parameters with HIV RNA/mL plasma (pooled data).
Parameter ≤500501–30003001–10,00010,001–30,000>30,000Kruskal–Wallis TestBivariate Correlation
MedianIQRMedianIQRMedianIQRMedianIQRMedianIQRStatisticsp-ValueSpearman’s Rhop-Value
Total WBC4.82.524.451.924.491.234.532.813.943.138.650.070−0.150.005 *
Lymphocyte count1.9721.543.152.2911.12.331.441.548.523.750.441−0.050.379
Platelets 27211033113925938.82261422711329.43<0.001 *−0.21<0.001 *
Monocyte count0.520.590.540.770.565.340.570.320.586.50.520.9710.040.524
CD3+CD45+ count15641450118516841055926128758586798041.07<0.001 *−0.32<0.001 *
CD3+CD45+ percentage75.5157610743790.7574164.280.369−0.180.731
CD3+CD4+CD45+ count41550516718443240241215511025556.04<0.001 *−0.38<0.001 *
CD3+CD4+CD45+ %19.524.312153016249.7571738.76<0.001 *−0.27<0.001 *
CD3+CD8+CD45+ count893885711111053552187159155691014.170.007 *−0.170.002 *
CD3+CD8+CD45+ %4530.864234314.34956122.523.83<0.001 *0.22<0.001 *
CD16+CD56+ (NK) count246264186238839015014512814836.52<0.001 *−0.29<0.001 *
CD16+CD56+ (NK) %118.25949.5694.51385.890.2080.020.776
CD19+CD45+ (B cells) count20925916313915918012922610512841.81<0.001 *−0.34<0.001 *
CD19+CD45+ (B cells) %10108111410.3114.75108.52.210.698−0.060.266
CD4/CD8 ratio0.470.840.140.3070.790.120.470.250.120.27848.76<0.001 *−0.35<0.001 *
IQR: interquartile range; * statistically significant result (p < 0.05). The bakcground color for the last two columns marks a separation from the other statistics.
Table 6. Diagnostic accuracy of NK and B cell levels in indicating AIDS status among HIV patients.
Table 6. Diagnostic accuracy of NK and B cell levels in indicating AIDS status among HIV patients.
Parameter AUC (95%CI)SEp-ValueYouden’s IndexCutoff ValueSensitivitySpecificity
NK cell count0.629 (0.568, 0.689)0.031<0.00010.242<73 cells/mm395.5%28.7%
B cell count0.683 (0.626, 0.740)0.029<0.00010.290<166.5 cells/mm363.6%65.4%
CD3+ count0.809 (0.761, 0.857)0.024<0.00010.547<876.5 cells/mm391.4%63.2%
AUC: area under the corresponding Receiver Operating Characteristics (ROC) curve; SE: standard error.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Al-Mughales, J. Immuno-Diagnostic Interest in Monitoring CD16+CD56+ (Natural Killer) Cells and CD19+CD45+ (B Lymphocytes) in Individuals Newly Diagnosed with HIV in a Tertiary Care Center. J. Clin. Med. 2024, 13, 1154. https://doi.org/10.3390/jcm13041154

AMA Style

Al-Mughales J. Immuno-Diagnostic Interest in Monitoring CD16+CD56+ (Natural Killer) Cells and CD19+CD45+ (B Lymphocytes) in Individuals Newly Diagnosed with HIV in a Tertiary Care Center. Journal of Clinical Medicine. 2024; 13(4):1154. https://doi.org/10.3390/jcm13041154

Chicago/Turabian Style

Al-Mughales, Jamil. 2024. "Immuno-Diagnostic Interest in Monitoring CD16+CD56+ (Natural Killer) Cells and CD19+CD45+ (B Lymphocytes) in Individuals Newly Diagnosed with HIV in a Tertiary Care Center" Journal of Clinical Medicine 13, no. 4: 1154. https://doi.org/10.3390/jcm13041154

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop