Next Article in Journal
Establishing Core Competencies and a Professional Curriculum for the Care Service Department in Vocational High Schools in Taiwan
Previous Article in Journal
The Lived Experiences of Fathers in Mealtimes: A Thematic Synthesis of Qualitative Literature
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Prediction Model for the Risk of HIV Infection among MSM in China: Validation and Stability

1
School of Public Health, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China
2
Department of Occupational Health, School of Public Health, China Medical University, Shenyang 110122, China
*
Authors to whom correspondence should be addressed.
These authors contributed equally to this work.
Int. J. Environ. Res. Public Health 2022, 19(2), 1010; https://doi.org/10.3390/ijerph19021010
Submission received: 30 October 2021 / Revised: 8 January 2022 / Accepted: 11 January 2022 / Published: 17 January 2022
(This article belongs to the Section Infectious Disease Epidemiology)

Abstract

:
The impact of psychosocial factors on increasing the risk of HIV infection among men who have sex with men (MSM) has attracted increasing attention. We aimed to develop and validate an integrated prediction model, especially incorporating emerging psychosocial variables, for predicting the risk of HIV infection among MSM. We surveyed and collected sociodemographic, psychosocial, and behavioral information from 547 MSM in China. The participants were split into a training set and a testing set in a 3:1 theoretical ratio. The prediction model was constructed by introducing the important variables selected with the least absolute shrinkage and selection operator (LASSO) regression, applying multivariate logistic regression, and visually assessing the risk of HIV infection through the nomogram. Receiver operating characteristic curves (ROC), Kolmogorov–Smirnov test, calibration plots, Hosmer–Lemeshow test and population stability index (PSI) were performed to test validity and stability of the model. Four of the 15 selected variables—unprotected anal intercourse, multiple sexual partners, involuntary subordination and drug use before sex—were included in the prediction model. The results indicated that the comprehensive prediction model we developed had relatively good predictive performance and stability in identifying MSM at high-risk for HIV infection, thus providing targeted interventions for high-risk MSM.

1. Introduction

In 2020, there are 38 million people living with HIV worldwide, thus making HIV infection one of the most serious public health problems worldwide. Men who have sex with men (MSM) account for an integral part of new HIV infections, which has posed an unprecedented challenge to global HIV prevention and control [1]. In China, MSM also constitute a key population at high risk of HIV infection. The risk factors of HIV infection in MSM identified by current research mainly include sociodemographic, behavioral, and psychosocial aspects, such as sex, age, unprotected anal intercourse (UAI), multiple sexual partners (MSP), alcohol and drug use before sex and depression [2,3]. Given the variety of risk factors of HIV infection among MSM, identifying MSM at greatest risk for HIV infection by using predictive models could facilitate targeted prevention interventions specific to this population, while permitting more focused delivery of prevention resources. Therefore, it is necessary to predict the risk of HIV infection in a more holistic perspective for early targeted screening and prevention among MSM.
Since sexual behavior is considered as the main way of HIV transmission and infection among MSM, most HIV prevention interventions have focused on directly reducing unprotected sexual behavior, while neglecting the impact of psychological factors related to HIV infection. As a sexual minority group, the MSM is subject to stigma, discrimination, and isolation from society, which in turn causes a range of defensive social mentality encompassing cognition, feelings and behaviors, and ultimately contributes to various psychosocial problems [4,5]. The resulting psychosocial problems may lead to high-risk behaviors-such as UAI and MSP, which might provide immediately physical or emotional gratification but could increase the risk of HIV transmission and infection in the long run.
Considering the characteristics of MSM and the surrounding social environment, involuntary subordination, sexual impulsivity and social support are emerging as new public health concerns among MSM [6,7,8]. Involuntary subordination (IS) is a mechanism adapted by humans for competing resources that switch off fighting behaviors when a losing organism cannot struggle anymore (thus saving the organism from injury) [9]. Individuals with high levels of IS are more difficult to accept defeat or escape from a defeated situation, and then give up resistance or abandon themselves. This possibly makes MSM vulnerable to HIV infection. However, there are few studies focused on direct relationship between involuntary subordination and HIV infection among MSM. Recently, a cross-section study revealed that high level of IS was associated with multiple sexual partners [10]. As we all know, multiple sexual partners, as a sexual risk behavior, may increase the risk of HIV infection. Therefore, we speculate that IS, as a psychosocial variable associated with sexual behavior, may be a significant predictor of HIV infection among MSM. Sexual compulsivity (SC) is a trait characterized by sexual fantasies and behaviors that increase in intensity and frequency over time and, therefore, interfere with personal, interpersonal, or career pursuits [11,12]. Compared with MSM lacking this characteristic, MSM with sexual impulsivity experienced a higher incidence of HIV and sexually transmitted infections [13]. The MSM engaging in compulsive sexual behavior may also partly stem from the pressures and stigma they feel as a sexual minority. In addition, some evidence suggested that high levels of social support were associated with decreased likelihood of engaging in sexual risk behaviors [14,15,16,17]. Therefore, social support that can provide emotional and material resources to MSM could help reduce the risk of HIV infection [18,19]. In summary, especially among MSM, multidimensional factors, particularly emerging psychosocial factors, need to be considered in predicting the risk of HIV infection.
There are many studies at home and abroad that have developed HIV infection risk assessment tools targeting MSM, including the Menza score [20], the Denver score [21], and the San Diego Early Test score [22]. In addition, prediction models specific to Chinese MSM have been developed to estimate HIV infection risk [23,24]. However, all the models mentioned above included only socio-demographic and behavioral factors, not psychosocial factors, thus could not comprehensively predict the risk of HIV infection among MSM. To our knowledge, there are few studies using machine learning methods to construct multidimensional HIV infection prediction models that include emerging psychosocial variables. In this study, we aimed to establish and validate an integrated HIV infection risk assessment tool that incorporated variables selected from demographic, psychosocial and behavioral factors using machine learning approach, and inform targeted interventions in MSM.

2. Methods

2.1. Study Population and Eligibility Criteria

A cross-sectional study was performed among MSM in four districts: Changning, Jing’an, Zhabei and Pudong New Area in Shanghai. Individuals were eligible to participate only if they were biologically male, at least 18 years of age, and reported to have had sex with men during the past 6 months. A total of 567 MSM agreed to participate in this study, and 20 MSM were not included in the final statistical analysis due to not completing the whole questionnaire. The final participants for assessment thus included 547 MSM (response rate of 96%). A flow diagram of the study design is shown in Figure 1.

2.2. Participants and Procedure

The “snowball” technique, which is advantageous in addressing population covertness, was adopted to target the participants. Initially, with the help of the local Center for Disease Control and Prevention and non-government organizations, we targeted 5 to 10 individuals (“seeds”) compatible with the inclusion criteria within each district. Subsequently, those harder-to-reach MSM of the same social-cultural background were identified through the indication of the so-called “seeds”, and so forth, forming the snowball. Anonymous face-to-face interviews were carried out between our outreach workers and the participating MSM. Participants were briefed about the study in detail. Written informed consent was obtained from the participants before the interview. Interviewers were requested to sign a form pledging that they had made clear explanations to the participants and answered all questions before the participants signed the informed consent. We made sure that, in the whole process of dealing with our participants, we strictly complied with American Psychological Association ethical standards. The recruitment procedure and the design of this study were approved by the Ethics Committee of School of Public Health, Shanghai Jiao Tong University.

2.3. Measurement

2.3.1. Socio-Demographic Variables

Socio-demographic information collected included age, income, current marital status, employment status, education level, registered residence status, self-reported sexual orientation.

2.3.2. Psychosocial Variables

Involuntary Subordination

Involuntary subordination was measured using the 32-item, 5-point Likert-type Involuntary Subordination Questionnaire [9]. Answers were rated ranging from 1 (strongly disagree) to 5 (strongly agree) where a higher total score indicates a higher level of involuntary subordination (Cronbach’s alpha coefficient = 0.883; range 35–136).

Sexual Compulsivity

The 10-item Likert-type Sexual Compulsivity Scale was applied to measure individual’s out-of–control sexual thoughts and behaviors [25]. Answers were rated ranging from 1 (strongly disagree) to 4 (strongly agree), with a higher total score indicating higher sexual compulsivity (Cronbach’s alpha coefficient = 0.854; range 10–40).

Social Support

We employed the Multiple Scales of Perceived Social Support to assess subjectively recognized social support from family, friends, and significant others [26,27]. The scale is a 7-point Likert scale with 12 items, ranging from 1 (very strongly disagree) to 7 (very strongly agree). A higher total score indicated higher social support (Cronbach’s alpha coefficient = 0.932; range 18–84).

2.3.3. Behavioral Variables

Multiple Sexual Partners

We asked participants “How many male sexual partners have you had anal sex with in the past 6 months?”. The options included “only one”, “two”, “three”, “four”, and “five or more”. We classified those who answer this question with more than one male partner as having existing MSP.

Unprotected Anal Intercourse

Participants were asked about the frequency of condoms use when they had sex with any male partner in the past 6 months. Responses were rated ranging from 1 (never) to 5 (every time). Unless the answer was “every time”, they were described as “engaging in UAI”.

Alcohol Use before Sex

We measured drug use by asking the participants whether they had drinks before sex. Those without a “never” answer was regarded as groups who drink before sex.

Drug use before Sex

We measured drug use by asking the participants whether they had used any drugs including ecstasy (3,4 methylenedioxymethamphetamine), crystal meth (methamphetamine), marijuana, cocaine, and others. Those without a “never” answer was regarded as groups who use drugs before sex.

Voluntary HIV Counseling and Testing (VCT)

Participants were asked if they had voluntarily received HIV counseling and testing in the past six months, and participant’s blood samples were voluntarily drawn and tested for HIV infection. Then, they were privately informed of their serostatus. If participants have a positive result, they are referred to a local hospital specializing in HIV/AIDS treatment.

2.4. Statistical Analysis

The 547 participants were randomly divided into a training set and a testing set for internal validation at the theoretical ratio of 3:1 [28,29]. The data from the training and testing sets were guaranteed not to be replaced during the analysis, thus improving the reliability and robustness of the study findings. Categorical variables are presented as frequencies and percentages and continuous variables are described as mean and SD (normal distribution) or median and quartile (non-normal distribution). Chi-square test, corrected Chi-square test, and Fisher exact test were used to assess differences in categorical variables between training and testing set. Differences in continuous variables between the training and testing set were evaluated using Unpaired t test (normal distribution) and Mann–Whitney U test (non-normal distribution). Previous studies have shown that least absolute shrinkage and selection operator (LASSO) regression analysis outperformed several other machine learning approaches [30,31,32]. Therefore, LASSO was adopted to reduce the data dimension of the training set and to pick out the most influential and optimal factors for the predictive model. LASSO regression analysis minimizes the prediction error of continuous dependent variables by imposing constraints on the model parameters that causes the regression coefficients of certain predictor variables to shrink to zero. Variables with non-zero regression coefficients are considered risk factors that are strongly correlated with the response variable. On the contrary, predictor variables with regression coefficients equal to zero after the contraction process are excluded from the model. The 10-fold cross-validation solves the problem of overfitting variables and multicollinearity during model extrapolation by centralizing and standardizing the included variables for optimizing lambda. After the risk factors were identified by LASSO regression analysis, a prediction model was constructed by using multiple logistic regression analysis on the training set, which was presented in the form of a nomogram. Then, a predictive nomogram based on identified risk factors was established as a tool to assess HIV infection risk visually and quantitatively among MSM.
The discrimination, calibration and stability of the HIV infection risk nomogram were thoroughly assessed based on the training set and the testing set, respectively. Discrimination of nomogram was quantified using the area under the curve (AUC) of the receiver operating characteristic (ROC) and Kolmogorov–Smirnov (K-S) test. The Kolmogorov–Smirnov test was used to assess the agreement between the predicted and actual probabilities of HIV infection risk and higher K-S values indicating greater ability of the model to discriminate the samples. It is generally considered that K-S > 0.2 denotes a strong risk differentiation ability of the model developed in the study. The calibration plots and Hosmer–Lemeshow test were used to evaluate the calibration of prediction model. Internal and external consistency of the discrimination and calibration performance were measured using bootstrap resampling procedure. In addition, population stability index (PSI) was introduced to monitor model stability by assessing the degree of population bias based on the difference between the expected distribution of the training sample and the actual distribution of the test sample. Lower PSI means smaller difference between the two datasets and better stability of the model, and generally considered as PSI between 0 and 0.1, indicating no significant population bias and excellent stability of the model.
All analyses were conducted with R statistical software (version 3.6.1) equipped with the “caret”, “glmnet”, “rms”, “rmda” packages. All tests were two-sided and p ≤ 0.05 was set as the level of significant difference.

3. Results

3.1. Characteristics of Participants

Among the 547 MSM in our study, 59 were diagnosed as HIV-positive (including 24 previously HIV-positive participants). The average age of the participants was 28.0 (25.0–33.0) years old. The specific demographic, psychosocial and behavioral characteristics in the training set and testing set were summarized in Table 1. There is no statistically significant difference between the characteristics of the training and testing datasets, which indicates that the random assignment to each subset is successful.

3.2. Construction and Development of a Risk Assessment Tool for HIV Infection among MSM

Significant predictors were selected by the LASSO method based on a minimum 1 standard error (1-SE) criterion to obtain the most regularized and efficient prediction model (Figure 2a,b). Among the 15 high-dimensional characteristics of MSM, four variables with non-zero coefficients were identified as risk factors closely associated with HIV infection including IS, UAI, MSP, and drug before sex. Those significant factors selected were included in the multivariate logistic regression to establish the prediction model. Logistic regression analysis demonstrated that all included variables were independent risk factors for HIV infection among MSM, and their coefficients were shown in Table 2. Then, the nomogram diagrams corresponding to the predictive model incorporating multidimensional risk factors was developed as a tool to assess the risk of HIV infection (Figure 3). Each selected risk factor was assigned a corresponding score according to its value on the nomogram. For example, suppose a man had unprotected sexual behavior with multiple sexual partners and took drugs before having sex. With an IS score equal to 32, the total nomogram point for the man was 110. As the nomogram showed, the estimated HIV infection risk of the man was approximately 0.3.

3.3. Internal Validation of Prediction Model

The ROC of the prediction model is presented in Figure 4a,b. In the training set, the AUC value was 0.764 (95% CI 0.684–0.845), which indicated that about 76% of the individual HIV infection risk would be correctly predicted by the model. In addition, the K-S value of 0.54 also indicated that the established prediction model has a strong risk differentiation ability (Figure 5). In the testing set, the AUC-ROC was 0.832 (95%CI 0.730–0.935) and the K-S value was 0.44. Therefore, the prediction model has moderately high discrimination for predicting the HIV infection risk among MSM. The calibration curve indicated good agreement between the actual HIV infection probabilities and predicted probabilities made using nomogram in both the training set and testing set (Figure 6a,b). From the Hosmer–Lemeshow test, the predicted and actual probability were highly consistent (training set, p = 0.474; testing set, p = 0.252). The PSI of 0.16 revealed that the model was stable due to the relatively smaller difference between the distributions of the training and testing sets (Figure 7). To summarize the internal random splitting validation results above, this model has a good predictive capability.

4. Discussion

Our study identified four risk factors of HIV infection in MSM from 15 sociodemographic, behavioral, and psychosocial predictors based on LASSO regression, including UAI, MSP, IS, and drug use before sex. We constructed a predictive model that incorporated the above risk factors of HIV infection in MSM, and further developed and validated a HIV infection risk nomogram. The nomogram we developed targeted at MSM has moderate predictive discrimination and calibration in the training set. In addition, the lower PSI also confirmed the relatively good stability of the prediction model. Finally, the internal validation of the nomogram supported its good predictive performance and accuracy in predicting HIV infection risk among MSM.

4.1. Association of UAI and MSP with HIV Infection among MSM

UAI and MSP were well-known as the main routes of HIV infection in MSM [33,34]. From adolescents to older men, UAI remained common and put MSM of all ages at higher risk for acquiring and infecting HIV [35,36,37]. In addition to UAI, the mean number of total sexual partners in MSM had increased significantly in recent years, and MSP was emerging as a high-risk factor for HIV infection [38]. Some studies provided further evidence that unprotected sexual behaviors with multiple sexual partners expedited HIV transmission among HIV-positive MSM [39,40]. Our study also confirmed that UAI and MSP were both independent predictors for HIV infection among MSM. However, the above two predictors were not the main HIV infection contributors in this nomogram, which may be related to the duration of the survey regarding previous sexual behavior. It is difficult to assess the impact of high-risk sexual behavior on HIV infection during the lifetime of MSM in cross-sectional studies; therefore, most cross-sectional studies generally investigate short-term (usually the past six months) high-risk sexual behavior among MSM.

4.2. Association of Drug Use before Sex with HIV Infection among MSM

Our study found that drug use before sex was a significant predictor for HIV infection among MSM. Previous studies have also reported that drug use was associated with high HIV infection risk [41,42]. This relationship is postulated to be attributed to the acute effects of drug use that can enhance sexual desire and increase intoxicating highs, thus stimulating risk-taking sexual behaviors, such as UAI, MSP [43,44]. In addition to illegal drugs, some popular recreational drugs that but not defined as illegal, such as rush poppers and erectile dysfunction medications, have similarly been confirmed to play a role in increasing rates of sexual risk behavior and HIV infection [45,46,47,48]. Furthermore, several epidemiological studies have also shown a positive dose–response relationship between frequency and number of drugs used or polydrug use and HIV risk to support our findings [49,50,51].

4.3. Association of IS with HIV Infection among MSM

Interestingly, our study found the risky role of involuntary subordination in HIV infection among MSM. As far as we know, we were the first to reveal this direct association between involuntary subordination and HIV infection risk among Chinese MSM. We confirmed that the higher the level of IS, the greater the risk of HIV infection. This may be explained by the fact that MSM suffered more discrimination, isolation, and stress than heterosexuals, thus making MSM more prone to feel stuck and low perception (poor social comparison), give up resistance, and abandon themselves when confronted with adversity, eventually resulting in risky sexual behavior [10] and mental disorder such as anxiety and depression [8]. Recently, some studies also confirmed that involuntary subordination fully mediated the relationship between self-esteem and depression among MSM [52,53]. In addition, IS was found to be associated with neediness [9] and dependency [54], which was also compatible with the characteristics of sexual minorities compared to heterosexuals. Currently, the concept of IS was expanded into a range of interconnected feelings and perceptions, including submissive behaviors, entrapment, social defeat and poor social comparison, which may stimulate multiple risk pathways that synergistically increase the risk of HIV infection, thus partly explaining the higher contribution of IS in the nomogram. This finding could inform future research, especially when additional mental health variables are available for predictive purposes. More research is needed to focus on the potential impact of involuntary subordination on HIV infection and provide further insights into the unclear causal relationship between IS and HIV infection.

4.4. Limitation

There were several limitations to this study. Firstly, since this study was a cross-sectional study, the ability to infer causality from these findings was insufficient. Secondly, participants’ responses might not be completely honest because sexual-related questions are relatively private in China and the questionnaire in our study was completed through face-to-face interviews. Therefore, it was inevitable to generate biases from the self-reported data collected. Finally, given the difficulty of including all psychosocial and behavioral factors that may be associated with HIV infection in the model, large prospective cohort studies containing more comprehensive variables are needed in the future to further expand the applicability of this HIV infection risk assessment tool.

5. Conclusions

Our study found that MSP, UAI, IS, and drugs before sex were significant predictors in terms of identifying risk for HIV infection. Furthermore, we developed and externally validated an HIV infection risk assessment model especially incorporating psychosocial and behavioral factors for MSM in China. The risk nomogram had a relatively high performance and stability in predicting HIV infection risk among MSM.

Author Contributions

Conceptualization, Y.C. and Y.W.; methodology, F.H.; software, Y.D.; validation, Y.D., S.L. and Y.L.; formal analysis, Y.D.; investigation and data curation, D.X., X.Y., C.X., H.C., R.W. and J.D.; writing—original draft preparation, Y.D. and S.L.; writing—review and editing, Y.D. and F.H.; visualization, Y.D.; supervision, Y.C. and Y.W.; project administration, Y.W. All authors have read and agreed to the published version of the manuscript.

Funding

This work was support by Shanghai Three-year Action Plan for Public Health Grant funded by the Shanghai Municipal Health Commission (Grant number: GWV-10.1-XK15, GWV-10.1-XK18, GWV-10.2-XD13); The National Nature Science Funds of China (No.71603166, No.71673187).

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Shanghai Jiao Tong University School of Medicine Public Health Ethics Committee.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available on reasonable request from the corresponding author.

Acknowledgments

We acknowledge all participants and interviewers involved in our research.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. 2020 Global AIDS Update—Seizing the Moment—Tackling Entrenched Inequalities to End Epidemics. Available online: https://www.unaids.org/en/resources/documents/2020/global-aids-report (accessed on 6 July 2020).
  2. Poteat, T.; Ackerman, B.; Diouf, D.; Ceesay, N.; Mothopeng, T.; Odette, K.Z.; Kouanda, S.; Ouedraogo, H.G.; Simplice, A.; Kouame, A.; et al. HIV prevalence and behavioral and psychosocial factors among transgender women and cisgender men who have sex with men in 8 African countries: A cross-sectional analysis. PLoS Med. 2017, 14, e1002422. [Google Scholar] [CrossRef] [Green Version]
  3. Rutledge, S.E.; Jemmott, J.B., 3rd; O’Leary, A.; Icard, L.D. What’s In an Identity Label? Correlates of Sociodemographics, Psychosocial Characteristics, and Sexual Behavior Among African American Men Who Have Sex With Men. Arch. Sex. Behav. 2018, 47, 157–167. [Google Scholar] [CrossRef]
  4. Frye, V.; Nandi, V.; Egan, J.; Cerda, M.; Greene, E.; Van Tieu, H.; Ompad, D.C.; Hoover, D.R.; Lucy, D.; Baez, E.; et al. Sexual orientation- and race-based discrimination and sexual HIV risk behavior among urban MSM. AIDS Behav. 2015, 19, 257–269. [Google Scholar] [CrossRef]
  5. Reilly, K.H.; Neaigus, A.; Jenness, S.M.; Wendel, T.; Marshall, D.M.; Hagan, H. Experiences of Discrimination and HIV Risk Among Men Who Have Sex With Men in New York City. Am. J. Mens. Health 2016, 10, 505–514. [Google Scholar] [CrossRef] [Green Version]
  6. Hirshfield, S.; Lewis, K.E.; Silver, M.; Gordon, R.J. Interpersonal Stigma, Mental Health, and Sexual Compulsivity Among an Online U.S. Sample of Men Who Have Sex with Men Living with HIV. AIDS Behav. 2021. online ahead of print. [Google Scholar] [CrossRef]
  7. Mo, P.K.H.; Chen, X.; Lam, E.H.K.; Li, J.; Kahler, C.W.; Lau, J.T.F. The Moderating Role of Social Support on the Relationship Between Anxiety, Stigma, and Intention to Use Illicit Drugs Among HIV-Positive Men Who Have Sex with Men. AIDS Behav. 2020, 24, 55–64. [Google Scholar] [CrossRef]
  8. Zhang, S.; Wang, S.; Wang, Z.; Wang, Y.; Jiang, X.; Xu, G.; Cai, Y. The association between involuntary subordination and common mental disorders among men who have sex with men (MSM) in Shanghai, China. BMC Psychiatry 2019, 19, 369. [Google Scholar] [CrossRef] [Green Version]
  9. Sturman, E.D. Involuntary subordination and its relation to personality, mood, and submissive behavior. Psychol. Assess. 2011, 23, 262–276. [Google Scholar] [CrossRef]
  10. Wang, Z.; Zhao, X.; Zhang, Z.; Luo, M.; Shen, Q.; Dong, Y.; Wang, Y.; Cai, Y. Co-Occurring Psychosocial Problems and Multiple Sexual Partners among Men Who Have Sex with Men in Shanghai, China: A Syndemic Approach. J. Sex Res. 2018, 55, 892–901. [Google Scholar] [CrossRef]
  11. Brown, M.J.; Serovich, J.M.; Kimberly, J.A. Outcome Expectancy and Sexual Compulsivity Among Men Who Have Sex with Men Living with HIV. AIDS Behav. 2016, 20, 1667–1674. [Google Scholar] [CrossRef] [Green Version]
  12. Jerome, R.C.; Woods, W.J.; Moskowitz, J.T.; Carrico, A.W. The Psychological Context of Sexual Compulsivity Among Men Who Have Sex with Men. AIDS Behav. 2016, 20, 273–280. [Google Scholar] [CrossRef] [Green Version]
  13. Parsons, J.T.; Grov, C.; Golub, S.A. Sexual compulsivity, co-occurring psychosocial health problems, and HIV risk among gay and bisexual men: Further evidence of a syndemic. Am. J. Public Health 2012, 102, 156–162. [Google Scholar] [CrossRef]
  14. Qiao, S.; Li, X.; Stanton, B. Social support and HIV-related risk behaviors: A systematic review of the global literature. AIDS Behav. 2014, 18, 419–441. [Google Scholar] [CrossRef] [Green Version]
  15. Saleh, L.D.; van den Berg, J.J.; Chambers, C.S.; Operario, D. Social support, psychological vulnerability, and HIV risk among African American men who have sex with men. Psychol. Health 2016, 31, 549–564. [Google Scholar] [CrossRef]
  16. Hermanstyne, K.A.; Green, H.D., Jr.; Cook, R.; Tieu, H.V.; Dyer, T.V.; Hucks-Ortiz, C.; Wilton, L.; Latkin, C.; Shoptaw, S. Social Network Support and Decreased Risk of Seroconversion in Black MSM: Results of the BROTHERS (HPTN 061) Study. J. Acquir. Immune Defic. Syndr. 2018, 78, 163–168. [Google Scholar] [CrossRef]
  17. Shuper, P.A.; MacLachlan, D.J.; Joharchi, N.; Guimond, T.H.; Maxwell, J.; Adam, B.D. HIV Risk and Protective Factors in the Context of Alcohol and Substance Use During Pride. AIDS Behav. 2018, 22, 2797–2806. [Google Scholar] [CrossRef]
  18. Cohen, S.; Wills, T.A. Stress, social support, and the buffering hypothesis. Psychol. Bull. 1985, 98, 310–357. [Google Scholar] [CrossRef]
  19. Wohl, A.R.; Galvan, F.H.; Myers, H.F.; Garland, W.; George, S.; Witt, M.; Cadden, J.; Operskalski, E.; Jordan, W.; Carpio, F. Social support, stress and social network characteristics among HIV-positive Latino and African American women and men who have sex with men. AIDS Behav. 2010, 14, 1149–1158. [Google Scholar] [CrossRef]
  20. Menza, T.W.; Hughes, J.P.; Celum, C.L.; Golden, M.R. Prediction of HIV acquisition among men who have sex with men. Sex. Transm. Dis. 2009, 36, 547–555. [Google Scholar] [CrossRef] [Green Version]
  21. Haukoos, J.S.; Lyons, M.S.; Lindsell, C.J.; Hopkins, E.; Bender, B.; Rothman, R.E.; Hsieh, Y.H.; Maclaren, L.A.; Thrun, M.W.; Sasson, C.; et al. Derivation and validation of the Denver Human Immunodeficiency Virus (HIV) risk score for targeted HIV screening. Am. J. Epidemiol. 2012, 175, 838–846. [Google Scholar] [CrossRef] [Green Version]
  22. Hoenigl, M.; Weibel, N.; Mehta, S.R.; Anderson, C.M.; Jenks, J.; Green, N.; Gianella, S.; Smith, D.M.; Little, S.J. Development and validation of the San Diego Early Test Score to predict acute and early HIV infection risk in men who have sex with men. Clin. Infect. Dis. 2015, 61, 468–475. [Google Scholar] [CrossRef] [Green Version]
  23. Hu, P.; Zhong, F.; Cheng, W.B.; Xu, H.F.; Ling, L. Study on the infectious risk model of AIDS among men who have sex with men in Guangzhou. Zhonghua Liu Xing Bing Xue Za Zhi 2012, 33, 667–671. [Google Scholar]
  24. Yin, L.; Zhao, Y.; Peratikos, M.B.; Song, L.; Zhang, X.; Xin, R.; Sun, Z.; Xu, Y.; Zhang, L.; Hu, Y.; et al. Risk Prediction Score for HIV Infection: Development and Internal Validation with Cross-Sectional Data from Men Who Have Sex with Men in China. AIDS Behav. 2018, 22, 2267–2276. [Google Scholar] [CrossRef]
  25. Kalichman, S.C.; Rompa, D. Sexual sensation seeking and Sexual Compulsivity Scales: Reliability, validity, and predicting HIV risk behavior. J. Pers. Assess. 1995, 65, 586–601. [Google Scholar] [CrossRef]
  26. Cella, D.F.; Cherin, E.A. Quality of life during and after cancer treatment. Compr. Ther. 1988, 14, 69–75. [Google Scholar]
  27. Dahlem, N.W.; Zimet, G.D.; Walker, R.R. The Multidimensional Scale of Perceived Social Support: A confirmation study. J. Clin. Psychol. 1991, 47, 756–761. [Google Scholar] [CrossRef]
  28. Mo, R.; Shi, R.; Hu, Y.; Hu, F. Nomogram-Based Prediction of the Risk of Diabetic Retinopathy: A Retrospective Study. J. Diabetes Res. 2020, 2020, 7261047. [Google Scholar] [CrossRef]
  29. Wu, B.; Niu, Z.; Hu, F. Study on Risk Factors of Peripheral Neuropathy in Type 2 Diabetes Mellitus and Establishment of Prediction Model. Diabetes Metab. J. 2021, 45, 526–538. [Google Scholar] [CrossRef]
  30. Krakower, D.S.; Gruber, S.; Hsu, K.; Menchaca, J.T.; Maro, J.C.; Kruskal, B.A.; Wilson, I.B.; Mayer, K.H.; Klompas, M. Development and validation of an automated HIV prediction algorithm to identify candidates for pre-exposure prophylaxis: A modelling study. Lancet HIV 2019, 6, e696–e704. [Google Scholar] [CrossRef]
  31. Marcus, J.L.; Hurley, L.B.; Krakower, D.S.; Alexeeff, S.; Silverberg, M.J.; Volk, J.E. Use of electronic health record data and machine learning to identify candidates for HIV pre-exposure prophylaxis: A modelling study. Lancet HIV 2019, 6, e688–e695. [Google Scholar] [CrossRef]
  32. Yuan, Z.; Zhou, B.; Meng, S.; Jiang, J.; Huang, S.; Lu, X.; Wu, N.; Xie, Z.; Deng, J.; Chen, X.; et al. Development and external-validation of a nomogram for predicting the survival of hospitalised HIV/AIDS patients based on a large study cohort in western China. Epidemiol. Infect. 2020, 148, e84. [Google Scholar] [CrossRef] [Green Version]
  33. Aghaizu, A.; Wayal, S.; Nardone, A.; Parsons, V.; Copas, A.; Mercey, D.; Hart, G.; Gilson, R.; Johnson, A.M. Sexual behaviours, HIV testing, and the proportion of men at risk of transmitting and acquiring HIV in London, UK, 2000–2013: A serial cross-sectional study. Lancet HIV 2016, 3, e431–e440. [Google Scholar] [CrossRef] [Green Version]
  34. Zhong, F.; Liang, B.; Xu, H.; Cheng, W.; Fan, L.; Han, Z.; Liang, C.; Gao, K.; Mai, H.; Qin, F.; et al. Increasing HIV and decreasing syphilis prevalence in a context of persistently high unprotected anal intercourse, six consecutive annual surveys among men who have sex with men in Guangzhou, China, 2008 to 2013. PLoS ONE 2014, 9, e103136. [Google Scholar] [CrossRef] [Green Version]
  35. Baggaley, R.F.; White, R.G.; Boily, M.C. HIV transmission risk through anal intercourse: Systematic review, meta-analysis and implications for HIV prevention. Int. J. Epidemiol. 2010, 39, 1048–1063. [Google Scholar] [CrossRef] [Green Version]
  36. Li, Y.Z.; Xu, J.J.; Qian, H.Z.; You, B.X.; Zhang, J.; Zhang, J.M.; Hu, Q.H.; Chu, Z.X.; Liu, S.Y.; Jiang, Y.J.; et al. High prevalence of HIV infection and unprotected anal intercourse among older men who have sex with men in China: A systematic review and meta-analysis. BMC Infect. Dis. 2014, 14, 531. [Google Scholar] [CrossRef] [Green Version]
  37. Yang, Z.; Huang, Z.; Dong, Z.; Zhang, S.; Han, J.; Jin, M. Prevalence of high-risky behaviors in transmission of HIV among high school and college student MSM in China: A meta-analysis. BMC Public Health 2015, 15, 1272. [Google Scholar] [CrossRef] [Green Version]
  38. Chapin-Bardales, J.; Rosenberg, E.S.; Sullivan, P.S.; Jenness, S.M.; Paz-Bailey, G. Trends in Number and Composition of Sex Partners Among Men Who Have Sex With Men in the United States, National HIV Behavioral Surveillance, 2008–2014. J. Acquir. Immune Defic. Syndr. 2019, 81, 257–265. [Google Scholar] [CrossRef]
  39. Castillo, R.; Konda, K.A.; Leon, S.R.; Silva-Santisteban, A.; Salazar, X.; Klausner, J.D.; Coates, T.J.; Cáceres, C.F. HIV and Sexually Transmitted Infection Incidence and Associated Risk Factors Among High-Risk MSM and Male-to-Female Transgender Women in Lima, Peru. J. Acquir. Immune Defic. Syndr. 2015, 69, 567–575. [Google Scholar] [CrossRef] [Green Version]
  40. Zhang, Y.; Chen, F.; Ding, F.; Lin, X.; Wang, X.; Liu, N.; Liu, X.; Wang, W.; Zhang, H. Unprotected sexual behaviors and related factors of HIV-positive MSM with multiple sexual partners. Zhonghua Liu Xing Bing Xue Za Zhi 2016, 37, 517–521. [Google Scholar] [CrossRef]
  41. Chen, J.; Huang, Y.L.; Chen, H.L.; Xia, J. Nitrite inhalants use, sexual behaviors and HIV/syphilis infection among men who have sex with men in Chongqing, China. Infect. Dis. Poverty 2020, 9, 127. [Google Scholar] [CrossRef]
  42. Nguyen, T.V.; Van Khuu, N.; Nguyen, P.D.; Tran, H.P.; Phan, H.T.T.; Phan, L.T.; Detels, R. Sociodemographic Factors, Sexual Behaviors, and Alcohol and Recreational Drug Use Associated with HIV Among Men Who Have Sex with Men in Southern Vietnam. AIDS Behav. 2016, 20, 2357–2371. [Google Scholar] [CrossRef]
  43. Carey, J.W.; Mejia, R.; Bingham, T.; Ciesielski, C.; Gelaude, D.; Herbst, J.H.; Sinunu, M.; Sey, E.; Prachand, N.; Jenkins, R.A.; et al. Drug use, high-risk sex behaviors, and increased risk for recent HIV infection among men who have sex with men in Chicago and Los Angeles. AIDS Behav. 2009, 13, 1084–1096. [Google Scholar] [CrossRef]
  44. Xu, J.J.; Zhang, C.; Hu, Q.H.; Chu, Z.X.; Zhang, J.; Li, Y.Z.; Lu, L.; Wang, Z.; Fu, J.H.; Chen, X.; et al. Recreational drug use and risks of HIV and sexually transmitted infections among Chinese men who have sex with men: Mediation through multiple sexual partnerships. BMC Infect. Dis. 2014, 14, 642. [Google Scholar] [CrossRef] [Green Version]
  45. Mao, X.; Leuba, S.I.; Hu, Q.; Yan, H.; Wang, Z.; Lu, L.; Zhuang, M.; Chen, X.; Fu, J.; Geng, W.; et al. Use of multiple recreational drugs is associated with new HIV infections among men who have sex with men in China: A multicenter cross-sectional survey. BMC Public Health 2021, 21, 354. [Google Scholar] [CrossRef]
  46. Rosińska, M.; Gios, L.; Nöstlinger, C.; Vanden Berghe, W.; Marcus, U.; Schink, S.; Sherriff, N.; Jones, A.M.; Folch, C.; Dias, S.; et al. Prevalence of drug use during sex amongst MSM in Europe: Results from a multi-site bio-behavioural survey. Int. J. Drug Policy 2018, 55, 231–241. [Google Scholar] [CrossRef]
  47. Guerras, J.M.; Hoyos Miller, J.; Agustí, C.; Chanos, S.; Pichon, F.; Kuske, M.; Cigan, B.; Fuertes, R.; Stefanescu, R.; Ooms, L.; et al. Association of Sexualized Drug Use Patterns with HIV/STI Transmission Risk in an Internet Sample of Men Who Have Sex with Men from Seven European Countries. Arch. Sex. Behav. 2021, 50, 461–477. [Google Scholar] [CrossRef]
  48. Park, J.W.; Dobs, A.S.; Ho, K.S.; Palella, F.J.; Seaberg, E.C.; Weiss, R.E.; Detels, R. Characteristics and Longitudinal Patterns of Erectile Dysfunction Drug Use Among Men Who Have Sex with Men in the U.S. Arch. Sex. Behav. 2021, 50, 2887–2896. [Google Scholar] [CrossRef] [PubMed]
  49. Santos, G.M.; Coffin, P.O.; Das, M.; Matheson, T.; DeMicco, E.; Raiford, J.L.; Vittinghoff, E.; Dilley, J.W.; Colfax, G.; Herbst, J.H. Dose-response associations between number and frequency of substance use and high-risk sexual behaviors among HIV-negative substance-using men who have sex with men (SUMSM) in San Francisco. J. Acquir. Immune Defic. Syndr. 2013, 63, 540–544. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  50. Guerras, J.M.; Hoyos, J.; García de Olalla, P.; de la Fuente, L.; Herrero, L.; Palma, D.; Del Romero, J.; García-Pérez, J.N.; Belza, M.J.; The Methysos Project, G. Comparison of Polydrug Use Prevalences and Typologies between Men Who Have Sex with Men and General Population Men, in Madrid and Barcelona. Int. J. Environ. Res. Public Health 2021, 18, 11609. [Google Scholar] [CrossRef]
  51. Sewell, J.; Miltz, A.; Lampe, F.C.; Cambiano, V.; Speakman, A.; Phillips, A.N.; Stuart, D.; Gilson, R.; Asboe, D.; Nwokolo, N.; et al. Poly drug use, chemsex drug use, and associations with sexual risk behaviour in HIV-negative men who have sex with men attending sexual health clinics. Int. J. Drug Policy 2017, 43, 33–43. [Google Scholar] [CrossRef] [PubMed]
  52. Chang, R.; Wang, H.; She, R.; Zhang, S.; Tsamlag, L.; Shen, Q.; Shi, Y.; Wang, Z.; Lau, J.T.F.; Wang, Y.; et al. Feelings of Entrapment and Defeat Mediate the Association Between Self-Esteem and Depression Among Transgender Women Sex Workers in China. Front. Psychol. 2019, 10, 2241. [Google Scholar] [CrossRef] [PubMed]
  53. Shen, Q.; Shi, Y.; Zhang, S.; Tsamlag, L.; Wang, H.; Chang, R.; Peng, Z.; Wang, Y.; Shang, M.; Cai, Y. How involuntary subordination and social support influence the association between self-esteem and depression: A moderated mediation model. BMC Psychiatry 2019, 19, 390. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  54. Gilbert, P.; Allan, S.; Trent, D.R. Involuntary subordination or dependency as key dimensions of depressive vulnerability? J. Clin. Psychol. 1995, 51, 740–752. [Google Scholar] [CrossRef]
Figure 1. Flow diagram of study design.
Figure 1. Flow diagram of study design.
Ijerph 19 01010 g001
Figure 2. Predictors selection using the least absolute shrinkage and selection operator (LASSO) binary logistic regression model: (a) A coefficient profile plot was constructed against the log (lambda) parameters. Four variables with nonzero coefficients were selected by deriving the optimal lambda; (b) Selection of optimal parameter (lambda) in the LASSO model used 10-fold cross-validation error curve and was based on 1 standard error of the minimum criteria (1-SE criteria). The partial likelihood deviance (binomial deviance) curve was plotted versus log (lambda). The dotted lines were drawn at the optimal values by the minimum criteria and 1-SE criteria.
Figure 2. Predictors selection using the least absolute shrinkage and selection operator (LASSO) binary logistic regression model: (a) A coefficient profile plot was constructed against the log (lambda) parameters. Four variables with nonzero coefficients were selected by deriving the optimal lambda; (b) Selection of optimal parameter (lambda) in the LASSO model used 10-fold cross-validation error curve and was based on 1 standard error of the minimum criteria (1-SE criteria). The partial likelihood deviance (binomial deviance) curve was plotted versus log (lambda). The dotted lines were drawn at the optimal values by the minimum criteria and 1-SE criteria.
Ijerph 19 01010 g002
Figure 3. Developed nomogram to assess the HIV infection risk among MSM.
Figure 3. Developed nomogram to assess the HIV infection risk among MSM.
Ijerph 19 01010 g003
Figure 4. Receiver operating characteristic (ROC) validation of the HIV infection risk nomogram prediction in training set and testing set: (a) The area under the receiver operating characteristic curve (AUC) represents the discrimination performance of the model in the training set; (b) The area under the receiver operating characteristic curve (AUC) represents the discrimination performance of the model in the testing set.
Figure 4. Receiver operating characteristic (ROC) validation of the HIV infection risk nomogram prediction in training set and testing set: (a) The area under the receiver operating characteristic curve (AUC) represents the discrimination performance of the model in the training set; (b) The area under the receiver operating characteristic curve (AUC) represents the discrimination performance of the model in the testing set.
Ijerph 19 01010 g004
Figure 5. Kolmogorov–Smirnov test curve of the HIV risk prediction model in the training set and in the testing set. The horizontal coordinate of the curve is the “threshold” (the overall sample is divided into 10 equal parts in probability order), and the vertical coordinate is the value of TPR (true positive rate) or FPR (false positive rate), ranging from 0 to 1. The red and blue solid lines indicate the HIV-positive and negative cases in the training set, respectively. The maximum vertical distance between these two curves (gray dashed line) is the KS test value, and the corresponding horizontal coordinate is the threshold value that classifies the model best. The orange and green dashed lines indicate the HIV-positive and negative cases in the testing set, respectively.
Figure 5. Kolmogorov–Smirnov test curve of the HIV risk prediction model in the training set and in the testing set. The horizontal coordinate of the curve is the “threshold” (the overall sample is divided into 10 equal parts in probability order), and the vertical coordinate is the value of TPR (true positive rate) or FPR (false positive rate), ranging from 0 to 1. The red and blue solid lines indicate the HIV-positive and negative cases in the training set, respectively. The maximum vertical distance between these two curves (gray dashed line) is the KS test value, and the corresponding horizontal coordinate is the threshold value that classifies the model best. The orange and green dashed lines indicate the HIV-positive and negative cases in the testing set, respectively.
Ijerph 19 01010 g005
Figure 6. Calibration curves for the prediction of HIV infection risk nomogram The y-axis represented the actual risk of HIV infection. The x-axis represented the predicted risk of HIV infection. The diagonal dotted line represents a perfect prediction by an ideal model, the solid line represents the performance of the training set (a) and testing set (b), with the results indicating that a closer fit to the diagonal dotted line represents a better prediction.
Figure 6. Calibration curves for the prediction of HIV infection risk nomogram The y-axis represented the actual risk of HIV infection. The x-axis represented the predicted risk of HIV infection. The diagonal dotted line represents a perfect prediction by an ideal model, the solid line represents the performance of the training set (a) and testing set (b), with the results indicating that a closer fit to the diagonal dotted line represents a better prediction.
Ijerph 19 01010 g006
Figure 7. Population distribution plot of HIV infection risk prediction model in the training and testing sets. The horizontal coordinate indicates the 10 binning interval of the general distribution, and the vertical coordinate indicates the percentage of the population. The red and dark green squares indicate the actual population distribution in each binning interval for the training and testing sets, respectively. The green solid line and the orange dashed line indicate the expected population distribution in each binning interval for the training and testing sets, respectively.
Figure 7. Population distribution plot of HIV infection risk prediction model in the training and testing sets. The horizontal coordinate indicates the 10 binning interval of the general distribution, and the vertical coordinate indicates the percentage of the population. The red and dark green squares indicate the actual population distribution in each binning interval for the training and testing sets, respectively. The green solid line and the orange dashed line indicate the expected population distribution in each binning interval for the training and testing sets, respectively.
Ijerph 19 01010 g007
Table 1. Demographic, psychosocial and behavioral characteristics of the 547 MSM enrolled in the study according to randomization to the training and testing sets.
Table 1. Demographic, psychosocial and behavioral characteristics of the 547 MSM enrolled in the study according to randomization to the training and testing sets.
CharacteristicTotal Populations
(n = 547)
Training Set
(n = 410, 75%)
Testing Set
(n = 137, 25%)
p-Value
Age (years old)28.0 (25.0, 33.0)28.0 (25.0, 33.0)27.0 (24.5, 34.0)0.933
Employment status 0.991
     Employed447 (81.7)335 (81.7)112 (81.8)
     Unemployed100 (18.3)75 (18.3)25 (18.2)
Highest education level 0.858
     Senior high school or less157 (28.7)119 (29.0)38 (27.7)
     College degree or above390 (71.3)291 (71.0)99 (72.3)
Current marital status 0.195
     Single434 (79.3)329 (80.2)105 (76.6)
     Married 182 (15.0)62 (15.1)20 (14.6)
     Divorced or widowed31 (5.7)19 (4.7)12 (8.8)
Income(CNY) 0.332
     <3000 2133 (24.3)99(24.1)34(24.8)
     3000–6000211 (38.6)152(37.1)59(43.1)
     >6000203 (37.1)159(38.8)44(32.1)
Residence status 0.079
     Local389 (71.1)127 (31.0)31 (22.6)
     Non-local158 (28.9)283 (69.0)106 (77.4)
Sexual orientation 0.714
     Non-homosexual157 (28.7)116 (28.3)41 (29.9)
     Gay/homosexual390 (71.3)294 (71.7)96 (70.1)
Have had a VCT 0.822
     No251 (45.9)187 (45.6)64 (46.7)
     Yes296 (54.1)223 (54.4)73 (53.3)
Alcohol use before having sex 0.415
     No278 (50.8)213 (52.0)65 (47.4)
     Yes269 (49.2)197 (48.0)72 (52.6)
Drug use before having sex 1.000
     No530 (96.9)397 (96.8)133 (97.1)
     Yes17 (3.1)13 (3.2)4 (2.9)
MSP 0.502
     No250 (45.7)184 (44.9)66 (48.2)
     Yes297 (54.3)226 (55.1)71 (51.8)
UAI 0.188
     No249 (45.5)180 (43.9)69 (50.4)
     Yes298 (54.5)230 (56.1)68 (49.6)
Involuntary subordination80.52 ± 18.0680.31 ± 18.2381.12 ± 17.570.649
Social support61.0 (51.0, 70.0)62.00 (51.0, 70.0)59.00 (50.5, 69.0)0.448
Sexual compulsivity23.0 (19.0, 26.0)23.00(19.0, 26.0)23.00(20.0, 26.0)0.906
1 Marital status refers to only heterosexual marriage. Homosexual marriage is still not legalized in Mainland China. 2 CNY 3000 equivalent to USD 450; 6000 equivalents to USD 900. Note: VCT, voluntary HIV counseling and testing; MSP, multiple sexual partners; UAI, unprotected anal intercourse. Data are mean ± SD, median (interquartile range) or n (%).
Table 2. Logistic regression analysis of the predictors for the risk of HIV infection among MSM.
Table 2. Logistic regression analysis of the predictors for the risk of HIV infection among MSM.
Intercept and VariablesEstimatePrediction Model
Wald ValuesOdds Ratio (95% CI)p-Value
Intercept−6.86251.8350.000 (0.000, 0.006)<0.001
UAI0.9315.6982.536 (1.219, 5.703)0.017
MSP1.0436.7202.837 (1.338, 6.591)0.010
Drug use before sex1.5225.3234.579 (1.184, 16.437)0.021
Involuntary subordination0.04017.1611.041 (1.022, 1.061)<0.001
Note: CI, confidence interval.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Dong, Y.; Liu, S.; Xia, D.; Xu, C.; Yu, X.; Chen, H.; Wang, R.; Liu, Y.; Dong, J.; Hu, F.; et al. Prediction Model for the Risk of HIV Infection among MSM in China: Validation and Stability. Int. J. Environ. Res. Public Health 2022, 19, 1010. https://doi.org/10.3390/ijerph19021010

AMA Style

Dong Y, Liu S, Xia D, Xu C, Yu X, Chen H, Wang R, Liu Y, Dong J, Hu F, et al. Prediction Model for the Risk of HIV Infection among MSM in China: Validation and Stability. International Journal of Environmental Research and Public Health. 2022; 19(2):1010. https://doi.org/10.3390/ijerph19021010

Chicago/Turabian Style

Dong, Yinqiao, Shangbin Liu, Danni Xia, Chen Xu, Xiaoyue Yu, Hui Chen, Rongxi Wang, Yujie Liu, Jingwen Dong, Fan Hu, and et al. 2022. "Prediction Model for the Risk of HIV Infection among MSM in China: Validation and Stability" International Journal of Environmental Research and Public Health 19, no. 2: 1010. https://doi.org/10.3390/ijerph19021010

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop