Next Article in Journal
Evaluating the Clinical Reasoning of Student Health Professionals in Placement and Simulation Settings: A Systematic Review
Previous Article in Journal
Do Attitudes towards Work or Work Motivation Affect Productivity Loss among Academic Employees?
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 19
Issue 2
10.3390/ijerph19020935
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

A New Story on the Multidimensionality of the MSPSS: Validity of the Internal Structure through Bifactor ESEM

by
César Merino-Soto
1,
Alicia Boluarte Carbajal
2,
Filiberto Toledano-Toledano
3,4,*,
Laura A. Nabors
5 and
Miguel Ángel Núñez-Benítez
6
1
Instituto de Investigacion de Psicologia, Universidad de San Martin de Porres, Lima 34, Peru
2
Facultad de Ciencias de la Salud, Universidad Cesar Vallejo—Lima Norte, Lima 39, Peru
3
Unidad de Investigacion en Medicina Basada en Evidencias, Hospital Infantil de Mexico Federico Gomez Instituto Nacional de Salud, Dr. Marquez 162, Col. Doctores, Cuauhtemoc, Mexico City 06720, Mexico
4
Unidad de Investigacion Sociomedica, Instituto Nacional de Rehabilitacion Luis Guillermo Ibarra Ibarra, Calzada Mexico-Xochimilco 289, Arenal de Guadalupe, Tlalpan, Mexico City 14389, Mexico
5
School of Human Services, College of Education, Criminal Justice and Human Services, University of Cincinnati, Cincinnati, OH 45221-0068, USA
6
Unidad de Medicina Familiar 31, Ermita Iztapalapa 1771, 8va Amp San Miguel, Iztapalapa, Mexico City 09837, Mexico
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2022, 19(2), 935; https://doi.org/10.3390/ijerph19020935
Submission received: 29 October 2021 / Revised: 13 December 2021 / Accepted: 23 December 2021 / Published: 14 January 2022
Browse Figure
Versions Notes

Abstract

:
The internal structure of the Multidimensional Scale of Perceived Social Support (MSPSS) in adolescents has been evaluated with some factorial analysis methodologies but not with bifactor exploratory structural equation modeling (ESEM), and possibly the inconsistency in the internal structure was dependent on these approaches. The objective of the study was to update evidence regarding its internal structure of MSPSS, by means of a detailed examination of its multidimensionality The participants were 460 adolescents from an educational institution in the Callao region, Lima, Peru. The structure was modeled using unidimensional, three-factor and bifactor models with confirmatory factor analysis (CFA) and ESEM approaches. The models showed good levels of fit, with the exception of the unidimensional model; however, the multidimensionality indicators supported the superiority of the bifactor ESEM. In contrast, the general factor was not strong enough, and the interfactorial correlations were substantially lower. It is concluded that the MSPSS can be interpreted by independent but moderately correlated factors, and there is possible systematic variance that potentially prevented the identification of a general factor.

1. Introduction

Adolescence is an intermediate stage between childhood and adulthood. Adolescence begins at the age of puberty, but it does not have an end age; it is fundamentally defined by psychological characteristics, and its duration is determined by the socioeconomic and cultural context in which the person lives [1,2]. Adolescence represents 18% of the total population in Latin America and the Caribbean [3] and 16% worldwide [4]. It is not only an evolutionary stage of great challenges but also a stage of high vulnerability and psychosocial risk [5], which can be evidenced by the presence of risky behaviors such as alcohol consumption in young people aged 15–19 years (27%), with an onset age of less than 15 years [6]. Likewise, data on drug use show a higher incidence in young people aged 18 to 24 than in adults [7]. The adolescent pregnancy rate worldwide is 46 births per 1000 girls [8], suicide was the second leading cause of death in individuals aged between 15 and 29 years in 2016 [9] and depression continues to be a problem in this population [10].
Adolescent development is determined by individual and socioenvironmental factors, and, from the point of view of the ecological model [11], it is not only family and friends who exert a shaping influence [12] but also contexts such as school and community. The construct of social support in adolescence has been studied from different perspectives and approaches and is predominantly considered a protective factor [5] that acts to prevent psychological disorders, such as depression, and their serious repercussions on adolescent health [13]. Social support has been studied as a predictive factor of social skills [14] and well-being [15] and a mediating variable of emotional intelligence [16,17], and it contributes to reaffirming feelings of belonging and personal worth in adolescents [18].
Although there is no unique definition of social support, there is a certain level of agreement that it involves interactions between two or more people through the provision of affection, care, material help, services, advice, and other useful information for the well-being of the person [18,19,20]. Classically, social support has been divided into two dimensions: one is related to the number of support networks that are established, and the other is the subjective characteristics of support perception, which can be affective, instrumental, and informative [21]. Recently, it has become necessary to deepen the analysis of social support to understand the problems of adolescents immersed in juvenile delinquency, drug use culture, school failure, and suicide, especially in this stage where peer groups often determine the positive or negative behavior of the adolescent [22].
Social support, a multidimensional and controversial construct at the same time, has been measured with a variety of instruments, such as the social support survey (MOS, 21 items; [23]), and applied in populations with different health conditions and in adolescents. The Family and Friends Social Support Scale (AFA-R, 14 items; [24]) is specifically aimed at adolescents and young people. Additionally, the Children and Adolescent Social Support Scale (CASSS, 40 items; [25]) assesses perceived social support from four sources: parents, teachers, classmates, and friends. Furthermore, the Social Support Networks Scale (SSNS) was developed for family caregivers of children with cancer [26]. Finally, the Multidimensional Scale of Perceived Social Support (MSPSS; [27]) was the target measure for the present study.
The MSPSS has demonstrated versatility because it has been applied in different contexts and age groups. It provides information regarding the subjective evaluation of social support obtained from sources that have been grouped into three main subscales: family, friends, and significant others [27]. Methodological studies carried out with the scale in the adolescent population have confirmed this three-dimensional factorial structure; however, the literature shows differences in the significant others’ dimension [21,28,29,30], and this dimension has shown a high correlation with the friends’ dimension, which makes it difficult for adolescents to discriminate between the two dimensions. The MSPSS has also shown a relationship with other study variables; for example, Edwards [31] found a correlation (r = 0.53, p < 0.05) between the family dimension with family support from the Familism Scale [32] and between the satisfaction dimension with the Multidimensional Students Life Satisfaction Scale (MSLSS; [33]; r = 0.79, p < 0.05). Similarly, Canty-Mitchell [34] compared the Adolescent Family Caring Scale (AFCS; r = 0.76, p < 0.05) to the other dimensions of the MSPSS. Another study reported that MSPSS scores had a positive correlation with resilience and negative correlations with depression, levels of exposure to violence, and childhood trauma [35]. Additionally, Navarro-Loli et al. [36] demonstrated through a linear regression analysis the predictive power of the MSPSS for depressive symptoms in adolescents. Ramaswamy et al. [37] demonstrated concurrent validity, finding a positive correlation with the assistance-seeking dimension of the Coping Scale for Children and Youth (CSCY; [38]) and a negative correlation with the internalizing behavior scale of the Youth Self-Report (YSR; [39]) and the Adolescent Daily Hassles Scale (ADHS; [40]) and concluding that more social support means fewer complaints from the school. This literature essentially points out the theoretical and convergent coherence of MSPSS scores with other variables. However, the literature on the internal structure of the MSPSS and how the internal structure influences the use of MSPSS scores appears to be less consistent.
For the purpose of identifying the pattern of results and methodological procedures used in the study of the internal structure of the MSPSS, a rapid systematic review was carried out [41,42] using systematic review methods. The following keywords were used, and only psychometric studies conducted with adolescents were included: MSPSS, social support, adolescence, validity, and reliability. The review was performed through search engines in the ScienceDirect, Scopus, PubMed, and Google Scholar databases. Because the objective of this study was focused on adolescents and the internal structure of MSPSS, the exclusion criteria were a youth-adult sample and nonpsychometric studies (e.g., [43]). Table 1 shows the results of the search.
Based on Table 1, the predominant model that was retained included three correlated factors; with the exception of the study by Trejos-Herrera et al. [46] that conducted a priori testing of this model. However, in an analytical context, confirmatory factor analysis (CFA) provides the advantage of directly testing and contrasting models; for example, it is possible to conduct a priori testing of a unidimensional, multidimensional and/or multidimensional bifactor model in the same study. These models are usually considered in the evaluation of the multidimensionality of a measure [48,49,50]. MSPSS total and subscale scores are typically reported, and correlations between these scores vary between moderate to high levels; these correlations are generally obtained using the maximum likelihood (ML) estimator, which is an analytical procedure that is not robust with nonnormal data [51,52].
On the other hand, the predominant factorial design was of a unique group, in which the internal structure parameters were estimated in a single sample without adding internal replicability options. The replicability of results in quantitative methodologies is important, however, only some studies implemented measurement invariance or sample partitioning. The sequential application of exploratory factor analysis (EFA) or CFA in the same sample cannot be considered a replicability test because what is obtained is a change in the estimation method or fit estimators that the EFA does not have. Some inconsistencies in these reports were also found, such as omitting the interfactoria correlations or identifying their source (observed or factorial scores), the estimation method, and the type of correlational matrix used (although not reported in Table 1, these can be polychoric or Pearson correlations). Regarding the estimation method, the principal component analysis (PCA) and varimax rotation have not yet disappeared in the psychometric methodology applied to the MSPSS. Finally, the internal consistency or reliability has been predominantly estimated by α coefficients, and the derived information may be underestimated due to limitations that were rarely recognized in the studies reviewed (the literature on this is extensive; for an example see [53]). Two additional problems that stand out were the coincidence with one of the observations of the MSPSS systematic review by Dambi et al. [43], that it was rare that a study contrasted several measurement models (e.g., one-, two-, or three-factor models), and that the relationship with social desirability was not explored (see Table 1).
An additional implication of this literature review is that the MSPSS is an instrument with numerous studies for evidence of internal structure and other sources of validity, and differences in internal structure were found. These differences in internal structure may be due to the method of assessing dimensionality, sample idiosyncrasies, cross-cultural differences, or biases not explicitly assessed (careless response). But a major source of these differences, keeping things equal, is the dimensionality assessment methodology. With the advancement in this methodology over the years, and in the context that to date the MSPSS dimensionality research has not been updated, the present study focuses on this point, that is, to introduce ESEM modeling into MSPSS dimensionality research.
Table 1 also shows that the MSPSS has not yet been assessed with other structure analysis models, for example, exploratory structural equation modeling (ESEM; [54]) and bifactor ESEM [55]. Both are advanced models that were introduced to overcome the problems of CFA modeling and to better evaluate the structural properties of multidimensional measurements. ESEM contains exploratory and confirmatory approaches, and the characteristic procedure consists of estimating the crossed or divergent factor loads with the other factors analyzed, not only the convergent factor (which is hypothesized as the causal influence of the items). This estimation has been shown to influence the decrease in factor loadings and interfactor correlations [54,56]. In this way, the factorial solutions obtained by the ESEM approach are considered realistic [54] because they estimate the parameters that by design are not estimated in the CFA (factorial loads in the non-hypothesized factors). On the other hand, the bifactor has been rediscovered as an effective method to test the multidimensionality of a measure [48,49,50,57,58]. It basically consists of modeling a general factor common to all items, together with specific factors that represent the unique content of the items [54]. Integrating both approaches into a single model (i.e., bifactor ESEM) has not yet been used in the structural analysis of the MSPSS, although this approach could provide new information on the internal structure and modify the interpretation of the scores previously evaluated by CFA models.
Therefore, an attempt is made to advance the research on the internal structure by implementing the ESEM approach. This analytical alternative could yield greater precision in realistic conditions of factorial complexity in multidimensional instruments [55]. Because the MSPSS tends to show moderate or high correlations between its dimensions (see Table 1), the bifactor ESEM model can distinguish the sources of systematic variance by estimating the relationship of the items with all the dimensions and not only with its hypothesized dimension; that is, this approach results in estimates of cross loads that are not usually estimated in CFA models [55], including a general factor that can represent the systematic common variance in the MSPSS. Thus, this study aligns with the importance of examining instruments in new contexts and analysis conditions, as provided by guidelines from influential documents that guide the practice of test adaptation [59].

2. Materials and Methods

2.1. Participants

The study sample came from the Callao region, one of the areas in Lima (Perú) with high rates of violence, citizen insecurity, drug use, pollution, and poverty [60]. Due to availability and internal organizational ease, a public educational institution was chosen. This institution enrolls males and females at the high-school level, between 12 and 16 years old [61] and mostly from families of low socioeconomic status. Its administrative organization is the same as other Peruvian public institutions. The total population was 1042 enrolled students. The units sampled were classrooms that were randomly selected by simple sampling. Each classroom contained approximately 20 to 35 students, with 15 classrooms being selected. The total sample drawn from this selection was 510 adolescents between the ages of 11 and 18 years. Several exclusion criteria were taken into account: students identified with a disability condition, as defined by the “Service of Support and Advice for the Attention of Special Educational Needs” (SAANE; RM No 665, 2018), foreigners, and those who did not agree to participate voluntarily. After applying the exclusion criteria on irrelevant response patterns and multivariate extreme values (see below), the effective sample was 461 participants.

2.2. Instrument

The MSPSS [27] measures the perceived support that the person can receive from family, friends, and significant others. It is composed of 12 items, grouped into three dimensions: family (FAM; items: 3, 4, 8 and 11), friends (FRI; items: 6, 7, 9 and 12), and significant others (SO; items: 1, 2, 5 and 10). Response options are on an ordinal scale (1 = most of the time, 2 = sometimes, 3 = often, 4 = always or almost always). The approximate duration to complete the questionnaire is 10 min. All the items have a positive orientation, and higher scores indicate greater social support. This study used the Spanish adaptation made by Arechavala and Miranda [62], who varied the response scaling from the original version to facilitate the adolescents’ level of understanding and avoid biases in the answer trends. This version was compared with the content of version validated in a previous Peruvian study. [36], which was based on the version provided by the author of the instrument; for consistency check, it was also compared with other recent Latin American versions [46].

2.3. Ethical Considerations

This study is a part of the research project (HIM/2015/017/SSA.1207; “Effects of mindfulness training on psychological distress and quality of life of the family caregiver”) that was approved on 16 December 2014, by the Research, Ethics, and Biosafety Commissions of the Hospital Infantil de México Federico Gómez, National Institute of Health, in Mexico City. While conducting this study, the ethical rules and considerations for research with humans currently enforced in Mexico [63] and those outlined by the American Psychological Association [64] were followed. All family caregivers were informed of the objectives and scope of the research and their rights in accordance with the Declaration of Helsinki [65]. The caregivers who agreed to participate in the study signed an informed consent letter. Participation in this study was voluntary and did not involve payment. The caregivers who provided consent for their child to participate completed an informed consent letter. Youths provided assent and returned a survey if they wished to participate.

2.4. Procedure

2.4.1. Data Collection

Coordination was carried out with the principal’s office of the educational institution to obtain authorization to access the study sample, and the data were collected between June and August 2019. The informed consent form explaining the objective of the study had been previously delivered to parents through the tutors in the classrooms and requested the voluntary participation of their child. Adolescents whose parents agreed to participate filled out the informed consent form. No identifiable information on the participants was recorded. The scale was administered during the tutoring hour, and at the end of data collection, they were offered a talk on family functionality; completion of the process lasted approximately seven weeks. The entire procedure is aligned with the Declaration of Helsinki regarding anonymity, protection of responses and freedom of participation in the study.

2.4.2. Data Analysis

First, before examining the internal structure, outliers and irrelevant response patterns were identified. As a general measure of abnormal responses [66], the D2 distance [67] was used for each subject based on the quadratic distance of the multivariate centroid of the variables (i.e., the MSPSS items). Based on the metrics of the χ2 distribution (degrees of freedom equal to the number of items), the Bonferroni correction [68] was applied at a 0.05 confidence level to establish the cutoff point for detection (χ2 > 6.14). The detection of response patterns associated with insufficient effort was done by estimating the sequence of the longest string of identical responses given by a person (longstr), which, in this context of multidimensional measurement, suggests responses without variability [66]. For this purpose, the careless R program was used [69]. Second, a descriptive analysis of the items was performed, which included their distributional characteristics (i.e., distributional normality) and differences in response trends. For each subscale, these were examined using nonparametric analyses that included a measure of the magnitude of the difference [70] using the langtest [71] and MVN [72] R programs.
Third, the internal structure was evaluated through CFA-SEM to evaluate the MSPSS measurement model. First, the model established by the author Zimet et al. [27] and most subsequent studies, which consisted of three related dimensions (3F), were tested. The second model represented the use of the total MSPSS score, that is, it was defined by a single latent dimension (i.e., unidimensionality). The third model included bifactor modeling, which allows the identification of the general latent dimension (i.e., a general factor) and specific dimensions (i.e., specific factors). Because the literature generally finds moderate or strong associations between the dimensions of the MSPSS, ESEM modeling was applied in the model with three related factors (model 3F) and in the bifactor model. Both approaches (CFA and ESEM) can be applied in a complementary way to assess the multidimensionality of instruments. Oblique geomin rotation [56] was applied in the 3F-ESEM and bifactor-ESEM models. Due to its efficacy [51,52], the WLSMV estimator was used in all SEM modeling [73] with interitem polychoric correlations. Figure 1 shows the representation of each model tested.
In general, the fit of each of the models was evaluated using metrics such as CFI (≥0.95), RMSEA (≤0.05), WRMR (≤0.90; [74]) and gammaHat (≥0.95). For bifactor modeling (bifactor CFA and bifactor ESEM), several indicators suggested in the literature were used [49,50,58,75,76]: (a) the explained common variance (ECV) retained in the general factor Fg (ECVg), in the items of the Fg (I-ECV) and in the specific factors after removing the general variance (ECVf) had to exceed 0.70 or 0.80; (b) the reliable variance in the observed score (ωh) and in the specific factors after removing the variance in Fg from the observed score (ωhs) had a cutoff point of ≥0.80; and (c) the degree of replicability of the construct in each factor (H ≥ 0.80) and the absolute relative parameter bias (ARPB), which are in the 10% to 15% range for the difference between the factorial loads estimated in the general factor of a unidimensional model vs. a bifactor model, are acceptable [49,50]. Finally, the determinability of the factors (FD) was also estimated (FD > 0.90; [77]).
Furthermore, due to the predominance of the ECV in establishing the general bifactor dimension [78], ECV values higher than 0.70 suggest that there is an argument based on the common variance for the conclusion of an important unidimensional model [49,50]. An ECV below 0.70 suggests that the multidimensionality coming from the specific factors is nontrivial. The potential misspecifications were evaluated for each model using an approach that combines the statistical power and the size of the modification [79]. SEM modeling was carried out by the lavaan [80] and semtools [81] R programs.
Finally, based on Table 1, the interfactorial correlations were meta-analyzed using the random effects model, an appropriate method when the effect of the sampling error from different populations is presumed [82]. The Hartung–Knapp–Sidik–Jonkman estimator was used, since it can produce robust results with a small number of studies that have a high degree of heterogeneity [82,83].

3. Results

3.1. Response Bias Analysis

The relationship between the identification of D2 and longsting was −0.046 (p > 0.10), indicating that both measures detected different response patterns. Fourteen subjects (2.7%) with D2 > 41.78 exceeded the cutoff point (D2 ≥ 41.004) and were detected as multivariate outliers. Inspection of these responses revealed an inconsistent pattern of response. On the other hand, 35 subjects (6.8%) were identified with the same answers in all 12 items of the MSPSS, in categories 1 (1, 2.8%), 2 (1, 2.8%) and 3 (33, 95.2%). Altogether, 49 sample subjects (9.5%) were removed.

3.2. Descriptive and Correlational Analysis

3.2.1. Item Analysis

The responses to the items (see Table 2) showed apparent similarities in the direction of their distributional properties (skewness and kurtosis with the same sign) but also showed apparent differences in magnitude. The global difference in the responses on the SO scale was trivial (Friedman-χ2 = 2.3247, gl = 3, p = 0.507; r = 0.031, 95% CI = −0.061, 0.122), while the differences on the FAM scale (Friedman-χ2 = 265.53, gl = 3, p < 0.001) and FRI scale (Friedman-χ2 = 38.83, gl = 3, p < 0.001) were statistically significant but small in size (r = 0.191, 95% CI: 0.101, 0.277; and r = 0.249, 95% CI = 0.166, 0.329, respectively). Furthermore, the responses to the items were not distributed with statistical normality (see Cramer–von Mises test statistics for each item), and multivariate normality, based on the Henze and Zirkler [84] test, was not fulfilled in the set of items, z = 1.77942 (p < 0.001).

3.2.2. Meta-Analytic Interfactor Correlations

Using the CFA studies in Table 1 (studies 3, 4, 5, 8, 9 and 10), the meta-analytic interfactorial correlation between FRI and SO was 0.56 (p < 0.01, 95% CI = 0.38, 0.73), the interfactorial correlation between FRI and FAM was 0.39 (p < 0.01, 95% CI = 0.29, 0.48), and the interfactorial correlation between SO and FAM was 0.51 (p < 0.01, 95% CI = 0.54, 0.70). The estimates of variability for the FRI and SO relationship (Q = 625.69, p < 0.01, τ = 0.215, I2 = 98.92%, H2 = 95.53), FRI and FAM relationship (Q = 85.39, p <.01, τ = 0.115, I2 = 93.42%, H2 = 15.20), and FAM and SO relationship (Q = 204.713, p < 0.01, τ = 0.146, I2 = 96.91%, H2 = 32.37) indicated heterogeneity in the correlations that integrated the meta-analytic average. These meta-analytic interfactor correlations were used for comparison with the interfactor correlations in our sample (see Section 3.3.3).

3.3. Internal Structure

3.3.1. Model Fits

The general results in Table 3 indicated that the unidimensional model was of comparatively lesser statistical fit. Additionally, with both approaches (CFA and ESEM), the correlated three-factor model (congeneric model) fitted very well, but the bifactor model quantitatively outperformed it. The difference between the two approaches does not seem statistically substantial because both expressed a very good fit. The performance of the ESEM model was definitely superior to that of the CFA model, indicating that the estimation of the cross loads, as usually occurs in the ESEM, did have an impact on the amount of fit obtained.

3.3.2. CFA Modeling

Inspection of the parameters of interest in the CFA models (Table 4), with the exception of the bifactor-CFA model, revealed that the factor loadings were high in all models (λ > 0.70). In contrast, in the unidimensional model, although its fit was poor (see previous paragraph), its factor loadings were high (>0.66). In the 3-factor model, the interfactorial correlations were high (>0.60), suggesting insufficient discrimination between them. The bifactor-CFA model did not initially converge, and adjustment indicators could not be obtained. The apparent problem was item two of the SO factor, which showed excessive negative variance (=−678.20), and this factor load was identified as a Heywood case in its specific factor (λ = 26.04). Its load in the general factor (Fg-cfa) and the other items of its factor showed loads λ > 0.80. Because the SO factor presented similar factorial loads (tau-equivalence; see next paragraph), its loads were constrained to equality between them to control the negative variance.
As a complementary analysis for each factor, their factor loadings were constrained to equality (tau-equivalent model), while the remaining factors were freely estimated (congeneric models). Compared with the model that had free estimation of all factorial loads (three correlated factors), the results showed a good fit for the SO (WLSMV-χ2 = 70.802, p > 0.05, gl = 54, CFI = 1.00, ΔCFI = 0.000; WRMR = 0.788, ΔWRMR = −0.019; gammaHat = 0.993, Δgamma = 0.001) and FRI factors (WLSMV-χ2 = 83.593, p < 0.05, gl = 54, CFI = 0.999, ΔCFI = 0.001; WRMR = 0.856, ΔWRMR = −0.087; gammaHat = 0.989, Δgamma = 0.005). The tau-equivalent fit was acceptable for the FAM factor (WLSMV-χ2 = 199.502, p < 0.01, gl = 54, CFI = 0.997, ΔCFI = 0.003; WRMR = 1.323, ΔWRMR = −0.554; gammaHat = 0.950; Δgamma = 0.044). The estimated values of the fixed factorial load in the SO factor (λ = 0.843), FAM factor (λ = 0.853) and FRI factor (λ = 0.901) were very high.
In the analysis of the multidimensionality of the MSPSS with the bifactor CFA, the ECV indicators, ωh and H, showed the following: (a) the ECVg indicator of the general factor Fg retained a moderately high variance (ECVg = 0.751) and was highly differentiated compared with the specific factors (ECVf around 0.35), and at the item level (I-ECV), the general common variance was moderate in the FAM and FRI factor items, and high for the SO factor items; (b) the items were shown to be predominantly within the acceptable range of multidimensional bias (between 0.10 and 0.15; [50]); (c) the reliable variance (ωh) was high for Fg (>0.80) and low for the specific factors (<0.40); (d) the coefficient (H) indicated that the replicability was stronger for the general factor (H > 0.90) than for the specific factors (H < 0.70); (e) the determinability of the general factor Fg exceeded the minimum criterion (FD > 0.90) compared with the specific factors (FD < 0.90); and (f) ARPB occurred in a range that suggested that the general dimension Fg does not produce significant biases regarding the multidimensionality of the items (ARPB total = 0.121). Taken together, the CFA modeling showed that the bifactor model represents a dimensional solution where a general factor Fg is predominant and preferable over the consideration of specific factors as dimensions to interpret. The interfactor correlations could not be estimated due to the orthogonality constraint to run the CFA bifactor.

3.3.3. ESEM Modeling

Table 5 shows the results of the ESEM modeling. In the 3F-ESEM model, items with factorial complexity were found (divergent or crossed loads, λ ≥ 0.10; items 3, 6, 8, 10, 11), and these were contrasted with the CFA specification on this type of factor loading (hypothesized factor loads equal to zero). Compared with the 3F-CFA model (Table 4), the interfactor correlations were comparatively low. Comparing the meta-analytic correlations with the correlations in 3F-ESEM, the differences were statistically significant and comparatively substantial in FAM-SO (z = 9.63, p = 0.01, q = 0.45), SO-FRI (z = 7.18, p = 0.01, q = 0.33) and FAM-FRI (z = 6.05, p = 0.01, q = 0.28).
The bifactor ESEM model produced very different interfactor correlations than the 3F-ESEM, because they decreased between 25.2% and 65.2%. Compared with the meta-analytic correlations, statistically significant but small differences were found in FAM-SO (z = 1.94, p = 0.02, q = 0.09), SO-FRI (z = −4.88, p < 0.01, q = 0.22) and FAM-FRI (z = −4.27, p < 0.01, q = 0.19). However, factor loadings decreased in different proportions, sometimes by a moderately high amount, within the same factor: SO between 2.4% and 16.1%, FAM between 2.0% and 8.3%, and FRI between 4.7% and 14.8%. These decreases were more accentuated with the implementation of the bifactor ESEM.
In the multidimensional analysis, the bifactor ESEM produced values of the ECV, ωh, H and ARPB with the following characteristics: (a) the ECVg indicator of the general factor Fg was low compared to the recommended criterion, and that of the subscales was even lower (ECVf between 0.10 and 0.24), while at the item level (I-ECV), the general common variance was high only for FRI and low for FAM and SO; (b) the items showed a predominant multidimensional bias (ARPBitem > 0.15) in SO and FAM; (c) the reliable variance (ωh) for Fg was acceptable, and low for the specific factors (<0.10); (d) the coefficient (H) indicated that replicability was stronger with the general factor (H > 0.90) than with the specific factors; (e) the determinability (FD) of the general factor Fg and FAM exceeded the minimum criterion (FD > 0.90), however, it was moderately low for the other factors (SO and FRI); and (f) the degree of bias (ARPB total = 0.238) occurred in a range that suggested nontrivial differentiation between the factorial loads produced from a general dimension Fg and the multidimensional model. Altogether, ARPB (at the total and item level) and ECV for the general factor (ECVtotal) and the items (I-ECV) of the bifactor ESEM model showed that the general factor was not strong enough.

4. Discussion

In the evolution of the MSPSS, a study introduced CFA to evaluate its internal structure [44] and achieved an advance in the direct verification of its dimensional models (e.g., first-order and second-order factors) and of the equivalence of these. In this study, bifactor ESEM model was introduced for the first time to examine the internal structure of the MSPSS, and the results indicated a significant contrast with all previous studies that applied CFA.
Regarding factor loadings, they decreased across a mixed range of changes, indicating that the validity of each item was differentially affected by the model used. The greatest decrease occurred with the bifactor ESEM, which is aligned with the frequent results found with the ESEM modeling in general and with the bifactor ESEM in particular [55,56,57]. Another result to be emphasized is that CFA concealed some misspecifications associated with nontrivial cross-factorial loads. As is known in the methodological literature [55], nonrealistic restrictions are imposed with CFA, and this is the main reason why ESEM models tend to show a better statistical fit than CFA models.
The ωh obtained for the general factor (0.795) indicated that there is enough reliable variance in the total score to be interpreted, and H suggested that the greater replicability and better definition obtained in the estimation of a general factor compared with the specific factors was beyond doubt. On the other hand, H suggested that the definition of the factor from its items is poor for FRI and moderate for SO and FAM. Both the potential replicability and the high definition of the constructs are derived from the interpretation of H [75]. Finally, based on the ECV for the factors, the individual differences can be captured by the general factor and FAM, and moderately by SO. With an explained variance in ECV = 0.795, 79.5% represents the common variability source, and this amount cannot be considered trivial. Therefore, in conjunction with the determinability of the factorial solution, reliable variance, and replicability (FD coefficients, ωh, and H, respectively), it is tempting to accept the interpretation of a general factor even in the context of the ECV being below the recommended cutoff point (<0.70).
However, the complementary information at the item level (I-ECV and I-ARPB) suggested that the strength of the potential general factor did not guarantee accepting it as representative of a global construct of social support. Because the ECV is one of the key pieces to accept a bifactor model [78], its results at the item and general factor levels contrast with the rest of the general indices, it can be affirmed that there is a partial confirmation to accept the general factor; in addition, the parameters at the item level (i.e., I-ECV and I-ARPB) predominantly indicated a difference between multidimensionality and unidimensionality that cannot be ignored. One of the risks of accepting unidimensionality when I-ECV is less than 0.85 is the violation of local independence [85], and this is possibly more certain when half of the I-ECV values met the recommended cutoff point (>0.80).
The results of the bifactor CFA contrasted with the bifactor ESEM in the estimation of interfactorial correlations and in the conclusion about multidimensionality. In the bifactor CFA, the conclusion of unidimensionality seemed reasonable; however, this was shown to be overestimated when the structure of the MSPSS was evaluated by the bifactor ESEM. It was in this analysis that the general factor was not strong enough, and contradictory indications of multidimensionality were obtained. On the other hand, the interfactorial correlations reported by the bifactor ESEM were more differentiated among themselves compared with the rest of the factorial results, especially with the bifactor CFA. This differentiation adds an additional advantage in relation to the precision of these correlations because they realistically add the estimation of all factorial loadings of all the factors analyzed, which is characteristic of ESEM [54].
As observed, the interfactorial correlations estimated in the bifactor ESEM model can be interpreted as a structure with moderate or low conceptual dependence, and this has acceptability for the construction of instruments such as the MSPSS when measures that do not conceptually overlap are required. A large dependence between factors usually indicates insufficient conceptual discrimination between them [76,85], and to this same extent, the interpretation is factorially complex. Instead, the amount of dependence found in the bifactor ESEM may be more satisfactory and in line with what is found in other studies with different instruments (e.g., AFA-R: [24]; CASSS: [25]; MOS: [23]). In these studies, the support sources do not conclude that there is a strong degree of dependence between them, which would lead to potential problems of divergent validity in the instrument. Although it is plausible that social support sources covary in a subject, it is more reasonable to rely on the idea that they act independently unless the context of the person links them. Also, the multidimensionality of adolescent responses to MSPSS indicated that there is a need to examine perceptions of support in different areas, such as support of friends and significant others. Assessment of adolescents’ perceptions of different types of support may be critical to the design of prevention messaging and interventions to improve their functioning.
It is possible that the specific undetected variance from undetected processes is involved in the variance in the general factor, inflating the parameters of interest (e.g., factor loadings). If this is so, this general factor can represent the joint effect of careless responses, composed of average responses and social desirability [86,87], method variance or the interaction between these. As has been detected in other studies [66,86,87,88], this type of variance is omnipresent in measurements based on self-report and may be incorporated in the associated systematic variance in the content of the MSPSS.
The essential limitations of this study correspond to examining the invariance of the best fitted models (3F-ESEM and bifactor-ESEM models), as well as introducing methods that model the possible effect of irrelevant response styles. It is also important to point out sample size as an issue, because a larger size may be required to guarantee more stable results and a higher density in the groups compared for invariance measurements. However, our sample size is considered adequate, given the high trend in magnitude of the factor loadings (essentially > 0.60). Also, we did not have a measure assessing whether social desirability in the responding of the adolescents changed how items were characterized or reported. It may be beneficial to add a measure or questions to detect this in future studies. Although there is a report indicating that social desirability is not involved in MSPSS responses [89], for purposes of replicability and consistency in conclusions, this potential effect has yet to be studied. Finally, the chosen sample of participants does not ensure population representativeness, because this lack of representativeness may involve a range of experiences in social support that differ between low and other socioeconomic statuses.
The results introduced several implications for subsequent psychometric studies of the MSPSS, especially when examining its internal structure. One of the implications of the study is methodological. The distinction between the CFA and ESEM modeling was notable in the estimation of the parameters of interest (i.e., factor loadings and interfactor correlations) and consequently modified the conclusions about the conceptual relationships inferred from them. Both types of modeling provided two different figures to identify the best structural model that represented the structure of the MSPSS. In an analytical or methodological context, it is assumed that the implementation of both in a study can be considered canonical rather than complementary. Given the advance that ESEM modeling provides in understanding the internal structure of the MSPSS, it might be preferable to first choose the ESEM method to reveal the sources of variance in the items with better precision. Because it is common, reasonable, and realistic to find multidimensional variance in psychosocial measurement items [86], the ESEM effectively models this variance without imposing the known restrictions of the CFA.
Another implication is the requirement to consider modeling sources of systematic variance associated with irrelevant patterns or response styles that are particularly present in self-report measures (e.g., [66,87]), which are considered responses that are not necessarily related to the substantive content of the items. Some of these were detected in the study (i.e., extreme multivariate responses and long response sequences), representing a prevalence within the range reported in the literature (between 3.5% and 12%; [66]). However, an approach based on modeling can detect the variability in this phenomenon and look for other patterns, such as average responses and social desirability [88,89].

5. Conclusions

The present study highlights the effects of evaluation methods of the internal structure of the MSPSS carried out in previous studies and presents a more appropriate analysis of the multidimensionality or unidimensionality of the MSPSS in Peruvian adolescents. The MSPSS can be treated as a multidimensional measure, with its moderately associated dimensions in adolescents. However, the existence of a general dimension that is interpretable and expressed as a general score is not sufficiently justified because the general dimension was not psychometrically strong. These results, however, varied significantly as a consequence of the method of analysis of the internal structure used: with CFA modeling, a general dimension was recognized, but with ESEM modeling, the overall dimension was not strong enough. Additionally, with the CFA, there was a tendency to overestimate factorial loads and interfactor correlations. These overestimated interfactor correlations indicated a potential overall factor, with high strength to be treated as the single and representative MSPSS score. In contrast, ESEM modeling discovered moderate associations between the subscales and thus weakness of an overall factor. This finding is important, and points to the value of understanding adolescent perceptions of different types of support from family, significant others, and friends. Understanding the value and direction of supports needed from these different sources can be of value in designing interventions to improve social support of youth. Due to the limitations of the sample size in some interest groups, the equivalence of these results in the groups based on sex, age, and other groups requires investigation, as does a determination of the relationship of the MSPSS scores with external variables.

Author Contributions

Conceptualization, C.M.-S. and A.B.C.; methodology, C.M.-S., M.Á.N.-B., F.T.-T. and L.A.N.; software, C.M.-S. and A.B.C.; validation, L.A.N. and A.B.C.; formal analysis, C.M.-S. and A.B.C.; investigation, F.T.-T., M.Á.N.-B. and L.A.N.; resources, F.T.-T., M.Á.N.-B. and L.A.N.; data curation, C.M.-S. and A.B.C.; writing-original draft preparation, C.M.-S. and F.T.-T.; writing-review and editing, F.T.-T.; visualization, C.M.-S., M.Á.N.-B. and L.A.N.; supervision, C.M.-S. and L.A.N.; project administration, F.T.-T. and C.M.-S.; funding acquisition, F.T.-T. All authors have read and agreed to the published version of the manuscript.

Funding

This work is one of the results of the research project HIM/2015/017/SSA.1207, “Effects of mindfulness training on psychological distress and quality of life of the family caregiver. Main researcher: Filiberto Toledano-Toledano Ph. D. The present research was funded by federal funds for health research and was approved by the Commissions of Research, Ethics and Biosafety (Comisiones de Investigación, Ética y Biosegurida), Hospital Infantil de México Federico Gómez National Institute of Health. The source of federal funds did not control the study design, data collection, analysis, or interpretation or the decisions regarding publication.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Commissions of Research, Ethics and Biosafety (Comisiones de Investigación, Ética y Bioseguridad), Hospital Infantil de México Federico Gómez National Institute of Health.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Acknowledgments

The authors thank Franklin Eduardo Perea Flores for his support.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Sawyer, S.M.; Azzopardi, P.S.; Wickremarathne, D.; Patton, G.C. The age of adolescence. Lancet Child Adolesc. Health 2018, 2, 223–228. [Google Scholar] [CrossRef]
  2. Gaete, V. Desarrollo psicosocial del adolescente. Rev. Chil. Pediatr. 2015, 86, 436–443. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  3. United Nations Children’s Fund. An Approach to the Situation of Adolescents and Young People in Latin America and the Caribbean. 2015. Available online: https://www.unicef.org/lac/sites/unicef.org.lac/files/2018-04/UNICEF_Situacion_de_Adolescentes__y_Jovenes_en_LAC_junio2105.pdf (accessed on 12 February 2021).
  4. United Nations Children’s Fund. Peace, Dignity and Equality on a Healthy Planet. 2019. Available online: https://www.un.org/es/sections/issues-depth/youth-0/index.html (accessed on 12 February 2021).
  5. Jessor, R. Contextual Content in Problem Behavior Theory. In Problem Behavior Theory and the Social Context, 1st ed.; Springer International Publishing: Cham, Switzerland, 2017; Volume 3, pp. 177–209. [Google Scholar]
  6. World Health Organization. Informe de Estado Global sobre Alcohol y Salud 2018; Organización Mundial de la Salud: Geneva, Switzerland, 2018. [Google Scholar]
  7. United Nations Office on Drugs and Crime. World Drug Report 2018 Methodology Report. Research and Trend Analysis Branch UNODC. Available online: https://www.unodc.org/wdr2018 (accessed on 15 October 2021).
  8. Pan American Health Organization; United Nations Population Fund; United Nations Children’s Fund. Accelerating Progress toward the Reduction of Adolescent Pregnancy in Latin America and the Caribbean; Report of a Technical Consultation; Pan American Health Organization: Washington, DC, USA, 2018. [Google Scholar]
  9. World Health Organization. Suicide. 2021. Available online: https://www.who.int/es/news-room/fact-sheets/detail/suicide%0A (accessed on 15 October 2021).
  10. Huberty, T. Differential diagnosis of depression and mood disorders. In Anxiety and Depression in Children and Adolescents: Assessment, Intervention, and Prevention; Huberty, T.J., Ed.; Springer: New York, NY, USA, 2012; pp. 139–166. [Google Scholar]
  11. Vaux, A. An ecological approach to understanding and facilitating social support. J. Soc. Pers. Relatsh. 1990, 7, 507–518. [Google Scholar] [CrossRef]
  12. Toledano-Toledano, F.; Moral, J.; McCubbin, L.D.; Cauley, B.; Luna, D. Brief version of the coping health inventory for parents (CHIP) among family caregivers of children with chronic diseases. Health Qual. Life Outcomes 2020, 18, 104. [Google Scholar] [CrossRef] [Green Version]
  13. Simoni, Z.R.; Bauldry, S. Moving during adolescence and depressive symptoms. Youth Soc. 2018, 52, 639–660. [Google Scholar] [CrossRef]
  14. Kerres, M.C.; Kilpatrick, D.M. Measuring perceived social support: Development of the child and adolescent social support scale (CASSS). Psychol. Sch. 2002, 39, 1–18. [Google Scholar] [CrossRef]
  15. Cho, H.; Yi Tan, H.; Lee, E. Importance of perceived teammate support as a predictor of student-athletes’ positive emotions and subjective well-being. Int. J. Sports Sci. Coach. 2020, 15, 364–374. [Google Scholar] [CrossRef]
  16. Malinauskas, R.; Malinauskiene, V. The Relationship between emotional intelligence and psychological well-being among male university students: The mediating role of perceived social support and perceived stress. Int. J. Environ. Res. Public Health 2020, 17, 1605. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  17. Ye, J.; Yeung, D.Y.; Liu, E.S.C.; Rochelle, T.L. Sequential mediating effects of provided and received social support on trait emotional intelligence and subjective happiness: A longitudinal examination in Hong Kong Chinese university students. Int. J. Psychol. 2018, 54, 478–486. [Google Scholar] [CrossRef]
  18. Durá, E.; Garcés, J. La teoría del apoyo social y sus implicaciones para el ajuste psicosocial de los enfermos oncológicos. (The theory of social support and its implications for the psychosocial adjustment of the oncological patients). Rev. Psicol. Soc. 1991, 6, 257–271. [Google Scholar] [CrossRef]
  19. Toledano-Toledano, F.; Moral, J.; Nabors, L.A.; Domínguez-Guedea, M.T.; Escudero, G.S.; Pérez, E.R.; Luna, D.; López, A.L. Predictors of quality of life among parents of children with chronic diseases: A cross-sectional study. Healthcare 2020, 8, 456. [Google Scholar] [CrossRef] [PubMed]
  20. House, J.S. Work Stress and Social Support; Addison-Wesley: Reading, MA, USA, 1981. [Google Scholar]
  21. Fachado, A.; Menéndez, M.; Gonzáles, L. Apoyo social: Mecanismos y modelos de influencia sobre la enfermedad crónica (Social support: Mechanisms and models of influence on chronic disease). Cad. Aten. Primaria 2010, 19, 118–123. [Google Scholar]
  22. Díaz, A.M.; Parra, S.M.; Aravena, V.J.; Barriga, O.A. Validez y confiabilidad de una escala de apoyo social percibido en población adolescente. (Validity and reliability of a perceived social support scale on adolescent population). Enferm. Glob. 2015, 14, 125–136. [Google Scholar] [CrossRef] [Green Version]
  23. Sherbourne, C.D.; Stewart, A.L. The MOS social support survey. Soc. Sci. Med. 1991, 32, 705–714. [Google Scholar] [CrossRef]
  24. Ramírez, M.T.G.; Hernández, R.L. Propiedades psicométricas de la escala de apoyo social familiar y de amigos (AFA-R) en una muestra de estudiantes. Acta Investig. Psicol. 2014, 4, 1469–1480. [Google Scholar] [CrossRef] [Green Version]
  25. Malecki, C.K.; Demaray, M.K.; Elliott, S.N. The Child and Adolescent Social Support Scale; Northern Illinois University: DeKalb, IL, USA, 2000. [Google Scholar]
  26. Toledano-Toledano, F.; Moral, J.; Frometa, R.R.; Betanzos, F.G.; Guzmán, L.V.; García, M.S. The social support networks scale (SSNS) for family caregivers of children with cancer: A psychometric evaluation. Int. J. Environ. Res. Public Health 2020, 17, 7820. [Google Scholar] [CrossRef] [PubMed]
  27. Zimet, G.D.; Dahlem, N.W.; Zimet, S.G.; Farley, G.K. The Multidimensional Scale of Perceived Social Support. J. Personal. Assess. 1988, 52, 30–41. [Google Scholar] [CrossRef] [Green Version]
  28. Chou, K.-L. Assessing Chinese adolescents’ social support: The Multidimensional Scale of Perceived Social Support. Personal. Individ. Differ. 2000, 28, 299–307. [Google Scholar] [CrossRef]
  29. Laksmita, O.D.; Chung, M.H.; Liao, Y.M.; Chang, P.C. Multidimensional Scale of Perceived Social Support in Indonesian adolescent disaster survivors: A psychometric evaluation. PLoS ONE 2020, 15, e0229958. [Google Scholar] [CrossRef] [Green Version]
  30. Navarro-Loli, J.S.; Merino-Soto, C.; Dominguez-Lara, S.; Lourenço, A. Estructura interna de la Multidimensional Scale of Perceived Social Support (MSPSS) en adolescentes peruanos (Internal structure of the Multidimensional Scale of Perceived Social Support (MSPSS) in Peruvian adolescents). Rev. Arg. Cs. Comp. 2019, 11, 38–47. [Google Scholar] [CrossRef]
  31. Edwards, L.M. Measuring perceived social support in Mexican American youth: Psychometric properties of the Multidimensional Scale of Perceived Social Support. Hisp. J. Behav. Sci. 2004, 26, 187–194. [Google Scholar] [CrossRef]
  32. Sabogal, F.; Marín, G.; Otero-Sabogal, R. Hispanic familisim and acculturation: What changes and what doesn´t? Hispanic J. Behav. Sci. 1987, 9, 397–412. [Google Scholar] [CrossRef]
  33. Huebner, E.S. Preliminary development and validation of a Multidimensional Life Satisfaction Scale for children. Psychol. Assess. 1994, 6, 149–158. [Google Scholar] [CrossRef]
  34. Canty-Mitchell, J.; Zimet, G.D. Psychometric properties of the Multidimensional Scale of Perceived Social Support in Chilean university students. Am. J. Community Psychol. 2000, 28, 391–400. [Google Scholar] [CrossRef]
  35. Bruwer, B.; Emsley, R.; Kidd, M.; Lochner, C.; Seedat, S. Psychometric properties of the Multidimensional Scale of Perceived Social Support in youth. Compr. Psychiatry 2008, 49, 195–201. [Google Scholar] [CrossRef]
  36. Alsubaie, M.M.; Stain, H.J.; Webster, L.A.D.; Wadman, R. The role of sources of social support on depression and quality of life for university students. Int. J. Adolesc. Youth 2019, 24, 484–496. [Google Scholar] [CrossRef] [Green Version]
  37. Ramaswamy, V.; Aroian, K.J.; Templin, T. Adaptation and psychometric evaluation of the Multidimensional Scale of Perceived Social Support for Arab American adolescents. Am. J. Community Psychol. 2009, 43, 49–56. [Google Scholar] [CrossRef]
  38. Brodzinsky, D.M.; Elias, M.J.; Steiger, C.; Simon, J.; Gill, M.; Hitt, J.C. Coping Scale for Children and Youth: Scale development and validation. J. Appl. Dev. Psychol. 1992, 13, 195–214. [Google Scholar] [CrossRef]
  39. Crijnen, A.A.; Achenbach, T.M.; Verhulst, F.C. Problems reported by parents of children in multiple cultures: The Child Behavior Checklist syndrome constructs. Am. J. Psychiatry 1999, 156, 569–574. [Google Scholar] [CrossRef]
  40. Seidman, E.; Allen, L.; Aber, J.L.; Mitchell, C.; Feinman, J.; Yoshikawa, H.; Comtois, K.A.; Golz, J.; Miller, R.L.; Ortiz-Torres, B.; et al. Development and validation of adolescent-perceived microsystem scales: Social support, daily hassles, and involvement. Am. J. Community Psychol. 1995, 23, 355–388. [Google Scholar] [CrossRef]
  41. Grant, M.J.; Booth, A. A typology of reviews: An analysis of 14 review types and associated methodologies. Health Inf. Libr. J. 2009, 26, 91–108. [Google Scholar] [CrossRef] [PubMed]
  42. Khangura, S.; Konnyu, K.; Cushman, R.; Grimshaw, J.; Moher, D. Evidence summaries: The evolution of a rapid review approach. Syst. Rev. 2012, 1, 10. [Google Scholar] [CrossRef] [Green Version]
  43. Dambi, J.M.; Corten, L.; Chiwaridzo, M.; Jack, H.; Mlambo, T.; Jelsma, J. A systematic review of the psychometric properties of the cross-cultural translations and adaptations of the Multidimensional Perceived Social Support Scale (MSPSS). Health Qual. Life Outcomes 2018, 16, 80. [Google Scholar] [CrossRef] [Green Version]
  44. Cheng, S.T.; Chan, A.C.M. The Multidimensional Scale of Perceived Social Support: Dimensionality and age and gender differences in adolescents. Personal. Individ. Differ. 2004, 37, 1359–1369. [Google Scholar] [CrossRef]
  45. Wilson, A.; Yendork, J.S.; Somhlaba, N.Z. Psychometric properties of Multidimensional Scale of Perceived Social Support among Ghanaian adolescents. Child Indic. Res. 2016, 10, 101–115. [Google Scholar] [CrossRef]
  46. Trejos-Herrera, A.M.; Bahamón, M.J.; Alarcón-vásquez, Y.; Vélez, J.I.; Vinaccia, S. Psychosocial intervention validity and reliability of the Multidimensional Scale. Psychosoc. Interv. 2018, 27, 56–63. [Google Scholar] [CrossRef] [Green Version]
  47. Aloba, O.; Opakunle, T.; Ogunrinu, O. Psychometric characteristics and measurement invariance across genders of the Multidimensional Scale of Perceived Social Support (MSPSS) among Nigerian adolescents. Health Psychol. Rep. 2019, 7, 69–80. [Google Scholar] [CrossRef]
  48. Reise, S.P. The rediscovery of bifactor measurement models. Multivar. Behav. Res. 2012, 47, 667–696. [Google Scholar] [CrossRef] [PubMed]
  49. Rodriguez, A.; Reise, S.P.; Haviland, M.G. Applying bifactor statistical indices in the evaluation of psychological measures. J. Personal. Assess. 2016, 98, 223–237. [Google Scholar] [CrossRef]
  50. Rodriguez, A.; Reise, S.P.; Haviland, M.G. Evaluating bifactor models: Calculating and interpreting statistical indices. Psychol. Methods 2016, 21, 137–150. [Google Scholar] [CrossRef]
  51. Li, C.H. Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behav. Res. Methods 2016, 48, 936–949. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  52. Li, C.H. The performance of ML, DWLS, and ULS estimation with robust corrections in structural equation models with ordinal variables. Psychol. Methods 2016, 21, 369–387. [Google Scholar] [CrossRef] [PubMed]
  53. Green, S.B.; Yang, Y. Evaluation of dimensionality in the assessment of internal consistency reliability: Coefficient alpha and omega coefficients. Educ. Meas. Issues Pract. 2015, 34, 14–20. [Google Scholar] [CrossRef]
  54. Asparouhov, T.; Muthén, B. Exploratory structural equation modeling. Struct. Equ. Model. Multidiscip. J. 2009, 16, 397–438. [Google Scholar] [CrossRef]
  55. Morin, A.J.S.; Arens, A.K.; Marsh, H.W. A Bifactor exploratory structural equation modeling framework for the identification of distinct sources of construct-relevant psychometric multidimensionality. Struct. Equ. Model. Multidiscip. J. 2015, 23, 116–139. [Google Scholar] [CrossRef]
  56. Mansolf, M.; Reise, S.P. Exploratory bifactor analysis: The Schmid-Leiman orthogonalization and Jennrich-Bentler analytic rotations. Multivar. Behav. Res. 2016, 51, 698–717. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  57. Gignac, G.E. The higher-order model imposes a proportionality constraint: That is why the bifactor model tends to fit better. Intelligence 2016, 55, 57–68. [Google Scholar] [CrossRef]
  58. Reise, S.P.; Bonifay, W.E.; Haviland, M.G. Scoring and modeling psychological measures in the presence of multidimensionality. J. Personal. Assess. 2013, 95, 129–140. [Google Scholar] [CrossRef] [PubMed]
  59. American Educational Research Association; American Psychological Association; National Council on Measurement in Education. Standards for Educational and Psychological Testing; American Educational Research Association: Washington, DC, USA, 2014. [Google Scholar]
  60. Instituto Nacional de Estadística e Informática. Situación Demográfica, Económica y Social Provincia Constitucional del Callao: Focalización de los Grupos Vulnerables. (Demographic, Economic and Social Situation Constitutional Province of Callao: Targeting of Vulnerable Groups). Available online: https://www.inei.gob.pe/media/MenuRecursivo/boletines/presentacion_callao.pdf (accessed on 11 January 2021).
  61. Ministry of Education. Ministerial Resolution N° 665: Norm that Regulates School Enrollment and Transfer in Educational Institutions and Basic Education Programs. Available online: https://www.gob.pe/institucion/minedu/normas-legales/223235-665-2018-minedu (accessed on 10 January 2021).
  62. Arechabala, M.; Miranda, C. Validación de una escala de apoyo social percibido en un grupo de adultos mayores adscritos a un programa de hipertensión de la región metropolitana (Validation of a perceived social support scale in a group of elderly affiliated to a hypertension program of the metropolitan region). Cienc. Enferm. 2002, 8, 49–55. [Google Scholar] [CrossRef]
  63. Sociedad Mexicana de Psicología. Código Ético del Psicólogo (Ethical Code of the Psychologist); Trillas: Mexico City, Mexico, 2010. [Google Scholar]
  64. American Psychological Association. Ethical Principles of Psychologists and Code of Conduct. With the 2016 Amendment to Standard 3.04; American Psychological Association Press: Washington, DC, USA, 2017. [Google Scholar]
  65. World Medical Association. World medical association declaration of Helsinki. JAMA 2013, 310, 2191–2194. [Google Scholar] [CrossRef] [Green Version]
  66. Meade, A.W.; Craig, S.B. Identifying careless responses in survey data. Psychol. Methods 2012, 17, 437–455. [Google Scholar] [CrossRef] [Green Version]
  67. Mahalanobis, P.C. On the generalized distance in statistics. Proc. Natl. Inst. Sci. India 1936, 12, 49–55. [Google Scholar]
  68. Penny, K.I. Appropriate critical values when testing for a single multivariate outlier by using the Mahalanobis Distance. J. R. Stat. Soc. Ser. C Appl. Stat. 1996, 45, 73–81. [Google Scholar] [CrossRef]
  69. Yentes, R.D.; Wilhelm, F. Careless: Procedures for Computing Indices of Careless Responding. R Package Version 1.1.3. Available online: https://cran.r-project.org/web/packages/careless/index.html (accessed on 31 July 2020).
  70. Tomczak, M.; Tomczak, E. The need to report effect size estimates revisited. An overview of some recommended measures of effect size. Trends Sport Sci. 2014, 21, 19–25. [Google Scholar]
  71. Mizumoto, A. Langtest (Version 1.0). Available online: http://langtest.jp (accessed on 15 July 2020).
  72. Korkmaz, S.; Goksuluk, D.; Zararsiz, G. MVN: An R package for assessing multivariate normality. R J. 2014, 6, 151–162. [Google Scholar] [CrossRef] [Green Version]
  73. Muthén, B.O.; du Toit, S.; Spisic, D. Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes. 1997; Unpublished Manuscript. [Google Scholar]
  74. Yu, C. Evaluating Cutoff Criteria of Model Fit Indices for Latent Variable Models with Binary and Continuous Outcomes. Unpublished. Doctoral Dissertation, University of California, California, CA, USA, 2002. [Google Scholar]
  75. Hancock, G.R.; Mueller, R.O. Rethinking construct reliability within latent variable systems. In Structural Equation Modeling: Present und Future—A Festschrift in Honor of Karl Joreskog; Cudeck, R., Du Toit, S., Sorbom, D., Eds.; Scientific Software International: Lincolnwood, IL, USA, 2001; pp. 195–216. [Google Scholar]
  76. Stucky, B.D.; Edelen, M.O. Using hierarchical IRT models to create unidimensional measures from multidimensional data. In Handbook of Item Response Theory Modeling: Applications to Typical Performance Assessment; Reise, S.P., Revicki, D.A., Eds.; Routledge/Taylor & Francis Group: New York, NY, USA, 2015; pp. 183–206. [Google Scholar]
  77. Gorsuch, R.L. Factor Analysis; Lawrence Erlbraum Associates: Hillsdale, NJ, USA, 1983. [Google Scholar]
  78. Quinn, H.O.C. Bifactor Models, Explained Common Variance (ECV), and the Usefulness of Scores from Unidimensional Item Response Theory Analyses. Ph.D. Thesis, The University of North Carolina, Chapel Hill, NC, USA, 2014. [Google Scholar]
  79. Saris, W.E.; Satorra, A.; van der Veld, W.M. Testing structural equation models or detection of misspecifications? Struct. Equ. Model. Multidiscip. J. 2009, 16, 561–582. [Google Scholar] [CrossRef]
  80. Rosseel, Y. Lavaan: An R package for structural equation modeling. J. Stat. Softw. 2012, 48, 1–36. [Google Scholar] [CrossRef] [Green Version]
  81. Jorgensen, T.D. semTools: Useful Tools for Structural Equation Modeling. R Package Version 0.5-3. Available online: https://CRAN.R-project.org/package=semTools (accessed on 15 October 2021).
  82. Veroniki, A.A.; Jackson, D.; Viechtbauer, W.; Bender, R.; Bowden, J.; Knapp, G.; Kuss, O.; Higgins, J.P.T.; Langan, D.; Salanti, G. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res. Synth. Methods 2016, 7, 55–79. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  83. IntHout, J.; Ioannidis, J.P.; Borm, G.F. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med. Res. Methodol. 2014, 14, 25. [Google Scholar] [CrossRef] [Green Version]
  84. Henze, N.; Zirkler, B. A class of invariant consistent tests for multivariate normality. Commun. Stat. Theory Methods 1990, 19, 3595–3617. [Google Scholar] [CrossRef]
  85. Stucky, B.D.; Edelen, M.O.; Vaughan, C.A.; Tucker, J.S.; Butler, J. The psychometric development and initial validation of the DCI-A short form for adolescent therapeutic community treatment process. J. Subst. Abuse Treat. 2014, 46, 516–521. [Google Scholar] [CrossRef] [Green Version]
  86. Chen, F.F.; Hayes, A.; Carver, C.S.; Laurenceau, J.P.; Zhang, Z. Modeling general and specific variance in multifaceted constructs: A comparison of the bifactor model to other approaches. J. Personal. 2012, 80, 219–251. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  87. He, J.; van de Vijver, F.J.R. A general response style factor: Evidence from a multi-ethnic study in the Netherlands. Personal. Individ. Differ. 2013, 55, 794–800. [Google Scholar] [CrossRef]
  88. He, J.; Van de Vijver, F.J.R. Assessment of the general response style: A cross-cultural validation. Acta Investig. Psicol. 2019, 9, 14–24. [Google Scholar] [CrossRef]
  89. Dahlem, N.W.; Zimet, G.D.; Walker, R.R. The Multidimensional Scale of Perceived Social Support: A confirmation study. J. Clin. Psychol. 1991, 47, 756–761. [Google Scholar] [CrossRef]
Ijerph 19 00935 g001 550
Figure 1. Representation of the CFA and ESEM models tested. G: General factor; SO: Others support factor; FAM: Family support factor; FRI: Friends’ support factor.
Figure 1. Representation of the CFA and ESEM models tested. G: General factor; SO: Others support factor; FAM: Family support factor; FRI: Friends’ support factor.
Ijerph 19 00935 g001
Table
Table 1. Psychometric studies in adolescent samples.
Table 1. Psychometric studies in adolescent samples.
Version and CountryParticipantsFactor Analysis DesignFactorial ConfigurationMethodFactor RelationshipInternal ConsistencyInvariance
1. Chou [28]
Hong Kong		N = 475
Age: 17–18		1° PCA2 factors:
FAM: 9, 10, 11, 12
FRI: 1, 2, 3, 4, 5, 6, 7, 8		Estimator: NR
Rotation: Varimax		FRI and others = NR
FRI and FAM: 0.23
FAM and SO: NR		α:
FRI: 0.94
FAM: 0.86
SO: NR		NR
2. Canty-Mitchell et al. [34]
EEUU		N = 196
Age: M = 15.8
SD = 0.97		1° EFA3 factors:
FAM: 3, 4, 8, 11
FRI: 6, 7, 9, 12
SO: 1, 2, 5, 10		Estimator: Principal axis
Rotation: Oblique		FRI and SO = 0.66
FRI and FAM: 0.53
FAM and SO: NR		α:
FRI: 0.89
FAM: 0.91
SO: 0.91		NR
3. Cheng and Chan [44]
Hong Kong		N = 2105
Age: M = 14.8
SD = 1.58		1° CFA3 factors:
FAM: 3, 4, 8, 11
FRI: 6, 7, 9, 12
SO: 1, 2, 5, 10
2 factors:
F1: 6, 7, 9, 12, 1, 2, 5, 10
F2: 3, 4, 8, 11		Estimator: ML
Model: 3 and 2 factors		FRI and SO = 0.85
FRI and FAM: 0.35
FAM and SO: 0.42		α:
FRI: 0.76
FAM: 0.78
SO: 0.69		Age and sex
Configuration
4. Bruwer et al. [35] South AfricaN = 502
Age: 11–23		1° CFA3 factors:
FAM: 3, 4, 8, 11
FRI: 6, 7, 9, 12
SO: 1, 2, 5, 10		Estimator: WLSMV
Model: 3 factors		FRI and SO = 0.676
FRI and FAM: 0.616
FAM and SO: 0.747		α:
FRI: 0.86
FAM: 0.86
SO: 0.88		NR
5. Ramaswamy et al. [37]
EEUU		N = 635
Age: 11–15		1° CFA3 factors:
FAM: 3, 4, 8, 11
FRI: 6, 7, 9, 12
SO: 1, 2, 5, 10		Estimator: ML
Model: 3 factors		FRI and SO = 0.30
FRI and FAM: 0.27
FAM and SO: 0.35		α:
FRI: 0.75
FAM: 0.63
SO: 0.72		NR
6. Edwards [31] EEUUN = 290
Age: 11–18		1° PCA3 factors:
FAM: 3, 4, 8, 11
FRI: 6, 7, 9, 12
SO: 1, 2, 5, 10		Estimator: NR
Rotation: Varimax		NRα:
FAM: 0.88
FRI: 0.90
SO: 0.86		NR
7. Mosqueda et al. [22]
Chile		N = 247
Age:
14–19		1° PCA3 factors:
FAM: 3, 4, 8, 11
FRI: 6, 7, 9, 12
SO: 1, 2, 5, 10		Estimator: NR
Rotation: Varimax		NRα:
FAM: 0.850
FRI: 0.887
SO: 0.786		NR
8. Wilson et al. [45] GhanaN = 717
Age:
15–18		1° CFA;
2° EFA		3 factors:
FAM: 3, 4, 8, 11
FRI: 6, 7, 9, 12, 10
SO: 1, 2, 5		CFA
Estimator:
MLR
Model:
3 factors
EFA
Estimator: ML
Rotation: NR		FRI and SO: 0.35
FRI and FAM: 0.37
FAM and SO: 0.50		α:
FAM: 0.73
FRI: 0.61
SO: 0.74		Sex: Metric
9. Trejos-Herrera et al. [46]
Colombia		N = 763
Age: 14–18		1° CFA;
2° EFA		3 factors:
FAM: 3, 4, 8, 11
FRI: 6, 7, 9, 12
SO: 1, 2, 5, 10		CFA
Estimator:
ML
Model:
7 models
EFA
Estimator: ML
Rotation: NR		FRI and SO: 0.46
FRI and FAM: 0.33
FAM and SO: 0.42		α:
FAM: 0.82
FRI: 0.84
SO: 0.75		NR
10. Navarro-Loli et al. [30]
Perú		N = 242
Age: 12–16		CFA3 factors:
FAM: 3, 4, 8, 11
FRI: 6, 7, 9, 12
SO: 1, 2, 5, 10		Estimator:
Robust ML
Model: 3 factors
Bifactor		FRI and SO = 0.69
FRI and FAM: 0.37
FAM and SO: 0.62		α:
FAM: 0.814
FRI: 0.874
SO: 0.824
Omega: (>0.85)
FAM: NR
FRI: NR
SO: NR		NR
11. Aloba et al. [47]
Nigeria		N = 1335
Age: 13–18		CFA3 factors:
FAM: 5, 6, 7, 8
FRI: 9, 10, 11, 12
SO: 1, 2, 3, 4
2 factors:
FAM and SO: 1, 2, 3, 4, 5, 6, 7, 8
Am.: 9, 10, 11, 12		Estimator: ML
Model: 3 factors and Hierarchical		FRI and SO = 0.65
FRI and FAM. = 0.67
FAM and SO = 0.82		α:
FAM = 0.82
SO = 0.80
FRI = 0.78		Sex
scalar
12. Okki et al. [29] IndonesiaN = 299
Age: 12–18		CFA3 factors:
FAM.: 3, 4, 8, 11
FRI.: 6, 7, 9, 12
SO: 1, 2, 5		Estimator: ML
Model: 3 factors		FRI. and SO = 0.79
FRI. and FAM = 0.62
FAM. and SO = 0.71		α:
FAM. = 0.81
FRI. = 0.78
SO = 0.79		Sex: scalar
Note. NR = does not report. EFA = exploratory factorial analysis. CFA = confirmatory factorial analysis. PCA: principal components analysis. FAM: Family support dimension. SO: significative others dimension. FRI: Friend support dimension.
Table
Table 2. Descriptive statistics for the MSPSS items (n = 461).
Table 2. Descriptive statistics for the MSPSS items (n = 461).
MSDSkKuCVM aSpearman’s Correlation b
SexAge
mps13.4611.150−0.302−0.8232.7540.058−0.115 *
mps23.4071.297−0.341−1.0062.5380.081−0.068
mps53.4311.259−0.313−0.9842.5500.007−0.082
mps103.4241.269−0.320−0.9752.5150.042−0.058
mps33.8271.052−0.617−0.3393.675−0.100 *−0.174 **
mps43.6621.167−0.495−0.7723.256−0.071−0.193 **
mps83.0631.256−0.079−1.0232.239−0.077−0.142 **
mps113.3551.282−0.260−1.0532.460−0.055−0.141 **
mps63.3631.173−0.085−1.0002.6340.129 **−0.025
mps73.3921.233−0.162−1.0402.5900.073−0.018
mps93.3421.353−0.181−1.2222.6710.110 *−0.035
mps123.1591.379−0.096−1.2322.2360.092 *0.004
Note. CVM: Cramer–von Mises univariate normality test. a All are p < 0.001. b Correlations equal to or higher than 0.10: p < 0.05. Sk and Ku: skew and kurtosis coefficients. * p < 0.05. ** p < 0.01
Table
Table 3. Fit indicators of the main tested models.
Table 3. Fit indicators of the main tested models.
CFAESEM
3 FactorsOne FactorBifactor3 FactorsBifactor
WLSMV-χ267.399942.20751.57119.80011.605
df5154456056
CFI1.00 0.979 1.00 1.00 1.00
WRMR0.769 2.875 0.672 0.417 0.319
RMSEA0.0260.1890.0180.000.00
IC 90%(0.00)(0.042)(0.179)(0.200)(0.00)(0.03)(0.00)(0.00)(0.00)(0.00)
GamaHat0.994 0.759 0.997 1.0 1.0
Note. CFA: confirmatory factorial analysis. ESEM: exploratory structural equation modeling.
Table
Table 4. Measurement models evaluated in the MSPSS using CFA (n = 461).
Table 4. Measurement models evaluated in the MSPSS using CFA (n = 461).
3 Factors1 FactorBifactor—CFA a
SOFAMFRIFgSOFAMFRIICEVARPB
mps10.838 0.7800.8260.145 0.9700.056
mps20.861 0.8040.8480.136 0.9750.052
mps50.836 0.7800.8230.146 0.9690.052
mps100.834 0.7750.8210.146 0.9690.056
mps3 0.890 0.8070.728 0.463 0.7120.109
mps4 0.928 0.8470.727 0.627 0.5730.165
mps8 0.747 0.6740.582 0.483 0.5920.158
mps11 0.784 0.7100.627 0.461 0.6490.132
mps6 0.8800.8320.740 0.4530.7270.124
mps7 0.8940.8530.736 0.5060.6790.159
mps9 0.9350.8940.765 0.5300.6760.169
mps12 0.8880.8540.698 0.5830.5890.223
Correlations
SO1 ---
FAM0.7851 ---
FRI0.8050.6521 ---
Bifactor indicators
ECV----0.7510.0290.3700.334--
FD----0.9560.2840.8800.859--
ωh----0.8710.0260.3340.314--
H----0.9450.0770.5990.601--
Note. SO: significant others. FAM: family. FRI: friends. GF: general factor. CEV: common explained variance. ωh: omega coefficient with the variance retained in the own analyzed factor. H: replicability coefficient. FD: factor determinability. a Bifactor-CFA model estimated with tau equivalence in the SO factor.
Table
Table 5. Measurement models evaluated in the MSPSS using ESEM (n = 461).
Table 5. Measurement models evaluated in the MSPSS using ESEM (n = 461).
3 FactorsBifactor—ESEM
SOFAMFRIFgSOFAMFRII-ECVI-ARPB
mps10.7030.0990.0580.6110.4720.0840.0780.6130.277
mps20.8940.0230.0420.6000.6230.0220.0570.4790.340
mps50.8160.0120.0450.6740.5460.0190.0660.6000.157
mps100.7250.0210.1470.7030.4830.0350.0400.6770.102
mps30.0510.8160.1410.6020.0280.6240.0700.4780.341
mps40.0130.9850.0360.5520.0020.7770.0520.3340.534
mps80.2300.7270.2340.3960.1680.5760.0480.3020.702
mps110.1270.8000.1290.6040.1110.6220.1020.4710.175
mps60.1350.0190.7500.7670.0750.0020.3720.8030.085
mps70.0440.0510.8260.7500.0060.0330.5370.6600.137
mps90.0190.0710.8710.8890.0190.0190.2960.8990.006
mps120.0150.0500.9300.8580.0200.0790.3240.8680.005
Correlations
SO1 1
FAM0.7671 0.5741
FRI0.7480.6011 0.3840.2091
Bifactor indicators
ECV---0.6080.1770.2380.106
FD---0.9360.8400.9240.712
ωh---0.7950.0590.0850.029
H---0.9330.6290.7680.449
Note. SO: significant others. FAM: family. FRI: friends. Fg: general factor. ECV: common explained variance. ωh: omega coefficient with the variance retained in the own analyzed factor. H: replicability coefficient. I-ECV: explained common variance in item level. I-ARPB: absolute relative parameter bias in item level. FD: factor determinability.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Merino-Soto, C.; Boluarte Carbajal, A.; Toledano-Toledano, F.; Nabors, L.A.; Núñez-Benítez, M.Á. A New Story on the Multidimensionality of the MSPSS: Validity of the Internal Structure through Bifactor ESEM. Int. J. Environ. Res. Public Health 2022, 19, 935. https://doi.org/10.3390/ijerph19020935

AMA Style

Merino-Soto C, Boluarte Carbajal A, Toledano-Toledano F, Nabors LA, Núñez-Benítez MÁ. A New Story on the Multidimensionality of the MSPSS: Validity of the Internal Structure through Bifactor ESEM. International Journal of Environmental Research and Public Health. 2022; 19(2):935. https://doi.org/10.3390/ijerph19020935

Chicago/Turabian Style

Merino-Soto, César, Alicia Boluarte Carbajal, Filiberto Toledano-Toledano, Laura A. Nabors, and Miguel Ángel Núñez-Benítez. 2022. "A New Story on the Multidimensionality of the MSPSS: Validity of the Internal Structure through Bifactor ESEM" International Journal of Environmental Research and Public Health 19, no. 2: 935. https://doi.org/10.3390/ijerph19020935

APA Style

Merino-Soto, C., Boluarte Carbajal, A., Toledano-Toledano, F., Nabors, L. A., & Núñez-Benítez, M. Á. (2022). A New Story on the Multidimensionality of the MSPSS: Validity of the Internal Structure through Bifactor ESEM. International Journal of Environmental Research and Public Health, 19(2), 935. https://doi.org/10.3390/ijerph19020935

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop