A Confirmatory Evaluation of an Educational Orientation Tool for Pre-University Students

Abstract

This study incorporates the design and validation of a questionnaire for the evaluation of Careers Advisory Services and the systematic processes that influence it (family, peer groups, socioeconomic status, etc.). In addition, it examines its psychometric properties within a multicultural population of students attending educational centres in the south of Spain. It seeks to create a valid instrument that is reliable as a measurement tool and useful for evaluating decision making situations relevant to the future working context. A perspective of working life is given through consideration of the degree choices made by those involved in the decision-making process. The metrics used showed high content and construct validity. Structural equation modelling (SEM) and confirmatory factor analysis (CFA) were employed. Indicators described by Wald and Lagrange were used to examine and modify the model in order to obtain a model that best fits relevant theory and goodness of fit criteria.

1. Introduction

In recent years significant changes have taken place in contemporary Western society. The difficulty of accessing employment, delayed emancipation and the proliferation of paid jobs that are not characterised by specific training (disc-jockey (DJ), youtubers, influencers, etc.), have led young people to reconsider their career opportunities in a way that differs from two decades ago. A number of research studies exist that focus on university education. These relegate issues that emerge throughout the process of basic education to the background. These have largely ignored the systematic processes (social relationships, the arts, philosophy, etc.) required throughout learning in order to achieve better human development [1].

Despite challenges to the decision making processes of adolescents, parents respect them, assuming a subsidiary role of support and listening, that influences upon security and self-efficacy [2]. The present research focuses specifically on these systematic processes which, in one way or another, influence the subsequent development of our working youth for attaining the skills and tools to get a future job.

Education is a fundamental piece of reproduction, the structure of power relationships and of the symbolic relationships between classes. This puts emphasis on the importance of the inherited cultural capital within the family unit that goes hand in hand with education, as a key to success at school [3].

To this end, the family fulfils the task of passing on culture, values and traditions, and establishing the basic norms that will guarantee coexistence in society [4]. From an educational point of view, the difficulty of deciding, on behalf of students, when choosing an academic degree becomes a real problem [5,6,7,8]. This issue is particularly pertinent for students in the final stage of their education who lack motivation due to not being adequately informed [9]. This is the focus of the present study. Despite challenges to the decision making processes of adolescents, parents respect them, assuming a subsidiary role of support and listening, that influences upon security and self-efficacy [10].

Thus, during the stage of secondary education that coincides with academic choice, vocational maturity is still found in a state of crescendo. From this arises the urgent need to work with both families and educational centres in a way that encourages students to take decisions in order to overcome feelings of uncertainty [11].

Multiple factors have an influence on decision making. These include the influence of the peer group, the economic situation and individual characteristics, amongst others [12,13,14]. This being said, it has been noted that, amongst these factors, the parental context is one of the most significant variables [15,16,17,18].

At the time of making a decision, children feel that they are supported when they see parents who are involved and hold positive expectations [19,20]. One of the central variables is the educational level of parents. If the parents themselves possess university qualifications, it is more likely that their children will also decide to attend university [10].

Another of the determining factors in the choice of one’s future career is the family’s socioeconomic level [21]. This is significant given that it has as much of an effect on the resources available to invest in education, as on the urgency for the child to enter the workplace in order to contribute with an income [22].

Thus, achieving professional recognition and prestige, or obtaining a well-paid job, are extrinsic motives [23] in the decision to study for a specific academic career.

The vocational and professional orientation and guidance received from careers services, is another one of the key points in the present work. Studies conducted by Domínguez et al. [24] reported that interviewed pre-university students confessed to not having received sufficiently adequate careers advice. This could be because young people rely on their inner social circle (relatives, friends and teachers at their school), to provide sources of information regarding the careers they should train for [25].

On the other hand, it has been indicated by Martínez-Vicente [26], that modifications to study plans, increases in options and specialisation, and the creation of new degree courses, all of which are linked to the immaturity, doubts and lack of information of students make it necessary to design actions which stimulate vocational development and facilitate appropriate, realistic and responsible decision making [27,28,29,30,31].

Given this, it is fundamental that educational orientation, exactly as it is understood in the present research, acts to develop individuals who are capable of making the right decision at each moment of the educational stage. To achieve this, evaluation of careers advice through a standardised, reliable and valid questionnaire is vital [32].

In this context, the main objective of the present research is to develop an instrument that measures the systematic processes that influence the degree choices of pre-university students. To address this objective, the questionnaire will be required to meet established psychometric requirements for validity and reliability. In order to test these characteristics, the methodology of structural equation modelling (SEM) will be followed. This methodology is a type of analysis that uses Latent Class Models. This consists of multivariate regressions for relating response patterns to a set of factors that cannot be observed directly but exist through the responses of the people evaluated [33]. More concretely, SEM methodology consists of the following phases [34]:

• Specification of the measurement model. In this, the latent traits and their dimensions are established;
• Implementation of a structural equation system;
• Use of goodness of fit criteria. This has the aim of relating the validation results with the dimensional structure of the tool being evaluated;
• Repeat specification of the measurement model. Adding or removing associations between factors, always within that which is permitted by the theoretical basis of the research.

2. Results

2.1. Participants

In order to carry out the study a sample of 1302 s year Baccalaureate students was used (54.2% female and 45.7% male). The sample was aged between 17 and 19 years. The students came from seven centres of compulsory secondary and Baccalaureate education in the autonomous city of Melilla (87.6% of the sample came from a public school and 12.4% from a public/private mixed school). Participants represented 98.90% of the overall number of second year Baccalaureate students. The branches of Baccalaureate studies examined were: Science and Technology (33.10%), Humanities and Social Sciences (63.30%) and Art (4.60%).

2.2. Instrument

Members of a multi-disciplinary team from the University Careers Advisory office (University of Granada) dedicated to the personal, professional and academic orientation of pre-university and university students were involved. They created and validated a new questionnaire to measure the way in which this service influences university degree choices of Baccalaureate students. The questionnaire incorporated variables relating to socioeconomic level and family influence. The designed instrument follows the main theoretical foundations and international recommendations for the construction of tests. For data collection, the present study counted on the voluntary participation of students who presented for PEBAU tests (University Baccalaureate Assessment Test) in the autonomous city of Melilla. Permission was granted beforehand in order to access the schools.

For the development of the questionnaire the Social Survey 2010: Education and Homes in Andalusia (ESOC2010) [35] and the Questionnaire of Areas of Basic Professional Academic Interests: CIBAP, were taken into consideration.

With the aim of examining the understanding and clarity of items, the first version of the questionnaire was administered to a public school and the other to a mixed school (public-private). Following this pilot test, the initial instrument was modified under consideration of the results obtained. Items were eliminated that were difficult to understand and comprehend by the students. In the exploratory phase, the final version of the adapted instrument was developed. To achieve this, three rounds of analysis were carried out through discussion between members of the coordinating group. These analyses bore in mind the adjustments and corrections suggested by the expert group. The agreement percentage between members of the coordinating group in the first round of discussion ranged between K ≥ 60 and K ≥ 75. In the second round it ranged between K ≥ 71 and K ≥ 84; and in the third round it ranged between K ≥ 83 and K ≥ 91.

The items whose agreement percentage between judges from the coordinating group was K ≥ 70 and which were largely rated below three on the Likert scale by the expert group, were modified, eliminated or re-grouped. Following application of this method, various elements were not significantly modified, two were completely eliminated and some were re-grouped into two groups. This means that the final questionnaire was composed of 22 questions divided into seven dimensions or factors.

The subsequent version of the questionnaire remained composed of 22 items, according to which students were required to indicate the correct response option. Of these items, six were socio-demographic in nature. This version was administered to the sample of N = 1302 and was used to conduct the psychometric analysis of reliability and validity. Exploratory and confirmatory data analysis was utilised for this.

2.3. Procedure

Firstly, contact was established with the careers advisory services and the management teams of seven centres for Compulsory Secondary and Baccalaureate Education. These participated via a non-probabilistic, accidental and causal sampling approach. All of the schools were sent an email informing them about the voluntary and anonymous nature of the study, alongside its aims and objectives. Subsequently, members of the university’s careers advisory services administered the questionnaire to students in a paper format and through Google forms (online) (see Appendix A). The study was approved by the Vice Dean’s Ethical Committee for Social Responsibility of the Education and Sport Sciences Faculty in Melilla (University of Granada).

2.4. Data Analysis

Statistics for the univariate (kurtosis and asymmetry) and grouped (mean and standard deviation) items were initially calculated. Following this, the sample (N = 1302) was used to examine the dimensions of the questionnaire. The first step was to conduct an exploratory factor analysis (EFA) with the responses given to each questionnaire item for each source of information. A polychoric correlation was used to provide an entrance matrix for the data. Further, the extraction method used was principal component analysis and the Varimax rotation method was used with Kaiser correction [36]. The EFA was conducted with a pilot sample of N = 215. The Kaiser–Meyer–Olkin (KMO) index was calculated with the aim of analysing validity [37], with values greater than 0.5 being considered acceptable. Our measurement of KMO = 0.822, indicates to us that the method is appropriate. In addition, we performed the Bartlett sphericity test which measures the adequacy of the correlation matrix for carrying out the factor analysis. A value higher than 0.05 indicates that the conditions are not appropriate for conducting a factor analysis. In our analysis, the value obtained was significant at the level of 0.000, which corroborates that the conditions are appropriate for performing a factor analysis. Finally, we can analyse the variance between all of the variables analysed in the overall variance table. The result of this analysis of pilot sample supports the existence of seven factors that explain 59.077% of the overall variance. Through conducting the Cattell sedimentation test shown in Figure 1, it is graphically confirmed that the optimal number of factors with a value higher than one is seven [32]. Factor 1 relates to the social influence of the student at the time of making decisions, with this explaining 17.026% of the variance. Factor 2 refers to the parental influence upon the student at the time of making decisions, with this explaining 8.778% of the variance. Factor 3 pertains to the Careers Advisory Service’s role at the time of making decisions, with this explaining 8.358% of the overall variance. Factor 4 refers to the association between the PEBAU and decision making, with this explaining 7.584% of the overall variance. Factor 5 refers to the information received by the Student Orientation Department, with this explaining 7.108% of the overall variance. Factor 6 refers to decision making of the student, with this explaining 5.381% of the overall variance. Factor 7 refers to decision making under the influence of parents, with this explaining 4.842% of the overall variance [32]. For the confirmatory analysis, the number of factors was reduced, adapting it according to the weight of its variances, finally remaining at four factors.

Estimation of reliability of the scale ratings was made through Cronbach alpha = 0.912, this being appropriate for ordinal data [38].

2.5. Confirmatory Factor Analysis

Subsequently, confirmatory factor analysis (CFA) was performed using SEM methodology [34], which examined the multivariate regression coefficient produced from structural equations. Evaluation of the fit of the data to the model was conducted according to multiple criteria: χ2 / df, comparative fit index (CFI) and root mean square error approximation (RMSEA). The literature suggests that fit can be considered to be adequate when χ2 / df < 5, CFI > 0.90 and RMSEA < 0.08 [39]. The data were analysed through the statistical software packages SPSS 20, LISREIL v9.1 and PANTH GRAHF.

Given the existence of a questionnaire model with factor validation and evidence supporting the discriminatory power of its items, we conducted confirmatory factor analysis (CFA) using the structural equation modelling (SEM) methodology. Through this we examined the multivariate regression coefficients based on structural equations [34] in order to confirm the suitability of indicators and evaluate the latent variables. With regards to the application of indexes of goodness of fit between the derived data matrix and the matrix reproduced by the model, the difference between them was not statistically significant, so we determined that both matrices were close, indicating that the measurement model and the observed data fit together. This statistical test was performed with N = 1302 and analysed with the software LISREL, version 9.1. The CFA is presented through path diagrams, in which circles represent the latent variables and rectangles represent the observed variables. Arrows with a single point are used to indicate the direction of influences, whilst arrows with two points represent covariance between the four latent variables (Figure 1).

The results of the CFA confirm adequate fit of the data to the model (M1). This model originates from both exploratory factors and a theoretical model. Parsimonious fit was χ2 / df = 110.40/ 77.03; CFI was 0.911 and (RMSEA) = 0.063 [90% CI = 0.053–0.085]. Though the scores produced were adequate when two factors were eliminated, it is necessary to examine a number of other indices. The most interesting of these is the root mean square error approximation (RMSEA), which was slightly below the critical limit: 0.085. For this reason, focus is shifted to the adjusted model.

Examination of the CFA was conducted using SEM methodology via path analysis. Just as can be observed in Figure 1, all of the regression weights were higher than 0.05, whilst covariance between factors ranged between 0.12 and 1.52.

The evaluation conducted according to the SEM methodology verifies that the derived coefficients show positive agreement with the theory employed to configure the measurement model, with the exception of one weakly linked value.

In order to carry out the CFA, the ratio between the chi-squared output and degrees of freedom was observed. NPAR (parsimonious measures designed to achieve structural adjustment), this produced χ²/df = 2.965, with this value falling within the range of values accepted by Kline [31], being lower than 3 (

Table 1. NPAR, χ2, df, p and χ2/df.

Model	NPAR	χ2	df	p	χ2/df
Default model	116	2324.605	696	0.000	2.965
Saturated model	781	0.000	0
Independence model	36	10,8798.913	752	0.000	13.652

).

Analysis of the multivariate regression coefficients was conducted through examination of the covariance matrix of the observed variables. The program Lisrel 9.1 was used to carry this out.

The regression coefficients between the latent and observed variables are all positive in nature and range between 0.12 and 1.52. The influence exercised by the latent variable over the observed variables indicates that when the former increases by 1, the latter also increases to the same extent. Eight items and four of the seven previous factors are conserved. The factors referring to PEBAU were eliminated with the rest being regrouped, this meant the AFE items were eliminated.

The readjusted model (M2) arose following modification of the first model (Figure 1), with inappropriate elements [40,41] from M1 being eliminated. Eight items and three of the four factors from M1 were conserved.

In this way, the first factor is composed of elements related with the influence of the educational level of the parents, with father’s education being as influential as mother’s education [10]. The second factor is associated with the influence of parents, friendship groups, economic level and social status on decision making with regards to the degree to be studied, and the activities developed by the careers advisory services (for example, information about whether or not careers advisory services were accessed, which activities were carried out in order to learn whether students knew the university degree they wished to study) [12,13,14]. Finally, the third factor is associated with the influence of the students’ personal interests when making academic decisions and their motivation towards studying for an academic degree (for example, to exercise in the same profession as one’s parents, to acquire a better socioeconomic status, etc.) [10,20,42].

It is interesting to consider the level of RMSEA and to establish CFI, Tucker–Lewis Index (TLI), Normalized Fit Index (NFI) and Parsimonious Normalized Fit Index (PNFI) values (

Table 2. Comparative summary of fit of goodness and specifying model.

Model	CMIN	p	Absolute Adjustment Indices			Incremental Adjustment Indexes
			LO 90	HI 90	RMSEA	PNFI	NFI	CFI	TLI
Model 1: 4 factors, 14 items	1727.6	0.00	0.083	0.087	0.063	0.775	0.888	0.911	0.867
Model 2: 3 factors, 1 s	834.6	0.00	0.065	0.078	0.045	0.730	0.935	0.928	0.926
order factor, 12 items

). Wald and Lagrange [43] modification indices were used with approximation estimation values, in order to make a comparison with the Lagrange multipliers and missing parameters from the model. The Lagrange test suggests the introduction of new associations into the model, through a new second order factor called a coexistence factor, which subsumes the fourth factor. At the same time the Wald test suggests eliminating four elements and applying relationships between certain latent errors (Table 2).

All of the outcomes from the model were well adjusted to relevant theory. Adjusted goodness of fit, CFI and RMSEA values all satisfied the level of 0.045. This confirms is validity according to previous research where adequate fit is considered at χ2 / df < 5, CFI > 0.90 and RMSEA < 0.08 [39] (Figure 2).

3. Discussion

The present research presents the results of the validation of a questionnaire to identify the factors that influence academic degree choice. Whilst the individuals themselves have a strong opinion about their own self-efficacy [44], there are other social agents and/or factors that have an effect on this decision. The SEM methodology validates the construct according to a three-factor structure. This denominated one factor as parental educational level, and influence of parents, friendship groups, economic status and social status on decision making regarding what degree to study. The final two were activities developed by the careers advisory services, and the influence of student’s personal interests on making academic decisions and their motivation towards studying for an academic degree. Overall, the results show the validity and reliability of the questionnaire, demonstrated by the table of factor loadings, all of which show appropriate saturation.

From an educational point of view, the prevailing perception held by society is to continue with university studies following completion of the Baccalaureate. This idea is socialised practically from the day we are born. Using the words of Delors [45], education is “the passport to life”. However, when the moment arrives to make this academic decision, we do not know if the choice is appropriate or not. From this data, it is revealed that the basic factors influencing decision making are related to the likes and interests of pupils. Parental orientations are then added to this, given that at these ages, parental advice continues being important for students [46].

Analysis of the items demonstrates a positive overall correlation to the right of the item (ri-t) for all items, with values between 0.016 and 0.641. The exploratory factor was produced using the extraction method, having previously used goodness of fit and AFE indicators [47]. There are four factors that explain 69.087% of total variance, with an orthogonal Equamax rotation and a limit in the degree of the correlation between the variable and the factor proposed by Comrey [48], of 0.3 [32].

The confirmatory factor analysis (CFA) was conducted through progressive verification of the two structural equation models, with SEM being performed followed by adjustment according to a set of fit indices [49]. M1 has a parsimonious index with PNFI being close to one. M2 is more complex and hierarchical, thus being considered a more appropriate setting for the goodness of fit approach, including RMSEA = 0.045 and CFI = 0.928 [50,51,52,53]. These structural equations allow each item to be evaluated and for viable modifications to be made, factors five and thirteen were eliminated and fourteen was reset to factor three, taking a stepwise approach, until M2 is reached. This contains a second order factor called the decision factor and three first order factors. These are the educational level of parents, influence of parents, friendship groups, economic status and social status on decision making, the influence of students’ own personal interests on making academic decisions, and motivation towards studying for an academic degree. These data were drawn from a sample of students from different secondary schools in a multicultural context, from which correlations were calculated and a model was specified using exploratory factor analysis (EPT). From these, validity was thoroughly demonstrated.

Internal consistency, estimated according to the index of Cronbach alpha, is considered adequate for each of the considered factors. Despite this, it should be kept in mind that the model was modified based on data drawn from a single specific sample. Thus, the capitalisation of chance through cross-validation should be studied in order to extend the results beyond the population of the current study.

With regards to other influential social factors steering academic choices, the students recognised that friends are important for providing support but that their influence is not sufficient enough for making decisions [20].

Recommendations of careers advisors are not considered by students at the moment of making a decision. This provides a glimpse of the scant or complete lack of participation of these professionals in this educational transition that is so important for students. In this sense, Martínez and Zurita [54] point out that educational orientation and guidance is an indispensable key element within the educational context.

The results also identify socioeconomic status as an influential factor in decision making. Within the context that is the object of the present study, it determined the selection of educational options close to the students’ place of residence whilst also considering the balance between costs and expenses of embarking upon a university degree [21].

Instruments created previously for measuring students’ motivation evaluate the teaching–learning processes that improve motivation [55,56,57]. However, the factors themselves that influence motivation at the time of choosing a specific degree course have not been previously considered, this was the central theme of the present research.

4. Conclusions

The present study provides as its main outcome the elaboration and validation of a measurement instrument of the educational orientation and influential factors at the time of making an academic choice in Compulsory Secondary Education and Baccalaureate students. It obtained good fit indices in both the exploratory and confirmatory analyses. Validation reduced the scale to a total of 8 items, which were grouped into three dimensions derived from the initial theoretical model – educational level of parents-influence of parents, friendship groups, economic status and social status on decision making, the influence of students’ personal interests when making academic decisions, and motivation towards studying for an academic degree. From these results, the present research study, in the same way as others [7,58,59], urges the need to initiate orientation and assessment processes during the first years of secondary school.

The scarce and, at times, inexistent guidance received at these educational stages means that families, specifically parents, influence decision making pertaining to studies. Friendships appear as social agents that impact students’ educational decisions, however, these are not decisive at the time of making an educational choice [10,14,20,42].

The absence of vocational guidance, a lack of information provision and fear of making mistakes [38] are the main determinants leading students to be swayed by the opinions of their parents.

According to recent research studies, students have negative perceptions of their orientation and require a training system in order to have broad and flexible training opportunities. Further, the process of personal discovery should take place from early ages in order to strengthen the decision-making capacity throughout the academic and professional journey [60,61,62,63,64,65].

The present study outlines the need to involve careers advisory services as an essential aspect in the training and professional development of the student body [66] for influence the subsequent development of our working youth for attaining the skills and tools to entrepreneurship for future job.

These conclusions propel us to continue conducting research on this topic, given that not many sociological research studies deal with career orientation from a secondary and baccalaureate educational perspective.