Psychometric Properties of the Affective Dimension of the Generic Macro-Competence Assessment Scale: Analysis Using Rasch Model

Abstract

The study of the affective dimension of transversal competences is essential for the development of responsible behaviors and maintaining attitudes committed to sustainable development. The importance attributed to each of these factors can predict behavior implementation and awareness of values for sustainable development that reflect the acquisition and internalization of sustainability-related generic competences. This study aimed to determine the psychometric properties of the affective dimension of the Generic Macro-Competence Assessment (AGMA) scale by applying Rasch measurement model to a sample of Spanish university students, comprising 387 Spanish university students (74.9% women; mean age = 21.24; WD = 3.54; range: 17–34). Results demonstrated a lack of adjustment to the Rasch model due to item 1, and all items showed disordered response category thresholds. The remaining nine-item scale achieved all requirements of the model (χ2 = 61.46; p = 0.052), including unidimensionality. Thus, the scale’s psychometric properties indicate an easy-to-apply instrument for screening these factors for coping strategies in undergraduate and graduate Spanish students. The results can help in justifying the design of interdisciplinary intervention programs, in which affective factors are essential for sustainable development education.

1. Introduction

The socio-affective factors of knowing others and ethical aspects of professional practice can be considered as predictive factors for the implementation of behaviors and awareness of values committed to sustainable development; moreover, these factors reflect the acquisition and internalization of such generic competences. If a student scores higher in this affective dimension, their commitment to sustainable development and the maintaining attitudes favorable to sustainability will likely be greater. Therefore, this is a relevant dimension for promoting sustainable development consistent with the UN-approved 2030 Agenda for Sustainable Development and sustainability education. This work is part of the Educational Innovation Project titled “New proposals for the integration of competences through transversal training evaluation and ICT” [1].

Globalization and rapid technological changes entail a need for labor that not only has specialized knowledge and skills, but also the generic competences necessary to adapt to new and emerging professional and technological demands [2]. The World Conference on Higher Education stresses the key role that higher education plays in the development of societies committed to values such as freedom, ethical principles, peacebuilding, and the principles of democracy [2]. The project Definition and Selection of Competences (DeSeCo) indicates that competences go beyond the juxtaposition of knowledge and skills, and also involve the ability to meet complex demands [3,4,5]. Competences can be classified into two types: specific and generic or transversal. Specific competences correspond to a certain field of knowledge or activity, while generic competences refer to those that are common to all fields of knowledge [6].

Generic competences are those skills that students need to become more transversally capable in their field of study, work, and other aspects of their lives. They are an important and recognized aspect of university education [7,8,9,10,11]. Within the scope of the Tuning Project, generic competences are grouped into three classes [12]: (a) instrumental, which means methodological or procedural skills and includes the capacity for analysis and synthesis; (b) interpersonal, which involves the ability to work in a team and relate to other people; and (c) systemic, which is a set of skills for understanding a system or set, including the capacity for autonomous learning and leadership. However, there is no universally recognized definition or universally accepted list of generic competences [13].

To be specific, the aforementioned generic cross-cutting competences make reference to the capacity to communicate with experts in other areas, for criticism and self-criticism, interpersonal skills, to communicate and acquire an ethical commitment, and social and professional responsibility, among many others [14]. A recent factor analysis study [15] identified the following skills as generic cross-cutting competences: effective communication, working in different contexts, personal responsibility, adaptability and problem-solving, updating professional decision-making, and personal and social ethical commitment, including items such as “making an ethical commitment in professional work, evaluating one’s professional performance critically,” “assessing one’s personal performance,” and “knowing one’s competences and limitations.”

Although different models have been proposed to assess generic-transverse competences, this study adopts a consolidated educational model of transversal training from the University of Concepción, Chile [16,17]. It considers the importance of four generic macro-competences: (1) critical thinking, described as “autonomous and active higher-order thinking, oriented toward the systematic analysis of the needs of the social and environmental context”; (2) communication, defined as “the ability to express ideas, knowledge, and feelings clearly, coherently, and precisely, adapting to different contexts according to the characteristics of the target audience”; (3) entrepreneurship and interdisciplinary work; and (4) social responsibility, as the “capacity, intention, and obligation to respond to society for professional actions or omissions that impact the common good.” The model proposes a competence-training program based on the integration of cognitive, affective, and behavioral dimensions. Deeper learning of competences arises from considering not only cognitive and behavioral dimensions but also affective dimensions [18].

Among the generic competences that receive less attention, which could constitute the basis for the development of other related competences, are those referred to as personal knowledge and knowledge of others, the ability to communicate adequately, and to put oneself in other people’s shoes, as well as ethical aspects of professional practice [19,20,21]; these are part of the so-called affective dimension according to the model defended by Navarro et al. [16].

Relevant for the development of moral, social, and cultural values, such as commitment to sustainable development, is that the personal motivation of the student body is based on solid professional interests [22,23]. This is the case because, as Pozo [24] indicates, the possibility of stability and its execution and transfer, is greater when there is coherence between what a person does and what a person likes and believes in. According to [22], this justifies the need for teachers to incorporate the affective dimension of generic competences, which implies both favorable attitudes toward their learning and development, as well as the value conferred on their professional performance. In this case, these attitudes are considered relevant since a higher score in the affective dimension implies greater commitment to the values of sustainability and sustainable development that must be promoted as transversal content at the university level.

Other previous studies demonstrate the importance of affective learning in the development of generic competences; this is known as the affective dimension [18,25] and is considered relevant for the formation of generic competences, so that students value and perceive the importance of their development, showing an interest in and work toward the implementation of behaviors consistent with their learning [26]. This in turn allows the commitment and developing actions in favor of sustainability and for sustainable development from the educational context. According to Buissink-Smith et al. [26], evaluating the affective dimension of generic competences using scales, such as the AGMA, makes it possible to monitor students’ learning. Through this we can determine the impact of interventions for the development of the affective dimension; moreover, the understanding of the relationships between the affective dimension and the development of actions or behaviors that reflect the acquisition or development of such competences can be deepened.

In this sense, the importance of evaluating the affective dimension of generic competences (the factors of knowing myself and others, etc) can be considered a predictive factor for the implementation of behaviors and awareness of values committed to sustainable development that reflect the acquisition and internalization of such generic competences. If a student scores higher in this affective dimension, their commitment to actions devoted to sustainable development and the maintenance of attitudes in favor of sustainability will likely be greater. Therefore, this is a relevant dimension for promoting sustainable development, consistent with the 2030 Agenda for Sustainable Development approved by the UN, and sustainability education.

To implement an adequate higher education curriculum for training in generic competences, it is essential to have instruments with which to evaluate their development evaluating those factors, such as knowing myself, knowing others, integrating positions, and practicing the profession sustainably and for the common good [17]. An extensive search has been carried out in different databases to locate existing instruments for the assessment of transversal competences, most of which were implemented in other contexts. In this sense, different instruments have been reviewed, such as the Assessment Scale of Critical and Self-critical Ability [8], which includes seven items regarding the ability to analyze different situations from different viewpoints, to respectfully contrast different approaches and perspectives when discussing one’s ideas or arguments, and considering associated themes or contents that come from other sources; the adaptation of the Collegiate Learning Assessment (CLA), a test for assessing generic competences on an international scale [27]; a self-administered assessment instrument in the context of blended learning (b-learning-Evalsoft system) for the self-assessment of teamwork competences (soft skill) [28]; the instrument for assessing generic competences, such as communication and problem-solving in the context of the Australian education system known as Swinburne University Apprentice Skills Assessment (SUASA) [29]; and the assessment tool known as Progress Report on Generic Skills (PROG), designed in Japan to assess the generic competences of university students, such as communication and generic critical thinking skills [30]. Another instrument for the evaluation of the affective dimension of generic competences was constructed and validated by González et al. [18]; this instrument consists of 15 items rated on a five-point Likert-type scale and was applied to Chilean university students.

However, the psychometric properties of some of these instruments validated in other cultural contexts have scarcely been studied [31]. The affective dimension of the Generic Macro-Competence Assessment (AGMA) scale [17] was designed using the University of Concepción model [16] to evaluate the affective dimension of the four generic competences. This is a 10-item scale originally developed in Spanish. Its authors conducted a factorial analysis with a sample of 493 Chilean first-year university students from a random sampling of different academic degrees; the results indicated two factors [17]. The first factor comprised the first seven items and was called Knowing myself, knowing others, integrating positions, and expressing it adequately. The three remaining items were grouped under a factor called practicing the profession for the common good. However, there are no data from the AGMA scale for university students from Spain.

The AGMA instrument allows teachers to more effectively detect the affective dimension related to the development of sustainable behaviors. This implies the integration of knowledge, procedures, and attitudes as a reflection of the acquisition of transversal competences. For such teachers, who are the ones responsible for introducing these transversal contents focused on sustainability, this is a useful tool that allows them to measure the affective aspect of generic competences from a holistic perspective.

The Rasch model has been applied to determine the psychometric properties of the AGMA scale. This model has recently been used in different countries for analyzing the psychometric properties of an instrument. A selection of studies from different countries indicates that it is currently considered the most appropriate for this purpose; these studies are described below. In this sense, the Rasch model and its advantages and importance for effectively analyzing the psychometric properties of reliability and validity of an instrument can be examined in recent works such as Bond et al. [32]. Lönnfjord and Hagquist [33] examined the psychometric properties of a self-efficacy scale in a Swedish population using Rasch analysis.

More recently, Rahayu et al. [34] have validated the scale, also using the Rasch model, in a sample using the Indonesian: What Is Happening in this Class? (WIHIC) questionnaire. Another study [35] used the Rasch model on a Spanish and Colombian sample to demonstrate the factorial structure of the Transition to Adulthood Autonomy Scale (EDATVA), examining the local independence assumption of the variables of this scale. Another study in the same line and year [36] validated the Brazilian scale of Academic Expectations for Higher Education using the Rasch model to demonstrate the scale’s invariance. This model provides a true measure that verified the fitting degree of the scale items [36]; thus, it is considered the most appropriate technique currently available to validate an instrument.

This study aimed to determine the psychometric properties of the affective dimension of the Generic Macro-Competence Assessment scale by Navarro et al. [17], applying the Rasch model to a sample of Spanish university students. The hypotheses were as follows: (H1) The results will support the unidimensionality and scalability of the factors of the affective dimension of the generic competences instrument according to the Rasch measurement model (RMM); and (H2) No differences (DIF) will be found according to gender.

2. Materials and Methods

2.1. Participants

The sample comprised 387 students from the University of Granada (74.9% women) with an average age of 21.24 years (WD = 3.54; range: 17–34). Specifically, 263 were first- and second-year psychology students, while 124 were from the mandatory master’s degree for teachers in secondary and high school education, vocational training, and language teaching in Spain (known as MAES).

2.2. Instrument

The Affective Dimension of the AGMA scale assesses the affective dimension of generic competences in university students [17]. It consists of 10 items rated on a five-point Likert-type scale (not important, not very important, I am indifferent, important, very important). It has adequate psychometric properties.

2.3. Procedure

Some ex-post-facto crosscut design was used in this study. Participants were informed about the study and assured of their anonymity and confidentiality; subsequently, they signed an informed consent form. The average time for application of the instrument was 7 min. This work is part of the Educational Innovation Project titled “New proposals for the integration of competences through transversal training evaluation and ICT” (PIE19-159, Call 2019–2021, UMA, University of Granada, University of Malaga) [1].

The study design was approved by the Ethics Committee of the University of Granada (863/CEIH/2019).

2.4. Analysis

A Rasch analysis was carried out by applying the partial credit model [37] using the RUMM2020 software. An analysis was conducted to determine the construct validity and reliability of the scale. The Rasch model transforms ordinal raw scores into an interval unit of measurement called a logit [38]. This allows a calibration of the items in a scale according to the ease or difficulty with which they are endorsed by respondents. The sample size of the study guaranteed the stability of item location estimation with 99% confidence [39]. The analysis determined unidimensionality, overall fit to the Rasch model, individual item fit, targeting of participants, functioning of response categories, and differential item functioning (DIF). Extended information regarding the protocol for Rasch analysis performance can be found elsewhere [40].

3. Results

Most of the participants were first- and second-year psychology students, mostly women, and with an average age of 21.4 years. Most of these young university students can be considered to be at the developmental stage of late adolescence with the cognitive, physical, social, cultural, and affective factors and changes associated with this process of personal identity construction. The initial analysis of responses to the 10 items showed a lack of adjustment of the scale to the Rasch model using a significant chi-square of the item-trait interaction (χ2 = 79.67; p = 0.004).

The 10 items that constitute the so-called affective dimension are as follows: Listen in a respectful, attentive, and empathic way to the opinions or ideas of others; effectively manage elements of verbal and non-verbal communication to be able to express myself and understand others in the best possible way; consider the contribution of other areas of knowledge when addressing a problem; evaluate my way of thinking about things; contrast my ideas with the ideas of others and the contributions from other sources, to generate a more complete response; express my ideas and convictions clearly and without attacking others; have values that contribute to the justice, equity, and dignity of the person as an axis in my professional actions; reconcile the satisfaction of my personal needs with the satisfaction of the needs of others; consider the consequences of my professional actions in the short, mid-, and long term; and practice my professional role responsibly to contribute to society and my development.

All items showed disordered response category thresholds. This indicated that participants could not consistently adjust their responses to a gradient with five response options. Therefore, the order of thresholds was tested by merging adjacent response categories. The best solution was to reduce the response options from five to three (0 = not important, 1 = important, 2 = very important). The reduction of categories did not improve the adjustment to the model, but allowed us to identify that item 1 (listening in a respectful, attentive, and empathic way to the opinions or ideas of others) did not fit the model (p = 0.002) due to a high negative residual (−3.156). In such cases, deleting the item is the recommended solution. The scale with the remaining nine items showed an adequate adjustment to the Rasch model (χ2 = 61.46; p = 0.052).

Further, we found that the residuals for the items and people fit the acceptable values of a mean close to 0 and a standard deviation close to 1 [41]. The unidimensionality of the 9-item scale was examined using the procedure considered the most robust and demanding, as described by [40]. The results indicated that only 1.54% of the significant t-tests fell outside the 95% confidence interval. Therefore, the latent construct of the scale was unidimensional.

We then determined if any of the items showed differential item functioning (DIF) for any of the three factors in the sample: age (more/less than 20 years), gender (male vs. female), and education level (undergraduate vs. graduate).

In the Rasch model, reliability is measured by the Person Separation Index (PSI). A PSI of 0.70 indicated that the scale is useful for classifying people into two groups in terms of their construct level; that is, low or high affective dimension of generic macro-competences in the case of the AGMA scale.

Finally, as one of the main practical contributions to validate the construct, the Rasch analysis places items in hierarchical order, showing which features of the affective dimension are the easiest and most difficult for the university students to acquire.

Table 1. Hierarchy of items in the AGMA scale. Items are ordered by likelihood of participant endorsement.

	Location	Standard Error	Fit Residual	p of χ2
10. Practice my professional role responsibly to contribute to society and my development	−2.738	0.541	−1.202	0.675
7. Have values that contribute to the justice, equity, and dignity of the person as an axis in my professional actions	−0.623	0.135	−0.579	0.372
2. Effectively manage elements of verbal and non-verbal communication to be able to express myself and understand others in the best possible way	−0.178	0.127	−0.013	0.404
6. Express my ideas and convictions clearly and without attacking others	0.163	0.117	−1.41	0.193
4. Evaluate my way of thinking about things	0.164	0.097	−0.197	0.049
9. Consider the consequences of my professional actions in the short, mid-, and long term	0.313	0.111	−0.16	0.356
5. Contrast my ideas with the ideas of others and the contributions from other sources, to generate a more complete response	0.405	0.109	−1.501	0.062
3. Consider the contribution of other areas of knowledge when addressing a problem	0.413	0.097	−0.734	0.349
8. Reconcile the satisfaction of my personal needs with the satisfaction of the needs of others	2.08	0.093	−0.525	0.168
1. Listen in a respectful, attentive, and empathic way to the opinions or ideas of others				Misfit

shows that “practice my professional role responsibly to contribute to society and my development” (item 10) was the easiest item to endorse, whereas “reconcile the satisfaction of my personal needs with the satisfaction of the needs of others” (item 8) was the most difficult.

4. Discussion

The aim of this study was to determine the psychometric properties of the affective dimension of the AGMA scale on a sample of Spanish university students. Rasch analysis identified just one dimension in the latent construct that measures the scale. This finding does not confirm the two factors resulting from the analysis of the sample of Chilean students [17]. The unidimensionality of the scale solves a problem detected in the factorial solution found in the Chilean sample. In this sample, the factor knowing myself, knowing others, integrating positions, and expressing adequately, contained items that indicated a clear relationship with factor 2. For example, due to its content, it would be expected that item 7 (have values that contribute to the justice, equity, and dignity of the person as an axis in my professional actions) would be grouped in the second factor practice the profession for the common good. The possibility that there are items that, due to their content, could be attributed to any of the 2 factors indicates that they are not different dimensions, and therefore the unidimensionality observed in the Rasch analysis is a result consistent with the content of the items. Regarding the elimination of item 1 (listen with respect, attention, and empathy to the opinions or ideas of others), the high negative value of the residual indicates that the content of this item is already sufficiently measured by the rest of the items on the scale. The finding of redundant items makes it possible to shorten the measurement instruments without losing the capacity for evaluation.

Regarding the reliability under the Rasch model, the AGMA scale has the capacity to classify university students into two groups in terms of the dimension of generic competences. The brevity of the AGMA scale and its ability to classify students into groups with high or low dimensions make it a useful screening instrument. However, through the Rasch analysis, we identified methods for improving the scale. First, there are several gaps between the different difficulties of the competences to be acquired, which could be completed with new items. The first gap lies between items 10 and 7 (considered the easiest to acquire). The second gap is at the opposite pole, between items 3 and 8 (considered the most difficult).

The analysis of the hierarchy is enriching for teachers, as it identifies which aspects are easier for learners to acquire (exercise responsibility to contribute to society and themselves) as opposed to those that are more difficult and therefore require more resources (reconcile the satisfaction of one’s own needs with those of others).

The second potential improvement in the scale is to reduce the current five response options to three (four at most), as it is impossible for people to systematically discriminate between more than four ordered thresholds in most constructs [42].

The psychometric analysis of the AGMA scale shows its factorial invariance by means of two techniques recently used to obtain such evidence [36], such as a more classical technique like factor analysis and another that is currently considered suitable, such as analysis based on the Rasch model. The psychometric analysis supports the usefulness of this scale, which proves to be an adequate instrument for evaluating the affective dimension of transversal competences in university students; it has been applied and is consistent with theoretical models that have been consolidated in studies conducted at universities, such as that at the University of Concepción (UDEC) by Navarro et al. [17].

Specifically, in this work, we decided to follow this consolidated educational model of transversal training from the University of Concepción, Chile [17,22]. We prefer this approach since our own study has benefited from using Navarro et al.’s [17] studies for the theoretical or conceptual background, in which an operative definition of a complex construct such as the affective dimension of transversal competences is already provided. From the Strategic Plans of Spanish Universities and considering the objectives of the 2030 Agenda for Sustainable Development and the imperative need for sustainability education from university classrooms, there is an emerging line of research and development of training actions in coherence; specifically, this line includes the affective dimension of competence acquisition that aims to promote, for example, critical thinking and management based on social responsibility. To this end, [17,22] can be more productive and better tailored for university-based training of students’ generic competences and the study and development of their affective dimension, such as communicative competence, professional responsibility, environmental issues, and sustainability.

It should be noted that there are other studies that focus on Spanish students’ perceptions of generic skills (planning skills, higher cognitive skills, quality management and innovation, expression and communication, knowledge society, teamwork, etc.). For instance, in a recent study, Sáez-López et al. [43] evaluated the perception of competences in terms of their importance and skill, as well as other qualitative assessments of competences in a student sample (age 18–30) from two Spanish universities. They found differences between the importance and attitudes afforded to competences and their degree of skill or acquisition in the curriculum; further, the existence of deficiencies in the acquisition and development of competences was revealed. Specifically, the competences evaluated in this study [43] were those of management and planning, higher cognitive skills, quality management and innovation, expression and communication, and knowledge society and teamwork.

Spanish universities conduct ongoing comprehensive research on university students’ generic skills development, and use different conceptual backgrounds, methodologies, and approaches. The present study differs in principle from other national studies on the topic, which is important within national, regional, and international research dimensions. This is a field that has not yet been fully explored in the literature, particularly in the university context. Moreover, it is an instrument for evaluating a growing dimension that is necessary to promote university-level attitudes and behaviors for sustainable development, as well as management using increasingly responsible practices based on inclusion, quality of life, and social responsibility.

Regarding the applicability of the study, the results may be useful in terms of justifying the design of intervention programs in the development of cross-cutting issues in which affective factors are paramount. This study aimed to analyze Navarro et al.’s [17] instrument, a validated scale used to assess generic competences. Currently, we are attempting to build instruments for a more precise evaluation of the acquisition of competences in the university [28,44]. We believe that our study makes a significant contribution to the literature because it aims at identifying the gaps that can emerge when attempting to adopt instruments in various settings and finding ways to fill these gaps.

By allowing the adaptation of an instrument, the affective dimension in university students can be evaluated in a similar way across universities; these results are fundamental for the application of the competences required to promote favorable attitudes for the development of well-being and social responsibility in coherence with [17,18,45]. Specifically, Spanish universities are paying increasing attention to the development of training actions for sustainability, such as courses on waste management for teachers at the University of Granada. In fact, an increasing number of recent studies (e.g., [46]) have analyzed the attitudes, knowledge, and behaviors of university students regarding sustainable development education, which aims to promote strategies in favor of sustainability in the university setting. Such results can also be considered relevant for the acquisition and development of such personal attitudes that are part of the so-called humanistic–personal dimension of the teaching and education profession. This has been raised in competences teaching/learning models such as that of López [10] and Morales [19].

Regarding the limitations, although the sample is large, focusing on psychology and MAES students may only reflect a specific aspect of reality that could vary depending on the major. It would be interesting for future studies to increase the variability of the majors in the sample. Further research could incorporate a wider range of data regarding the students’ areas of training, as the concept of original study is to explore the generic competences development of students in three study areas (see, [17]). In this sense, the sample should be extended to include all other scientific areas, as well as using a longitudinal design that spans different academic years, centers, and degrees. When the instrument is widely applied to universities and students of all ages, and not just those closer to late adolescence, the generalizability of the results may increase. Likewise, qualitative data are available from the proposal of cases designed to resolve environmental and ethical problems and from the debate generated in class groups regarding students’ knowledge and attitudes toward what they understand by key constructs (such as communication, critical thinking, creativity, sustainable development, common good, ethical and social responsibility). The data obtained solely with this self-report measure could be complemented in further studies. To do so, in-depth interviews should be conducted with students, teachers, and other educational agents that would yield in-depth information on this dimension of transversal competences. However, this instrument is useful for assessing the dimension of transversal competences in university students, which is captured by few extant instruments as it is a construct that is difficult to operationalize.

5. Conclusions

This study’s findings highlight the structure of the AGMA scale based on the Rasch model. Through this model, the existence of a single dimension explaining the participants’ responses was discovered. The use of a 5-point Likert-type scale should be reconsidered, given that the participants in this study were unable to adjust their response consistently using this scale. Further research is warranted to analyze the structure of the questionnaire and its response modality.

In conclusion, the psychometric properties of the AGMA scale indicate that it is an easy-to-apply instrument for measuring the affective dimension of generic competences in undergraduate and graduate students in Spain.

These findings may be used to assess attitudes toward the affective dimension of transversal competences in university students, which is necessary, for example, to promote more sustainable attitudes and behaviors. The availability of this assessment instrument with adequate psychometric properties (reliability and validity) and easy application is considered useful for the university-level development of training and research actions, such as for the development of communicative competences, improvement of emotional management, and the promotion of attitudes and behaviors for sustainability, attention to diversity, and social inclusion.

Applying this instrument in the university environment could help to improve educational practices in a transversal way that improves self-knowledge and knowledge of others and the environment. This, in turn, would improve well-being and empathy and is necessary for the training of skills related to social responsibility, more effective communication, and the development of professional behaviors for the common good and a more inclusive and sustainable world. The evaluation of the affective dimension of transversal skills makes this a useful tool in an educational and social context in which the universities are undertaking joint efforts for achieving the objectives for sustainable development and the 2030 agenda.

The Organisation for Economic Co-operation and Development [3] has highlighted the need to promote this affective dimension of the development of personal transversal competences in current-enrolled students. These competences include the development of socioemotional skills like empathy, resilience, and ethical values and socio-professional commitments that simultaneously contribute to social and community well-being. For this purpose, we require scales that allow a more accurate and reliable assessment from the university context and ones that will help in developing training programs to foster this affective dimension of transversal competences.