Empirical Assessment of the Long-Term Impact of an Embedded Systems Programming Requalification Programme

Abstract

Digital transformation has increased the demand for skilled Information Technology (IT) professionals, to an extent that universities cannot satisfy it with newly graduated students. Furthermore, the economical downturn has created difficulties and scarcity of opportunities in other areas of activity. This combination of factors led to the need to consider requalification programmes that enable individuals with diverse specialisations and backgrounds to realign their careers to the IT area. This has led to the creation of many coding bootcamps, providing intensive full-time courses focused on unemployed people or unhappy with their jobs, and individuals seeking a career change. A multidisciplinary group of higher education teachers, in collaboration with several industry stakeholders, have designed and promoted an embedded systems programming course, using an intensive project-based learning approach comprising 6 months of daylong classes and a 9 months internship. Having finished two editions of the programme, a questionnaire was presented to the students that finished successfully, in order to evaluate the long-term benefits to graduates and companies. This paper presents a brief discussion of the programme organisation and pedagogical methodologies, as well as the results of the questionnaire, conducted following a Goal–Question–Metric (GQM) approach. The results demonstrate very positive outcomes, both for graduates and companies.

1. Introduction

The ongoing digital transformation, enabled by technological advances such as Big Data and the Internet of Things (IoT), has been a major trend with significant crosscutting impact on cross areas of society, such as health care, industry and services. This has promoted an ever growing need for Information Technology (IT) professionals. On the other hand, the effects of digitisation itself combined with a challenging economical conjuncture has led to a reduced demand for professionals in some other areas of expertise. It is therefore highly desirable that opportunities exist allowing the professional requalification of individuals with low IT expertise, thereby promoting the chance for a successful career realignment or advancement. This is advantageous not only for these individuals themselves, but also for society at large, because it helps to address the shortage of IT-skilled labour. Traditional college courses have not been able to comply with this high demand of IT skilled professionals, forcing industry to search for these skills using non-traditional approaches, namely by retraining professional from other areas. In fact, from the IT industry perspective, short courses, such as coding boot camps, are quite effective in inducing software development skills, both hard and soft, compared to traditional 4-year college courses [1,2].

Although there is a proliferation of short courses and bootcamps focusing in IT skill development, with varying degrees of success [3,4,5], these courses are often concerned with web, mobile or desktop development, whereas the specific area of embedded systems programming is not usually addressed, in spite of the increasing pervasiveness of cyberphysical systems, of which embedded systems are a major component. In fact, this area requires highly specialised hardware and software development skills, which can be challenging to acquire in the typically short time frame of these courses. In [6,7], the authors describe embedded systems related initiatives and the careful approaches they had to follow when teaching this specific topic.

To meet this demand, a professional requalification course, the Apostar em TI (AeT) programme [8,9], was jointly created by a higher education (HE) institution (Coimbra Polytechnic-ISEC) and several industry partners. The course is highly focused on the skills and competences identified as priority by the industry partners, and is targeted at highly motivated, full-time students with previous HE experience (i.e., have at least been enrolled in an undergraduate programme). No prior IT background was expected or required.

Having now successfully completed two editions of the programme, sufficient time has elapsed to allow assessment of benefits of the programme over a long time span. An evaluation of the long-term impact of the course was therefore conducted, focusing on the identification of relevant metrics that characterise the evolution of former students professional career and status. The remainder of this paper is structured as follows: Section 2 provides a description of the programme. Section 3 describes the results of both editions. Section 4 presents the description of the design of the evaluation methodology, results and discussion and Section 5 concludes the paper.

2. The Apostar em TI Programme

The Apostar em TI (AeT) programme addresses two areas of expertise: programming (C language [10], with some notions of software life cycle [11] and C++ [12]) and embedded systems [13] (digital electronics, computer architecture and organisation, interfaces and communication, real-time systems). A complementary internship training guarantees that students achieve a full integration with industrial environment and practices required by the partner companies. In this section, the AeT programme is presented.

2.1. Goals of AeT

The global market for embedded systems has evolved considerably in recent years. This includes the technology and the industries served. With the advent of the IoT and Industrial IoT (IIoT), embedded systems technology has become a central facilitator for the rapidly expanding industries dealing with smart ubiquitous devices and IoT.

Professionals specialised in embedded systems are currently in high demand and despite most of the regional companies in this sector being actively looking for professionals with these specific skills, the offer is unable to fulfil their needs. In fact, this scenario has specifically been pointed out by our industry partners and is a key motivation factor for their participation in this project. On the other hand, many other areas are experiencing a decline in demand due to changes in the global society skill requirements, e.g., civil engineering and chemical engineering [14]. This change creates a large body of professionals looking for career change opportunities and therefore for courses that enable them to obtain the required new skills. Computer related engineering is currently one of the most active areas of recruitment and it is with no surprise that professionals from other areas are identified as looking to change specifically to computer related professions.

In this context, one of the main goals of AeT is to offer an opportunity to candidates looking to change their careers to the IT industry. AeT offers the opportunity to acquire the required skills and support for initial placement in the industry partners. Moreover, AeT aims at providing the industry needs with IT experts by training new professionals with these high demand skills. From the perspective of a candidate, an option to acquire the necessary competences to start a career in IT would be to enrol in a traditional Bachelor or Master course. While this is a definite possibility, it requires a substantial amount of time investment. AeT offers an intensive and focused course with rapid placement in a paid internship. Even though AeT is not considered a Bachelor or Master degree, students are in fact granted a studies diploma from an official organisation and the curriculum is properly evaluated according to the European Credit Transfer and Accumulation System (ECTS), the standard framework for recognising higher education studies nationally and transnationally in Europe. This means that, if former students wish to pursue further studies to consolidate and develop their skills in standard IT courses, their AeT programme diploma can be recognised and considered sufficient demonstration of competence in the corresponding skills, with the practical effect of effectively waiving the necessity of the student retraining in the same topics again. In essence, time spent in AeT will be counted as time spent progressing in a Bachelor’s or Master’s degree in the same area of expertise. In fact, some former AeT students have chosen to enrol in such courses and have already benefited from the ECTS credits transfer system in such a way. What this means is that the AeT programme might be a good option for students wishing to train in the specific area of the programme, but want to be able to enter the work market in a shorter time frame than that which is possible with a more traditional Bachelor’s or Master’s course, while still making progress for that ultimate goal if that is their objective.

2.2. Learning Outcomes

This course aims at training students into understanding C and C++ programming concepts as well as embedded systems usage and implementation mechanisms. Moreover, it aims at developing soft skills in students, such as teamwork, resilience, communication, time management and responsibility.

Specific learning outcomes and required skills have been collaboratively defined by academic and industrial partners. A brief synopsis is provided in this section and further details and breakdown of the course structure may be found in [9].

2.2.1. Embedded System Programming Skills

At the end of the course, students should be able to program a typical “off the shelf” microcontroller with several characteristic peripherals.

Students are trained on C language programming, including all of the normal language structures and constructs. The usage of libraries as well as implementing low-level operations, such as bit manipulation and register level device configuration are introduced throughout the course. Furthermore, this low level coding skills implies training the notions of digital numbering and binary encoding.

Students are also required upon successful completion of the programme to be able to program both simple microcontrollers, such as classics 8051s derivatives, as well as complex ones, such as STMicroelectronics STM32F7 ARM series. They should also understand and use various peripherals such as timers, ADCs and external port pins (e.g., to control LED arrays). Additionally, standard serial communication peripherals such as UARTs, SPI or I2C have been used in a number of small exercises and projects throughout the course.

Students should also understand the object-oriented paradigm and be able to develop programs in C++. Consequently, students study and apply: classes, member functions, operator overloading, memory management, inheritance and polymorphism, and object construction and destruction.

Concerning the basic principles of real-time programming, students study real-time task-oriented development, real-time task scheduling policies and learn to use the RTOS real-time kernel to develop applications using periodic and aperiodic tasks, and also basic synchronisation and communication mechanisms such as queues and mutexes.

Software development and engineering practices, including software analysis and design, testing and version control, are also applied throughout the entire course.

2.2.2. Soft Skill Development

In the IT area, besides the level of education and the technical (or “hard”) skills of the candidates applying for a job, such as computer programming, the industry highly requires non-cognitive (or “soft”) skills. For example, in an inquiry of IT professionals in the mid-Southern USA, McMurtrey et al. [15] show that the most important skills required for entry-level employees were problem solving, critical thinking and team skills. Another research from Kovacs and Davis [16], based in the analysis of digital job posting in Pittsburg, discovered that communication skills were next to programming as the most demanded skills. In a study from Vooren et al. [2], in the Netherlands, based in a discrete choice experiment, the authors figured out that non-cognitive skills, such as listening and teamworking, are in the employers’ preferences for IT-retrainees, affecting their willing to pay them a 46 percent higher wage.

The AeT programme was therefore organised in such a way that students could find an environment similar to a company, so they can exercise and improve not only their technical skills, but also their non-cognitive skills. The school assigned one classroom for exclusive use of this course, meaning that students can stay all day long in classes and are free to manage they schedule and study time, which contributes to improve their self-management soft skills. It also encourages a significant shared environment allowing multiple students to remain on their own, promoting collaboration and competition. Teamwork, with rotating partners, and work presentations to the class, are also required.

2.3. Curricular Structure

The AeT programme is developed in two phases. The first is a lesson-based period where students learn programming in C and C++, embedded systems using 8051 [17] and the ARM-based STM32 [18], and real-time operating systems [19]. The second part consists on an internship in one of the industry partners.

1.

Academic Phase held at ISEC’s premises:

• Duration of twenty weeks, between February and July;
• Lecture of 200 h of theoretical-practical classes by ISEC teachers;
• 300 h tutorial training by teachers and instructors;
• Presentation of workshops from partner companies and other guests;
• Execution of a 3 weeks final embedding system project.

2.

Professional Internship Phase, held in one partner company:

• Duration of 9 months, between September and June;
• Paid professional internship;
• Supervision by ISEC teachers;
• Intermediate and final presentations at ISEC for all participants on the programme about their internship ongoing work and final results.

2.4. Pedagogical Methodology

The pedagogical approach is essentially based on practical training, with exposition based on examples and case studies, and on daily practical work. During the academic phase, students have a weekly average of 12 h of classes and 15 h of tutorial support. The expected workload from students is 36 h per week, summing a total of 720 h of effort, corresponding to 28 ECTS credits.

The lessons and scheduling of the subjects of the course were carefully thought and planned, with the close participation of the partner companies. It was decided to have two subjects being taught at the same time, meaning lessons related with programming and embedded Systems were interleaved during the week. This was aimed at preventing excessive impermeability across topics. Interleaving two topics would allow students to better relate them and think how one could be used with the other.

Each day was organised into three parts:

• During the morning, the lesson was basically theoretical, but always supported with practical examples and demonstrations;
• In the beginning of the afternoon, the first two hours (it could vary from one day to the other) was devoted to exercises for practice;
• Later in the afternoon, exercises for grading were given to the students.

The students were accompanied by one instructor that was present every afternoon, to clarify doubts and help solving impediments. The instructors were recruited PhD students and professionals with proven high skills in programming and embedded systems.

The evaluation of the students was essentially continuous, based on the quality of the work developed and presented. Each assignment had a set of deliverables that were submitted to GitHub Classroom and immediately evaluated, promoting continuous improvement. To encourage engagement, attendance of the students was measured and taken into account for evaluating purposes.

Each student was assigned one teacher as a tutor. This tutor was responsible for a more direct contact with their assigned students, making sure that their specific difficulties with the subjects, or other individual pedagogical requirements were listened to and taken care of.

All necessary devices, instruments and bibliography are provided by ISEC, however, each student was encouraged to have their own laptop.

3. First and Second AeT Editions

At this point, two editions of AeT have been concluded. The first edition of AeT started in February 2018, and the second edition in February 2019. This section describes the candidates and recruitment process, outcomes, placement and employers feedback.

3.1. Candidates and Recruitment Process Analysis

Over the course of the two editions, 236 candidate applications were received and analysed.

The profiles of the candidates was very diverse, as can be seen from

Table 1. Profile of initial candidates to AeT.

Academic Degre			Previous Studies in STEM		Gender		Age
None	BSc	MSc	Yes	No	M	F	<30	30–40	≥40
20%	58%	22%	53%	47%	78%	22%	40%	44%	16%

.

The original area of the candidates was very diverse ranging from engineering degrees to completely unrelated activities such as psychology or social animation. This diversity suggests that initiatives such a AeT not only are relevant to a very broad public, but also that they are needed as an opportunity for people wanting to radically change careers.

Their academic degrees ranged from Master (or even PhD-1 candidate included in the MSc column of Table 1) to Bachelor (Licenciatura, in Portugal) or even without any concluded degree; their areas of study were both in Science, Technology, Engineering and Mathematics (STEM) and non-STEM (e.g., in psychology, management, sports or nursing); they were aged between 21 and 49.

As observed, the majority of the candidates already had an academic degree, although about 20% did not finish their Bachelor. The fact that half come from non-STEM areas is interesting. Almost half of the candidates are aged between 30 and 40, and a few are more than 40 years old. As can be traditionally observed in any STEM-related programme, the majority of the candidates are male. It was also noted that a large percentage of the candidates (35% in the 1st edition and 25% in the 2nd) had a job and nevertheless were applying for this programme, in order to pursue better professional opportunities and a career change.

For the purpose of candidate selection, we resorted to the help of a company specialising in human resources management, with a history of close collaboration with some of our industrial partners. This experience was leveraged for the benefit of the candidate selection process. After an initial period of publicity and marketing, candidate applications were received and the candidate selection process ensued, following these steps:

1.

CV Screening: An initial CV screening eliminated candidates that did not meet the requirements;

2.

Phone Interview: A phone call interview allowed clarification of candidate profile and motivation, allowing further pruning of the candidate pool;

3.

Tests: Selected candidates were invited for a session of psycho-technical tests and team exercises;

4.

Interview: Finally those candidates that passed the tests went to a face-to-face interviews with a HR (Human Resources) specialist and a professor associated with the course.

In these tests and interviews, each candidate capabilities, motivation and expectation were assessed. Candidates could also understand what was expected from them, if they were selected.

Taking a look at the profile of the candidates of both editions in Table 1, can be contrasted with the profile of the accepted candidates (see

Table 2. Profile of accepted candidates.

Academic Degree			Previous Studies in STEM		Gender		Age
No	Bac	MSc	Yes	No	M	F	<30	<40	≥40
16%	63%	21%	63%	37%	79%	21%	50%	39%	11%

). There seems to be little to no significant difference regarding their academic degree and gender, meaning that these factors do not influence meaningfully the probability of a candidate being selected. Additionally, younger candidates look like having a higher probability of being selected.

Furthermore, we have also found that candidates with previous studies in STEM seem to be in an advantage: about 25% of the candidates with STEM background from both editions were selected, as opposed to 16% of the candidates without such background.

3.2. Results from First and Second Editions

We received highly motivated students, that were ready to work hard every day to achieve their goal of getting an internship in one of the partner companies. Some of them even dropped their stable jobs for a career change. Because it was necessary to maintain a high pace, to make sure students were capable of acquiring all required skills, instructors commitment and close proximity was kept constant throughout the programme and students performance and motivation closely monitored.

The aggregated results for both editions are presented in Figure 1. An approval rate of 76% was achieved, which can be evaluated as good, considering that the majority of the students had not had any previous contact with programming. Nevertheless, better results were expected due to the nature of the programme and the motivation of the students.

• The introduction of new topics and its consolidation must be carefully balanced, so the students are able to assimilate them (through daily exercises) without burning out;
• Periodic projects help consolidating the subjects, but need time;
• A final project motivates students, and allows them to have better results. However, they can work around difficulties, avoiding challenging subjects. The final project cannot, thus, replace the periodic projects.

We found that the topic most likely to present difficult challenges to non-STEM students is programming. These difficulties tend to manifest early on and can negatively impact the student’s long-term success. This prompted us to reinforce candidate selection procedures aiming to more specifically assess these types of difficulties (e.g., resorting to online gamified programming quizzes and activities, targeted to non-technical individuals) and also providing early and targeted tutorial support to these students to better help them overcome these difficulties from an early stage of the course.

Some of the students that failed the programme abandoned it before its conclusion. Two were not able to keep up with the pace. Another one tried to simultaneously keep a part-time job, in spite of all students being clearly advised against it from as early as the initial interviews, and ultimately failed to cope. Two other students abandoned after prolonged absence due to health issues. The latter three could not have been anticipated in the recruitment process and can be considered fortuitous failures somewhat unrelated to the specifics of the course.

3.3. Placement and Employees Feedback

The 1st edition of AeT started with 11 industrial partners, that committed to receive at least one of the students in a professional internship context, offering a total of 35 internship proposals. The 2nd edition started with six industrial partners that proposed a total of 22 internships. The partners agreed to internship proposals that were in accordance to the programme contents.

Most of these industrial stakeholders operate in Coimbra but some internship proposals have been presented to Lisbon, Porto, Aveiro, Braga and Castelo Branco, outside the region of Coimbra. The industrial partners played a relevant role in trying to attract students to their internship proposals. At the end of the academic phase, the students had the opportunity to choose their preferred internship, with those with better grades being able to be the first to choose. The aim of this methodology is to make industrial partners pitch the internship trying to attract the better classified students, thereby fostering competition that promotes the improvement of offered internships.

Informal student feedback indicated that the internship pitch was a very important deciding factor, based on criteria such as perspectives of future integration and additional training and support. One of the partners was clearly perceived by students as offering the best conditions with regard to those factors. Conversely, partners that did not engage as actively in this phase were generally not successful in attracting students preference. Partners who were more successful in captivating students to internships invested clearly more effort in captivating the students, by actions such as promoting in visits to the facilities and offering improved financial conditions.

4. Assessment of Long-Term Impact in Professional Career

4.1. Objective and Methodology

Evaluating the effectiveness and usefulness of the programme is of paramount importance to keep the course aligned with the needs of the industry and the needs and expectations of the target students. At the time, having already two editions of the course completed, such an evaluation was deemed necessary, in order to obtain the necessary feedback and identifying the need for corrective action and improvements.

To design this evaluation procedure, we resorted to the Goal–Question–Metric (GQM) approach [20]. GQM establishes a process and guidelines to assist organisations in conducting measurements for feedback and evaluation purposes. It is based on the premise that evaluation activities should not be disassociated from the organisational goals. Because measurement activities can impose a non-trivial burden and workload, GQM aids in determining the fundamental measurement activities that are both necessary and sufficient for assessing the achievement of the stated goals. As a consequence, both the effectiveness and efficiency of the measurement processes is improved. Unnecessary measurement effort is avoided and all relevant data is collected and analysed with a clear purpose that can be traced back to the fundamental organisational goals. GQM has been applied successfully in multiple domains for different purposes, such as business data management, quality assurance, or information system evaluation [21,22].

A specific application of GQM is survey design. In this context, GQM helps clarifying the goals of the survey, defining the questions that should be answered, and defining which metrics can be used to provide those answers. By resorting to GQM, we avoid using improvised and potentially biased ad hoc processes, which improves confidence in the obtained results. It is also worth pointing out that the answers are provided anonymously by former students which are no longer associated with AeT, which further reinforces confidence in the results.

The vision of the AeT organisation was, in creating this programme, to ensure that its students would be able to realign their professional career to the embedded systems or IT area. This goal seeks to improve student’s personal and professional success, and also tries to meet the demand of the local and national industry for IT and embedded system professionals. Embedded development was considered a specific variant of IT skills. By specialising in embedded development, it is expected that students gain sufficient skills to allow them to easily migrate and adjust, by option or necessity, to other specialisations within the realm of IT. Therefore, two goals can be defined as presented in

Table 3. Goals of the survey.

Goal	Description
G1	AeT graduates should achieve, or be able to achieve, a stable professional career in the IT area, either by changing career or by getting a job/better job.
G2	The IT-related industry should benefit from, and value, AeT graduates.

.

To evaluate if these goals are being met, 14 survey questions were developed to determine the achievement of all the goals. The relation of the questions to the goals, and the rationale behind it, is presented in

Table 4. Question synopsis and associated goal.

Question	Synopsis	Purpose	Type	Goals
1	Motivation	Verify if motivation for joining the programme was (1) unemployment/precarious employment, or (2) career change	Multiple choices	G1
2	Prior IT knowledge	Self-assessment of level of technical knowledge before AeT	Likert scale	G1
3	Prior IT profession	Check if before AeT, graduates (1) already worked in the IT area or (2) never worked in the area	Multiple choices	G1
4	Value of Lessons	Self-assessment of level of readiness at start of internship	Likert scale	G1
5	Value of Internship	Self-assessment of the value of the internship at the end of the programme	Likert scale	G1
6	Technical Skills	Self-assessment of technical or hard skills development	Likert scale	G1
7	Personal Skills	Self-assessment of personal or soft skills development	Likert scale	G1
8	Job market access	Assessing job opportunities immediately after graduate finishes internship, and job stability, by declaring if (1) he is still working in the same company where the placement took place, or (2) he started working in this company, but in the meantime left by the company’s decision, or (3) left by their own decision, or (4) was not invited by the company after internship, or (5) opted not to stay in this company after internship	Multiple choices	G2
9	Career access	Assessing short and long term career access, by declaring if (1) the graduate is still working in the IT area, or (2) after placement did not stay working in the area, or (3) after placement started working in the area but in the meantime he left	Multiple choices	G1
10	Long term technical skills	Self-assessment of the technical skills when compared to other professionals (colleagues) in the same workplace	Likert scale	G1, G2
11	Long term personal skills	Self-assessment of the soft skills when compared to other professionals (colleagues) in the same workplace	Likert scale	G1, G2
12	Relative career advancement	Self-assessment of the professional advancement when compared to colleagues in the same workplace which did not participate in the programme. This gives an indication of the contribution of the AeT programme towards a positive professional position of the former students relative to other professionals	Likert scale	G1, G2
13	Career Change	Self-assessment evaluating how significant was the contribution of the programme to obtain or consolidate IT knowledge necessary to securing an industry placement	Likert scale	G1
14	Career Advancement	Self-assessment evaluating how significant was the contribution of the programme as a whole to secure or advance a position in the industry in the IT area	Likert scale	G1

.

A set of 11 metrics associated with the questions (

Table 5. Identified metrics.

Metric	Name	Description	Rationale	Associated Questions
STS	Short Term Success	% of graduates that managed to secure an IT related job position immediately after conclusion	Assess if the graduates acquired the competences and will for an IT-area career	Q8, Q9
LTS	Long Term Success	% of graduates that work in the area of IT after at least one year post-conclusion	Assess if the graduates acquired the competences and will for keeping a long-term career in the IT area	Q8, Q9
JS	Job Stability	% of graduates with long term success that maintained work in the same company for over one year	Assess if the company was able to keep the graduates as their employees after 1 year	Q8
CS	Career Stability	% of graduates with long term success that maintained work in the same area for over one year	Assess if the graduates that started a career in the IT had the capacity or will to maintain a long-term career in the IT area	Q9
TS	Tech Skills	Graduate’s subjective perception of acquired technical (hard) skills after conclusion	Assess if the AeT programme provided the graduates with technical skills	Q6, Q4, Q5
PS	Personal Skills	Graduate’s subjective perception of acquired personal (soft) skills after conclusion	Assess if the AeT programme provided the graduates with the personal skills	Q7, Q4, Q5
CA	Career Advance	% of graduates that advanced in their career, by getting a job, or a better job	Assess if the AeT programme was able to prepare the graduates for getting a long-term job or better job	Q1, Q9
CC	Career Change	% of graduates that successfully changed to a career in IT	Assess if the AeT programme was able to change the graduates’ career to a long-term career in the IT area	Q1, Q9
LTCP	Long-Term Career Perspective	Graduate’s perspective of a long-term career success	Assess if the graduates have the skills, compared to co-workers, for a continuous career progress	Q12, Q10, Q11, Q9
E	Effectiveness	% of graduates with long term success that, before AeT, never worked in the IT area and had very low knowledge in this area	Assess how AeT graduates with low previous knowledge and no working experience are able to reach success	Q2, Q3, Q9
AGI	AeT General Importance	Graduate’s perception on the importance of AeT for their career change or advancement	Assess if AeT graduates recognise the importance of the programme	Q13, Q14, Q1, Q9

) are subsequently developed to analyse answers in a quantitative way. The formulas used for measuring each metric are seen in

Table 6. Metric’s formulas.

Metric	Metric Values from the Questions	Formula Rationale
STS	$\left(Q8\right(1\vee 2\vee 3)\vee Q9(1\vee 3\left)\right)/TA$	Percentage of graduates that after finishing the AeT programme were hired by the company where the placement took place or in any other IT-related company
LTS	$\left(Q8\right(1)\vee Q9(1\left)\right)/TA$	Percentage of graduates that, at least one year after finishing the AeT programme, were still working in the company where the placement took place or in any other IT-related company
JS	$Q8\left(1\right)/\left(Q8\right(1\vee 2\vee 3\left)\right)$	Percentage of graduates that, after finishing the AeT programme, were hired by the company where the placement took place, and kept their job at least for one year
CS	$Q9\left(1\right)/Q9(1\vee 3)$	Percentage of graduates that, after finishing the AeT programme, started working in any IT-related company, and were still working in the area at least one year after
TS	$\left(\right(Q6\times 0.5+Q4\times 0.25+Q5\times 0.25)-1)/6$	Evaluation of the technical competences developed by the graduates through the AeT programme, the full curricular part of the AeT, and the full industrial placement.
PS	$\left(\right(Q7\times 0.5+Q4\times 0.25+Q5\times 0.25)-1)/6$	Evaluation of the personal competences developed by the graduates through the AeT programme, the full curricular part of the AeT, and the full industrial placement
CA	$Q1\left(1\right)\wedge Q9\left(1\right)/Q1\left(1\right)$	Percentage of graduates that chose the programme to get a job or a better job and, after at least one year, were still working in the IT area
CC	$Q1\left(2\right)\wedge Q9\left(1\right)/Q1\left(2\right)$	Percentage of graduates that chose the programme to change career and, after at least one year, were still working in the IT area
LTCP	$\left(\right(Q12\times 0.5+Q10\times 0.25+Q11\times 0.25)\wedge Q9(1)-1)/6$	Relative to co-workers, evaluation of the professional progress, technical skills and personal skills of the graduates that, after at least one year, were still working in the IT area
E	$Q2(1\vee 2\vee 3)\wedge Q3\left(2\right)\wedge Q9\left(1\right)/Q2(1\vee 2\vee 3)\wedge Q3\left(2\right)$	Percentage of graduates with no previous working experience and with low knowledge in the IT area that were still working in this area, at least one year after finishing the programme
AGI	$\left(\right((Q13\wedge Q1(2)\wedge Q9(1\left)\right)\times 0.5+(Q14\wedge Q1(1)\wedge Q9(1\left)\right)\times 0.5)-1)/6$	Evaluation of AeT by the graduates that chose the programme to get a job or a better job, or chose the programme to change career and, at least one year, were still working in the IT area

TA—total answers; Qx—average value of responses to Likert scale question number x; Qx(n)—number of responses where option n was chosen as answer to multiple choice question Qx; Qx(n ∨ m)—number of responses where options n or m were chosen as answer to multiple choice question Qx; Qx(n) ∧ Qy(m)—number of surveys where options n and m were simultaneously chosen as answer to multiple choice questions Qx and Qy, respectively; Qx ∧ Qy(n)—average value of responses to Likert scale question Qx in surveys where answer n was provided to multiple choice question Qy.

.

Finally, the relation between the metrics and the goals was established in

Table 7. Goal and metric association.

Goal	Relevant Metrics	Rationale
Goal 1	STS	High short-term Success indicates that AeT provides access to an IT area career as intended, upon successful completion
	LTS	High long-term Success indicates that AeT graduates are able to achieve a long-term career in the IT area as intended, if they desire to do so
	CS	High career stability indicates that AeT graduates are able to maintain a career in the IT area for long time
	TS and PS	Technical and personal skills indicate AeT graduates’ subjective perception of course usefulness, with hindsight, being very relevant as an overall indicator of success
	CA	High career advance indicates that unemployed or unsatisfied people are able, through AeT, to access a long-term job
	CC	High career change indicates that, through AeT programme, people are able to change career and access a long-term career in the IT area
	LTCP	Subjective perception of AeT graduates that maintain a career in the IT area regarding their capacity for continuous career progress. A high long-term career perspective indicates that AeT graduates have the same or better skills for career progress compared to co-workers
	E	A high effectiveness indicates that previous experience or knowledge in the IT area are not a barrier for a successful career in the IT area
	AGI	The evaluation is of AeT general importance by the graduates that maintain a job in the IT area regarding AeT program’s importance in their career
Goal 2	STS	Short-term success is an indication of employers’ perception of the overall quality and potential of AeT graduates
	LTS	long-term success in a IT-area career is indicative of employers’ satisfaction, as it can be hard for an individual to maintain a long term career in the same area if employers are not generally satisfied with their or her behaviour
	JS	High job stability is indicative of satisfaction of employer with the performance of AeT graduates
	CA	High career advance indicates that employers are able, through AeT, to successfully access a large workforce of unemployed or unsatisfied people
	CC	High career change indicates that employers from the IT area are able, through AeT, to successfully access a large workforce of people from other areas
	LTCP	High long-term career perspective indicates that employers can be confident that AeT graduates are competitive among the company’s employees

, allowing the interpretation of the results.

Once the survey was designed and ready, it was sent to former students of both editions and their responses were collected and analysed.

4.2. Results Analysis

From the 27 AeT students that finished successfully both classes and internship, 21 responded to the questionnaire in Appendix A. An online form (Microsoft Forms) was used, guaranteeing that no personal information was collected, thus making the answers anonymous. Although a larger number of samples would be beneficial, it is worth pointing out that the study must obviously be restricted to the population of former AeT graduates. This means that answers were received from a significant majority of the entire population of interest (78%).

The analysis of all the answers allowed to reach the following conclusions regarding the previously identified metrics.

Table 8. Metric results.

Metric	STS	LTS	JS	CS	TS	PS	CA	CC	LTCP	E	AGI
Resulting Value	90%	81%	94%	89%	74%	67%	70%	82%	58%	83%	89%

captures the results of each metric.

• STS—Short-Term Success. 19 out of the 21 graduates (90%) responded that, right after finishing the AeT programme, they got a job in the company where the internship took place, and/or in the IT area. From these, 16 stayed in the same company of the internship and 3 moved to another company in the IT area, 2 because they did not get an offer from the company, and 1 by their own decision. From the 16 that stayed in the same company, 1 responded that he/she did not stay working in the IT area. Taking into account that the COVID-19 pandemics stroke the AeT 2nd edition students and companies during internship, forcing many companies to stop hiring or reducing personnel, even in the IT area, 90% of short-term success is a very positive indicator of the competences acquired by the AeT graduates for a career in this area, and of the companies perception of the quality and potential of these graduates.
• LT—Long-Term Success. Regarding this metric, 17 of the 21 respondents (81%) stated that they maintained an industry placement in the IT area during at least one year after the end of the programme. This value is frankly positive, suggesting that graduates from AeT have the necessary competences to keep working in the IT industry area, if they will. This value also suggests that the companies are satisfied with the AeT graduates’ behaviour.
• JS—Job Stability. From the 16 AeT graduates that were hired by the company when they finished the programme, 15 (94%) were still working in the same company at least one year after. The only graduate that left the company did it by their own choice. This value is frankly positive, suggesting that the employers are satisfied with the performance of these graduates, giving them enough incentive to stay with the company.
• CS—Career Stability. After graduation, 18 of the respondents started working in the IT area and, one year afterwards, 16 of them (89%) were still working in the area. This result suggests that AeT graduates have the skills to maintain a career in IT, even changing between companies (as 2 of the 16 (13%) did).
• E—Effectiveness. 12 of the respondents admitted a lack of knowledge (assigned themselves a score lower than 4 on a scale 1 to 7) and no working experience in the IT area before participating in the AeT programme. From these, 10 (83%) were able to maintain a job in the IT area for at least one year. The value for this metric is quite high, suggesting that the programme is effective in helping the graduates changing career to the IT area or improving their placement in the IT industry, even when their initial knowledge in the area was low or very low. This result is similar to the one obtained in metric CS (89%), that assesses essentially the same, disregarding previous knowledge. This suggests that a previous STEM/non STEM experience or education does not have much relevance to the program’s long term success.
• TS—Technical Skills. The graduate’s own perception of the value of the AeT programme concerning the technical skills required to work in the IT industry is of 74% (5.44 on a scale of 1 to 7). Although seemingly a bit low at first sight, this may reflect the diverse set of skills required to work in the IT industry, some of which might have not been specifically included in the goals of the AeT (embedded programming). However, it is also worth pointing out that AeT graduates feel confident enough to work on subjects not entirely foreseen nor covered by the programme. Considering this scenario, we believe this metric has a reasonably high value, although it suggests that, even though the current curriculum has been created with direct contribution of industrial partners, further opportunities might exist for improving the alignment with market needs.
• PS—Personal Skills. The graduates’ own and subjective perception of the AeT programme regarding the development of their soft skills necessary to pursue an IT career reached a value of 67% (5 on a scale of 1 to 7). Although this seems a relatively low value, the majority of these graduates already had a working experience (in other areas) and a high maturity, meaning that they already had soft skills regarding autonomy, organisation, ability to work in teams, ability to solve problems, etc. Additionally, they considered they had improved their soft skills after this programme (on average they gave to this metric a value greater than 50%).
• CC—Career Change. From the 11 AeT graduates that joined the programme with the main goal of changing to a new career, 9 (82%) were successful, as one year after finishing the internship they were still working in the area. This result is quite high and suggests that AeT plays a significant role helping people realigning their career to one in the IT area. This is very meaningful to the companies as well since, through these programmes, they gain access to a very significant workforce originating from other areas.
• CA—Career Advancement. 10 of the graduates’ joined this programme with the main goal of getting a job, no matter the area. From these, 8 (80%) were able to maintain a job at least one year after finishing AeT, and 7 (70%) maintained a job in the IT area. One of them stayed in the company where he/she was placed during the programme, but in a non-IT area. The resulting value for this metric is very satisfactory, confirming the added value that this programme brought to its graduates. From the employers’ perspective, these results show that, AeT or similar programmes, bring them the opportunity to access a workforce of unemployed or unsatisfied people with high value.
• LTCP—Long Term Career Perspective. This metric applies to the graduates that, after one year, were still working in the IT area. It captures their own perspective about career advancement relative to co-workers not involved in the AeT programme, meaning that a 50% mark corresponds to neutrality (same opportunities as the other co-workers). The obtained value of 58% means that AeT graduates feel that they have the same or even better skills to advance their career compared to their co-workers. This seems to be a very positive outcome, especially if one considers that AeT graduates have been specifically trained in IT for only a period of 6 months plus 9 months of internship, while their co-workers include professionals specifically trained in IT-related BSc and MsC degrees, which benefit of longer programmes (3 to 5 years, respectively). The employers can thus be confident that AeT graduates are able to compete with their remaining employees for better positions. This means that, although having a very short training in the IT area, AeT graduates have the skills to be as valuable to the company as any other worker.
• AGI–AeT General Importance. A score of 89% (6.34 on a scale of 1 to 7) among the graduates that maintained a job in the IT area for at least one year suggests that, from their own perspective, the AeT programme does have a very relevant and positive impact on their lives. These figures are very similar to both the AeT graduates who came to the programme to change their career or to get a new or better job.

4.3. Goals Analysis

According to the analysis in Section 4.2 and the resulting metric values, the satisfaction of both goals of the AeT programme stated in Section 4.1 has clearly been achieved:

• Goal G1—AeT graduates achieve a career change or a career advancement in the IT area. The relevant metrics, as stated in Table 7 are: STS, LTS, CS, TS, PS, CA, CC, LTCP, E and AGI.
  AeT graduates were highly successful in obtaining an industry placement (STS = 90%) and maintaining a job over a year (LTS = 81%). Considering just the graduates that started working in IT after graduating, and kept working in the same area at least one year after, the stability is also very good (CS = 89%).
  However, When considering the value of the programme for the improvement of the technical skills (TS = 74%) and the soft skills (PS = 67%), the indicator values are also favourable, although more modest than the previous metrics, and an analysis for these lower values is provided below in this section.
  Metrics CA (70%) and CC (82%) directly gauge ratio of graduates that were able do succeed in advancing in the IT career or changing to an IT career, respectively. As the values of these two metrics show, in particular the second, that AeT graduates were successful in these two aspects.
  Graduates exhibit a positive expectation of long term career success when compared to workmates which are not AeT graduates (LTCP = 58%). By having a greater than 50% result in LTCP metrics, graduates are manifesting that they feel more well suited for an IT career that non-graduates, which is a very positive indicator.
  Graduates that never worked in IT before the programme and had little or no knowledge of IT are highly successful in obtaining and maintaining a job in the IT area (E = 83%).
  Finally, the overall graduates’ perception of the value of the program to their career change or advancement is frankly positive (AGI = 89%).
• Goal G2—The industry benefits from, and values, AeT graduates. The relevant metrics to this goal (Table 7) are: STS, LTS, JS, CA, CC, LTCP.
  High indicator of success in obtaining a job after graduation (STS = 90%), and being able to keep working in the company or in the IT area (LTS = 81%) are indicators from the industry point of view about the value of graduates to the company where they work. The high values obtained for these two metrics strongly suggest that the AeT programme is aligned with the industry needs. The high ratio of graduates that maintain the job in the same company over a period of at least one year (JS = 94%) is an even more clear indicator of the value of the graduates to the industry and of the alignment of the program with the industry needs.
  The graduates’ success in changing career to the IT area (CC = 82%) means that the industry has access to new professionals and a renewed workforce. The graduates’ success in advancing their career (CC = 70%) also confirms that IT employers have access to a larger skilled workforce.
  The graduates’ confidence in long term career success (LTCP = 58%) gives confidence to employers that the AeT graduates are motivated and competent professionals that are interested in pursuing an IT career, and thus supplying the demands of the industry.

Even though all performance indicators were by far and large positive, it is nevertheless possible to identify aspects and issues where potential for improvement may exist:

• Impact for career change and career advancement—AeT was deemed useful for all students, but comparatively less so for those who decided to enrol on this programme primarily to get a job or a better job, than those that wanted to enter the IT work area. This can be explained, on the one hand, by the lower motivation or interest of those students when they entered this particular programme or, on the other hand, because they already demonstrated some inability to get a satisfying job or even to get a job.
  Maybe a way to improve these metrics could be by recruiting more motivated people in the IT area, or motivating them through the programme, by, e.g., visiting the companies; having industry experts proposing real challenges; or being more supported during the industrial placement.
• Personal Skills—The impact of the course on student’s soft skills was considered satisfactory but relatively modest. We can hypothesise several factors that may contribute to this, such as:

  -

  Most of AeTs students have formerly worked on a professional capacity in some area. It is therefore perfectly conceivable that their soft skills have already developed to a point of maturity where the additional contribution of AeT’s activities does not have a perceivable or significant impact.

  -

  The impact of AeTs group activities and projects on students soft skills is modest because they are not specifically designed nor in sufficient number to increment these capabilities, but to develop technical skills instead.

  Regardless, it appears that it is of no consequence to dwell on this. Considering all the other indicators, we can confidently state that graduates are not, for the most part, lacking in the soft skills actually required to ensure the successful establishment and pursuance of professional activity in the area. Nevertheless, potential improvements can be considered, such as group and anonymous peer-feedback to provide record, more visibility and improved self-perception of student’s soft skills.

5. Conclusions

This paper presents the results of an embedded system training and requalification programme. The AeT aims at reconverting people originating from different areas into professionals in the IT area, specifically embedded systems developers. The AeT initiative has enabled a number of career changes and advancement on individuals that applied to the programme with different motivations and a variety of skills.

Having passed more than 1 year after the end of 2nd edition of the programme, it is now possible to obtain information about the AeT graduates working experience and engagement with their employers’ objectives. The programme usefulness and effectiveness was evaluated by inquiring the graduates on their own opinion about the fulfilment of their initial expectations and on their current career position. This evaluation was undertaken using the Goal–Question–Metric (GQM) methodology to define the goals, a set of related questions to ask the graduates, and a set of metrics associated with the questions to extract information from the answers.

The results show that the majority of the AeT graduates were able to access and maintain an IT related job. Moreover, most AeT graduates were hired after their internship, thus suggesting employers satisfaction with the programme. The survey answers and its related metrics suggest that AeT was useful both in aiding people realigning their career into the IT area, and also helping to meet the industry demands by supplying skilful IT professionals.

As future work, we will seek to improve the AeT programme by experimenting with approaches for improving the development and evaluation of student’s soft skills. Another interesting area of development is to create mechanisms that allow continuous assessment of student professional progression. This study offers interesting insights but it informs only on the situation of students at the moment the survey is conducted. It would be interesting to devise automated mechanisms that allow live ongoing tracking the students professional careers by inspection of publicly available information on social media sites focused on professional networking such as LinkedIn. This would allow important progression indicators to be made continuously available.