Experiences and Lessons Learnt from the Evaluation of ICT Tools for and with Migrants

Abstract

As the number of migrants arriving in Europe increases, host societies face the challenge of supporting their smooth integration, respecting their needs and recognizing their competencies. A key element for their new life is their integration in the labour market. This paper presents a platform which offers a set of tools that has been developed to support migrants to find their way into the labour market in EU countries. This set includes tools for skill assessment, artificial intelligence tools providing recommendations for jobs that match their personal skills and needs, tools for suggesting training paths and options to empower their candidacy, and easy-to-use tools for creating their CV/portfolio. We focus on the evaluation of this platform in real life settings in two countries (Greece and Spain), which is part of our co-creation approach. The results are very promising and show the satisfaction of the migrants. Additionally, they provide valuable insights for both those that develop such tools and those that can use them in their work/activities with the migrants (such as the non-governmental organisations and host authorities).

1. Introduction

While the numbers of refugees and migrants keeps increasing (Eurostat 2021), their social inclusion remains a challenge. When referring to migrants, refugees, and asylum seekers, in essence we refer to displaced people (i.e., people residing in or transiting through a host country) who are found in this country without any prior arrangement for employment or studies. Societal inclusion goes hand-in-hand with the integration in the labour market, which comes with certain hurdles such as the language barrier and the difficulty in assessing and certifying their skills. Although displaced people tend to use social media and online platforms (such as Facebook and google applications) and many attempts have led to the development of online platforms aiming at supporting their integration, all these only address part of the journey from their arrival in a foreign country to their inclusion in the labour market and in the host society (Leligou et al. 2020).

A novel platform that targets the needs of displaced people in EU, and at the same time supports authorities, non-governmental organisations (NGOs), and companies in designing and delivering services to them has been designed and developed (Leligou et al. 2021). An initial indication that ICT tools are appropriate for reaching out to displaced people was the fact that digital tools are a communication means widely adopted by EU member states, local authorities, and NGOs to support migrants’ inclusion. Focusing on displaced people, the presented platform aspires to become the single point of reference for them across EU since the paradigm is applicable to all countries and cases. It offers rich functionality and multiple features: the skill assessment tool, the provisioning of suggestions/recommendations for both jobs and training, based on the already possessed skills and the step-by-step creation of their CV/portfolio, which can be used for job applications. These tools are anticipated to contribute in the familiarisation of the users with the European labour market and job search processes and in the promotion of an increasingly independent use of the job search tools from the displaced people side. Artificial intelligence and data analytics technologies were adopted to deliver an enhanced experience to the migrants.

The developed set of tools has been piloted in two different countries (Greece and Spain) with multiple groups of displaced people speaking European or non-European languages, namely Farsi and Arabic, which are among the popular non-European languages (other popular non-EU languages are Hindi, Urdu, Bengali, and Pashtu). Thorough feedback was collected from the end-users regarding ten different metrics for each of the provided services. The results are very promising and show that ICT tools are well accepted and appreciated the target group.

At this early point, it is worth presenting our methodological approach for the design of this platform in which we adopted the co-design process. Namely, as it was difficult to have the displaced people conceiving and describing the platform that would fit their needs, in the first phase, we opted for including “mediators” in the design of the platform, anticipating that they reflect the users’ perspectives. The “mediators” are the NGO representatives or employees that work with the displaced people to help them find a job. In other words, they are the people that talk with the displaced people, try to capture their needs, and understand what they can do and help them find an appropriate job. Once we designed the platform with the mediators from three countries (Spain, Greece, and Belgium), we developed the platform and piloted its Proof-of-Concept version with real users. This piloting took place in the premises of NGOs under the guidance of the mediators. In this phase, the user feedback, comments, and suggestions were captured to drive the development of the 2nd version of the platform, which was then piloted again with real users. It is the results of the piloting of the 2nd version of the platform that we focus on in this paper. The collected feedback will trigger another cycle of design-development-assessment prior to the platform’s commercialisation.

In this paper, we briefly describe the developed set of tools, and we focus on the evaluation and the collected results from the piloting activities. We also report the experiences from the moderators of the pilots which worked with the migrants. Sharing this experience is of high value (a) for all people designing and implementing solutions (providing them with valuable insights) and (b) for people working with the migrants so that they have more tools in their quiver and select the appropriate means to achieve their goals.

2. State-of-the-Art Review

In this section, we first discuss about existing platforms targeting migrants and refugees and then present the state-of-the-art of evaluation aspects on which we base our evaluation methodology.

Recognising the needs outlined in the introduction, authorities (local or national), non-governmental organizations, and consortia comprising a mixture of organizations have attempted to design and develop web-enabled platforms to support migrants with information and deliver concrete and valuable services to them. A representative list of such platforms is presented in (Leligou et al. 2020). The number of available platforms shows that the host societies have recognized online platforms as an easy and efficient way to reach migrants. Even though information provisioning is the straightforward reaction of the host authorities, it comes with two challenges, namely the high number of topics to be covered and the need for translation in many foreign languages (which cannot be delegated to automated translation tools for these languages as these offer low quality results). Additionally, while tools for skill assessment services exist in most European countries, these rely on machine learning algorithms which have been trained using datasets from European populations, neglecting the intricacies of other populations with different understandings, attitudes, and behavioural norms. Another difficulty is that while migrants would find a job matching their skills after they receive some training, the provisioning of structured information has still been an issue, since, at the moment, there are no tools that offer personalized recommendations for training. Such a recommender would significantly facilitate training search and the identification of the appropriate training that would boost the chance of finding a job position. Another gap today is that no platform provides recommendations for relocation taking into account the needs of the migrants (e.g., regular access to health infrastructure when a migrant has a child that suffers from some chronic disease).

With respect to the evaluation methodology, emphasis was placed on two issues which were considered important: (a) on the appropriate sampling of the users that would test the developed platform, and (b) on the definition of the method (questionnaires, focus groups, interviews, etc.) that would be adopted. In the literature, a sample is defined as a portion of a population or universe (Tailor 2005). There are two main categories of sampling methods: probability and non-probability sampling. On the one hand, the former refers to a sample whereby each participant has the same known probability of being randomly selected (Lavrakas 2008; Henry 1990). On the other hand, in non-probability sampling, randomization is not required when selecting participants from the population. Instead, they are selected based on the research scope as well as the need for promptness in data collection that the study may entail (Etikan et al. 2016; Lavrakas 2008). Scholarship identifies four types of non-probability sampling: convenience sampling, quota sampling, purposive sampling and snowball/chain-referral sampling (Tansey 2007). Convenience sampling refers to the sample that is within the immediate available reach of the researcher without any requirement for specific characteristics. Quota sampling on the other hand is characterized by specific proportions of the sample’s subgroups, which are relevant to the proportions of the entire population. A good example here is the demographic of ethnicity, whereby the quota sampling method would include ethnic groups in similar proportions as to the ethnic proportions of the entire population. In addition, the method of purposive sampling enables the researcher to identify and judge which participants serve the purpose of the research. Finally, the last non-probability sampling method is snowball sampling, which refers to populations where the identities of the people from the population cannot be easily traced (Tansey 2007). In the NADINE project, non-probability sampling was chosen, allowing the analysis to focus on specific steps in the development process of the platform. With respect to the way of collecting the feedback, we considered focus groups, questionnaires, and interviews. To quantify the received feedback, we decided to deploy questionnaires, while the NGOs (which acted as intermediaries for reaching the migrants) had the opportunity to engage them in informal discussions to capture their feelings and attitudes.

3. The NADINE ICT Tools for Migrants

The presented platform provides a set of tools as described in (Leligou et al. 2021). Starting from the simpler tools, it offers information relevant to legal, health and education topics, which are the most desired topics for migrants. In order to achieve this successfully, and to include as much information as possible, this platform fetches information and useful content mainly from public sources with the support of (crawling) tools and mechanisms. In addition, it supports authorized entities (i.e., local authorities, NGOs, companies) to upload such relevant information. To facilitate the search of the appropriate piece of information, the proposed platform integrates a conversational interface which is trained (adopting artificial intelligence algorithms) and “speaks” many different languages including non-European ones.

Additional functionality includes tools for skill assessment (both hard and soft). Skill assessment algorithms refer to a group of algorithms able to assess a person’s skill and quantify its ability to perform it. Many methodologies exist in the literature that measure different skills and competencies of people mainly in the context of education. Neural networks have been recently used in the context of assessing surgical doctors’ skills during manipulation of specialized operating tools. Regarding digital tools (Tountopoulou et al. 2021a), many exist that calculate performance in a normative way and solely based on the score of a specific exercise, while for more laborious tasks this is impossible without the subjective scoring of an expert. To conduct the linguistic skill assessment, the presented platform is directly connected to a mobile application, which offers tests enabling linguistic skill assessment (e.g., with respect to the level of knowledge of a European language). Adopting novel voice processing techniques, the communication competencies of the migrants can be assessed for the most widely used languages in the EU. With respect to the hard and soft skills, we employ neural networks algorithms, which are currently trained under different contexts to combine test results, action recognition, and speech recognition to an overall skill assessment. A first set of questionnaires is offered in multiple (currently in five) languages and thirty-five such skills are quantified reflecting cognitive ability, numeracy, and communication among other factors (Tountopoulou et al. 2020, 2021b). Once a migrant has gone through the questionnaire, his/her skill vector is available and can be included in any document (e.g., CV that is automatically created by the platform) he/she intends to prepare for accompanying a job application.

The proposed platform provides personalized recommendations to the migrants regarding appropriate job sectors based on the skill vector values (that are calculated by the skill assessment tools). So, instead of leaving the migrant to look for any job he/she could apply for, the proposed platform defines job categories that match his/her skills and lists specific job vacancies belonging to these job categories. This is anticipated to make finding a job significantly easier and simpler. At the same time, the host authorities or NGOs can use these tools to identify the skill-job matching and possibly recommend migrants to relocate to places where their skills are mostly needed. A win-win approach is thus implemented for both the migrants (more job offers) and the host societies (better inclusion and necessary skills for employers). This recommendation exploits the outcomes of well-accepted publicly available reports that provide the matching between job categories and personal skills.

Furthermore, the platform provides a list of the quality-of-life indices for each city and also support city ranking according to these indices. This allows the authorities to make decisions about potential sectors of improvements and the displaced people to make informed decisions based on their priorities (e.g., health over education or climate). To do so, it takes into account the user quality of life preferences as well as the quality-of-life categories ranking of the city (Num 2021).

Additionally, the platform recommends training to the migrants that would help them improve their skills to enhance their employability. The trainings include, first and foremost, language training, as well as training targeting other skills which are in demand. In this direction, on the one hand, the platform identifies the skill(s) to be improved. On the other hand, the trainings announced or offered through the platform are enriched with metadata. More precisely, the skill that each training aims at improving is used as metadata during the search. Based on this information, the matching is performed and presented/proposed to the migrant.

Apart from the skill assessment and improvement, the platform guides the migrant through an easy-to-follow wizard, enriched with video instructions to prepare his/her portfolio. The difference between preparing a portfolio vs. a curriculum vitae is that in the portfolio the applicant can provide indications and (informal) evidence of skills and competencies while in a curriculum vitae all the mentioned statements should be justified by an accompanying official document. This provides the migrants with the possibility to claim competences and support them through photos or video presenting their achievements in the cases that these cannot be justified/certified by an official documentation.

Finally, an AI based chatbot has been created to provide administrative information in an easy, user-friendly, and automatic way for the migrants. The chat-bot has been trained to be able to conduct chit chat and answer administrative questions regarding education, legislation, and employment (in the current version), while other categories can easily be added in the future.

4. The Evaluation Strategy

For the evaluation of the NADINE platform, the non-probability sampling method was chosen. This is due to the fact that the research scope of the project was not targeted to obtain results which are representative of the entire population but to test specific steps in the development process of the NADINE platform. Hence, a perfectly random sample was not required for the purpose of NADINE. Thus, the sampling method was purposive sampling. The main reason for selecting this sampling technique was that our interest was to conduct the pilots with participants who are asylum seekers, recognised refugees or vulnerable migrants. To clarify the differences of the user groups, asylum seekers in the EU context is defined as “a third-country national or stateless person who has made an application for protection under the Geneva Refugee Convention and Protocol in respect of which a final decision has not yet been taken”, recognised refugees are those provided with a Residence Permit for a certain time period and vulnerable migrants are those who change their country of usual residence, irrespective of the reason of migration or legal status and their vulnerability can be understood as based on: (1) individual factors; (2) household and family factors; (3) the community factor; (4) structural factors; and (5) situational factors.

Naturally, an attempt was made to encompass a wide demographic selection in this purposive sample e.g., gender, age, etc. The sampling method allowed to increase the number of women in the study to achieve a greater gender balance. Furthermore, pilot participants were asked to choose their age groups, and the selected participants had to have at least three years of education to be able to read. In fact, people with at least a minimum level of education were important to ensure they would be able to navigate the platform, as well as people with ICT skills due to the remote piloting conditions. Overall, the scope of this sampling method was to collect data from the right participants for the pilot in an easily accessible, cost-effective and timely manner. With respect to the use cases, the sample group was split into asylum seekers, recognised refugees and vulnerable migrants.

With regard to the demographic characteristics of the sample reached, the split over the different groups is shown in the following table (

Table 1. Share of participants per type of user.

Use Case	Number of Users
Asylum seekers	126
Refugees	79
Vulnerable migrants	81
Total	286

) with 56% of the users being male.

The age distribution of the users is shown in the following graph (Figure 1). Additionally, the users reported 31 different countries of origin.

The participants’ level of education is shown in

Table 2. Share of participants per level of education.

Level of Education	Percent
None	1%
Primary	7%
Secondary	45%
Post-secondary	19%
Professional training	1%
Higher education	27%
Grand Total	100%

. The education levels were harmonised in order to enable comparison between different participants, their use case and country of origin. Some deviations may have occurred as a result of this categorisation, especially considering the differing educational systems not only between countries but also across third countries and EU Member States. The table below (Table 2) provides the details of the different education levels of the pilot participants.

The components of the NADINE platform that were tested included: (a) skill assessment; (b) matching of skills and needs; (c) chat-bot; (d) e-portfolio; (e) training tools; and (f) web interface: career and social information. With respect to the process of piloting, a concrete plan, including the timing of the testing, was prepared and adopted in all pilot locations (i.e., Greece and Spain). It is worth mentioning that in each country multiple NGOs were recruited to carry out the piloting activities. Thus, the alignment of the plans was necessary to achieve comprehensive evaluations. A total of seven hours was devoted to each user group.

Each piloting phase concluded with a thorough evaluation of user experiences. Feedback was obtained on users’ experiences as well as their perceived satisfaction with the platform. The feedback was collected from the participants immediately after they completed the testing of the platform’s components through a questionnaire. The evaluation questionnaire covered several aspects regarding the functionality of the platform, such as the content, the level of easiness to navigate the platform, the user-friendliness, and the overall design of the platform. The questionnaire was available in several languages (Arabic, English, Farsi, French, and Greek) to ensure accessibility for participants speaking different mother tongues.

During the piloting sessions, users were supported by on-site trainers and cultural mediators made up of members of organisations assisting the project team in finding pilot participants. Consequently, feedback was also collected from the “mediators” who had the opportunity to share their opinions on the components of the platform and report any observations they made throughout the pilots about the participants’ experience with the components. All responses were collected anonymously to avoid any ethical issues regarding personal data protection. To ensure the effectiveness of the evaluation, key indicators were identified before the activities to measure the usefulness of the platform. Some of these Key Performance Indicators (KPIs) were automatically produced from the platform and not through the evaluation questionnaires. Four indicator types were distinguished: content indicators (e.g., the ability of the tests to identify the main skills of a user, usefulness of the platform); satisfaction indicators (e.g., satisfaction with the allocated time or the usability of the platform, user retention); software indicators (accessibility, navigability); and the project’s impact/objectives indicators.

5. Results and Lessons Learnt

For the collection of the feedback, we used primarily online surveys. However, not every participant of the pilot responded to the survey and/or all components, which is the reason that the number of opinions does not sum up to the total number of participants in some cases. Overall, the vast majority of pilot participants were satisfied (54%) or very satisfied (35%) with the experience with the NADINE platform, as shown in the Figure 2.

NADINE platform is overall comprehensive according to 80% of the participants. Vulnerable migrants overall were less positive (62%) than other use cases (Figure 3). Participants indicated better translations would make navigation in the platform much easier.

One of the main features of the NADINE platform is the e-portfolio. The e-portfolio gathers information on the participants’ education, work experience, and other aspects relevant when creating a CV. This enables the platform user to extract all the information they have filled in through the e-portfolio to a CV automatically. As it is depicted in Figure 4, the majority of the pilot participants seems to find the e-portfolio to be good/very good. Comparing the three use cases, refugees were more negative than the other two use cases, while the vulnerable migrants were on average the most positive in their responses. In general, participants would rather not have to indicate their age for the e-portfolio. The time required to fill or complete the e-portfolio was overall considered to be appropriate (74%). None of the asylum seekers were negative about the aspect of the time required, while 8% of refugees were. Negative feedback was often a result of less-than-optimal digital literacy compounded by the fact that some of the pilots were run remotely due to the pandemic, which made communications difficult. On other occasions, negative feedback was a response to imperfections in the translation of texts that appeared on the platform.

Turning our attention to the implemented chatbot which answers migrants’ questions related to job search and integration, the results shown in Figure 5 were obtained. During the conduction of the pilots, the participants asked questions and received answers in the chat. Overall, the chatbot component was considered to be good (66%), though with fewer positive responses as for other components of the platform. The lower satisfaction rate compared to the e-portfolio is mainly attributed to the themes covered by the current version of the chatbot.

The skills training module offers to the users a range of training options which should increase their employability on the EU labour market. Trainings are available in a wide variety of skills, ranging from soft skills, such as teamwork, to hard skills such as ICT skills, numeracy, etc. For the purpose of the piloting sessions, the users went through the skills training “service” (interface) to get an overview of the variety of the offered trainings and the structure of each training module. For each skill, users can learn about the theory and put it to practice through practical exercises. Finally, they can test their skills. Overall, the satisfaction with the offered skills training module was high (71%) as shown in Figure 6.

Participants were overall enthusiast about the module and having the possibility to train their skills. In addition, they enjoyed the idea that the training would feed into the recommender. The training offers were easy to understand according to 67% of participants (as shown in Figure 7).

Another novel component is the recommender, which recommends job offers to the platform users, as well as specific training modules based on their skills. It provides an overview of member states and the EU’s job markets. Based on participants’ skills assessment results, the recommender provides tailored job recommendations to participants. It should be pointed out that the NADINE recommender does not take into account the legal framework in each country and cannot decide whether a refugee or an asylum seeker is eligible for a job. Instead, it focuses on the skills required for the job position at hand, leaving the specificities of legal frameworks to be found out through other parts of the platform (e.g., the chat bot or during the interview with the potential employer). Overall, half of the respondents thought the recommender to be good (53%) as shown in Figure 8. On average, vulnerable migrants were more positive about the recommender (58%) than the other use cases (50% each). Asylum seekers were on average more negative about this component (18%). Looking into the level of usefulness of the skill training recommendation received, more respondents thought it was good (61%) than the overall average of this component. There are differences across use cases, with vulnerable migrants being the most positive about this aspect (73%) than asylum seekers (52%). This may be explained by the differences in the levels of qualifications between vulnerable migrants and asylum seekers.

From the informal discussion and feedback collected through free text, the following results were reached. First, the evaluation of the presented platform shows promising results in terms of satisfaction with the platform. A large majority of participants were satisfied or very satisfied with their experience. Second, apart from feedback on technical details, the mediators provided fruitful insights in the overall process. The mock-up pilots proved highly useful in terms of identifying any bottlenecks in the implementation of the pilots and technical difficulties and recommended mock-up pilots to be standardised in similar endeavours. With respect to the levels of engagement, the sessions should be kept as short as possible and long modules should be shortened. The explanations provided for each module is also important to foster participants’ autonomy during the process. In line with the co-creation approach, granting participants more autonomy could help them providing valuable feedback, and mitigate feelings of pressure and stress.

Finally, we can conclude that the vast majority of participants were enthusiastic to review their scores from skills assessment toolset, and to explore the job recommender. Interest was noted on employment support and available trainings. More than half of the participants (75%) would recommend NADINE to someone they know, according to Figure 9. Especially refugees considered recommending NADINE (87%), while vulnerable migrants were a bit more reluctant (66%).

6. Conclusions

The integration of the migrants in the society definitely passes through their successful integration in the labour market. The currently available platforms that focus on information provisioning do not suffice. In this article we briefly presented a novel platform that integrates a set of ICT tools that exploit the latest technological trends to assist displaced people in their daily life and more importantly in their way towards an appropriate job. We also presented the results from the piloting activities we conducted so as to shed light on what they welcome and accept. The collected feedback showed that such tools are well-accepted and appreciated and allowed us to better understand their needs. The acceptance achieved was mainly attributed to the multi-round co-design and co-creation approach that we adopted. This approach started from co-designing the platform with people that usually act as mediators between displaced people and the labour market and proceeded to the refinement of the Proof-of-Concept version of the platform and co-creation of the next version with multiple groups of displaced people with different demographics and different status (refugees, asylum seekers and vulnerable migrants). The deficiencies and suggestions received are not relevant to the functionality but to the way this is presented, which leads us to continue the development (towards the next version of our platform) taking into consideration the collected feedback. Additionally, we have drawn a set of recommendations for piloting ICT platform with migrants which comes with intricacies compared to any other platform testing. The main lessons learnt are: the displaced people are ready to use ICT tools that help them find an appropriate job and welcome them; an important aspect that affects their use is apart from the value of the tools, their presentation and appearance since they significantly affect their usability; the support of the mother language of the displaced people is important but not enough to make the tools easy-to-use; careful planning and guidance during piloting activities and in general activities targeting the exploitation of such tools is key for the target group to embrace them and provide accurate feedback. This audience is ready to invest-devote time to the use of tools that contribute in their integration which is a very promising message towards making our societies more inclusive.