Application and Evaluation of a Blockchain-Centric Platform for Smart Badge Accreditation in Higher Education Institutions

Abstract

Since its conceptualization in 2008, blockchain technology has advanced rapidly and been applied in multiple domains. In higher education, blockchain can be applied to develop ICT systems that can revolutionize student accreditation through certificate verification and micro-accreditations, which represent skills and other learning outcomes, in the form of digital/smart badges. While there are multiple studies that highlight the significance of blockchain in higher education and propose digital systems, few of those studies include the evaluation of such proposed systems by real users. As such, the research question of how useful a higher education blockchain system would be for its relevant stakeholders remains largely unanswered. In the research publication at hand, a blockchain-powered higher education platform was applied in the School of Electrical and Computer Engineering of the National Technical University of Athens, where it was used and evaluated by students and professors at the school. The evaluation of the platform was positive, and participants found that the smart badge functionality was among the most useful. Finally, the execution and evaluation of the pilot led to several lessons learned and policy recommendations towards dealing with existing barriers and further promoting blockchain in higher education.

1. Introduction

Higher education (HE) is for most people the last educational level that they will attend, before entering the labor market. As such, one of the main aims of higher education institutions (HEIs) should be to prepare their students to become strong candidates for employment by instilling in them and teaching them the required skills and abilities that are necessary in the career pathways that they are projected to follow. This means that HEIs should not be disconnected from the respective domains of the labor market that their graduates will enter. Given that labor market requirements evolve rapidly, especially in technology-oriented fields, HEIs need to be at the forefront of such evolutions and constantly strive to update their syllabi and course materials accordingly [1]. However, updating the courses of HEIs to follow current labor market requirements is easier said than done as it requires time-consuming analyses of technological and other advances, as well as professional requirements from career and job posting websites. Such information often appears in the form of text that is hard to analyze either by hand or by traditional data analytics, which in turn signifies that value-adding ICT (Information and Communication Technology) tools could help higher education stakeholders to achieve their goals [2].

When HEI students graduate, they have to build their Curriculum Vitae (CV) and apply to various positions so that they can start their career. CVs include, among others, information about past experiences, skills, and official certificates and qualifications that a job seeker has acquired prior. Such information needs to be assessed by job offerors, who must also check the validity of the declared certificates and qualifications if they want to be thorough, thus creating overheads for the Human Resources (HR) departments of organizations, which must complete these tasks through mostly manual labor [3]. University certificates are usually included in a job application in paper form, which means that employers need to communicate with the administrative bodies of universities to validate them. This step is very important before any hiring decision, as degree fraud constitutes a very real and growing problem. Indeed, fake degrees generated by degree mills and fake universities are on the rise and the worldwide fake degree industry has grown from US$1 billion in 2015 to US$22 billion in 2022 [4].

The same is true for CV and resume embellishments, which take place when job applicants insert false information in their CVs about their skills, credentials, and capabilities. According to a 2023 Forbes article [5], 70% of employees have admitted lying on their resumes. Furthermore, 15% of resumes include lies in the skills section, 11% include fake education credentials, and 5% have embellished or falsified technological capabilities. This creates an even bigger problem for hiring organizations, given that skill embellishments are harder to locate, even if the applicant’s certificate is verified. A university certificate is a general description of an educational outcome and usually does not include details about the acquisition of specific skills or even the level of proficiency of an applicant in a certain skill, technology, etc. If job seekers want to validate their proficiency in a skill, they have to complete an online course and receive the respective certification. On the other hand, if hiring organizations want to validate the validity of a skill, they must pose relevant questions during the interview, or even include a written test in the process. It can be easily surmised that this is not effective during a hiring process, as it shifts the focus on the validation of already declared skills and qualifications instead of focusing on questions about soft skills, ambitions, how well someone fits a position and/or working environment, etc., which are especially important in an entry-level job, where a worker has to learn many new things about how an organization operates, which technologies are used, etc.

Based on the above, it can be surmised that, currently, HEIs are in need of data analytics and decision support tools to update their courses, as well as a trustworthy and secure way to connect students with their gained certificates, skills, and other qualifications. The latter would not only increase the HEI’s reputation but would also alleviate much effort from both universities and hiring organizations when a certificate, skill, or qualification needs to be validated. In addition, by employing such a system, not only hard skills, but also soft skills would be able to be showcased and validated in a student’s profile.

In the context of this publication, we showcase the QualiChain platform that was developed for the EU-funded project QualiChain. For the purposes of the project, the platform was introduced and piloted in the Electrical and Computer Engineering School (ECE) of the National Technical University of Athens (NTUA), where it was used and evaluated by over 100 students and professors at the school. The aim of the QualiChain project was to provide Decentralized Qualifications’ Verification and Management for Learner Empowerment, Education Reengineering, and Public Sector Transformation. The platform offers several functionalities for students, professors, universities at large, and even job seekers, employers, and job offerors that include data mining and analytics, decision support, certificate verification, and skill accreditation via smart badges. In the present publication, we present the QualiChain platform, describe the pilot concept and execution methodology, and report on the results of the platform evaluation by students and professors, the results of the Key Performance Indicators (KPIs) that had been set to evaluate the success of the pilot, and finally, offer lessons learned and policy recommendations that were generated as a result of this research work.

The current publication is structured as follows. Section 1 is the introduction to the document and presents current challenges that led to the motivation of generating this research work. Section 2 offers a short but concise literature review of blockchain application in HE. Section 3 describes the QualiChain concept and platform, as well the methodology that was developed and followed during the preparation, execution, and evaluation of the pilot that took place at the ECE School of the NTUA. Section 4 presents the results of the pilot evaluation through the analysis and explanation of the respective KPIs. Section 5 offers some generated lessons learned and policy recommendations, and finally, Section 6 concludes the documents and offers suggestions and pathways for future research.

2. Related Work

In this chapter, bibliographic research was undertaken to assess the potential benefits of blockchain technology in education and explore how it can bring added value to education stakeholders. In short, blockchain can help in certificate management, examination, and grading because it can ensure data transparency and integrity [6]. In addition, it can be used to validate achievements, skills, and learning outcomes and set learning goals as it can help monitor student progress and improve the learning process [7]. It can also be used to manage credit transactions and tuition payments, as it can ensure the integrity of transactions and reduce the risk of fraud. In HE, this could translate into a more efficient and secure system for managing student data and academic records, reducing administrative burdens, and enhancing the value of educational credentials. There are more subdomains in HE where blockchain could find applications, such as remote teaching/learning, organization of exams and tests, and so on. However, in the following three sub-sections, emphasis will be put on the examination of the literature that relates to certificate management, skills/learning outcomes management and validation, and university process optimization, which are the specific areas that the QualiChain platform targeted.

2.1. Certificate Management

Starting with certificate management, certificates constitute a key piece of evidence of the skills and knowledge a person possesses in a particular field or profession. In today’s job market, being a holder of certificates can significantly improve one’s chances of finding a job and enhance his/her career advancement. Certificates such as university degrees are usually published in paper form, making them vulnerable on the one hand to loss and/or destruction and on the other hand to counterfeit. In the As-Is situation, the task of republishing a certificate or validating it requires the involvement of issuing organizations, thus creating overhead for their personnel. Blockchain can offer a solution to these challenges, as it can maintain a list of the issuer and recipient of each certificate along with the digital signature, i.e., the hash value in a public database, which is stored identically on multiple nodes of the network [8]. Digital certificates secured in this way on a blockchain have significant advantages over paper or even “normal” digital certificates. Firstly, they cannot be forged, given that each transaction produces a digital trail that confidently ensures that the document was issued by the university and for the person written on the document. Additionally, verification is performed by anyone with access to the chain. Moreover, blockchain requires no “middlemen” for the verification of a certificate, which can be verified even if the issuing organization no longer exists, given that the file itself cannot be deleted from a blockchain ledger. The aforementioned advantages have led to several research publications that propose methodologies and ICT tools aiming to facilitate certificate management and validation as well as increased security and data privacy [9,10,11,12].

Specifically, in [13], the authors propose a blockchain application for certificate management and leverage smart contracts to automate processes and combat certificate fraud. The authors used the Embark framework to develop the application on the Ethereum blockchain. The advantage of utilizing the Embark framework is that it offers a wide array of capabilities that allow developers to configure a blockchain application from the backend to the frontend. The application prototype is very promising and shows the feasibility of using blockchain to develop applications for certificate management. Similarly, in [14], the authors propose the design of a blockchain system for certificate management that aims to enhance security through decentralization and also leverages smart contracts to optimize certificate verification. In [15], the authors showcase the early stages of development for a blockchain-based certificate management platform called BeCertify. What is noteworthy about the publication is that the authors provide a step-by-step overview of the early development of the platform, including the deployment of smart contracts, private key management, and the APIs that are necessary for a school to issue and verify a certificate. This process can be used by other research efforts as a guide on the steps for developing blockchain applications. Finally, in [16,17], the authors performed comprehensive bibliographic reviews of existing blockchain solutions for certificate management showcasing the increasing interest of the research community in exploring the capabilities of the technology to develop robust platforms for optimized and secure certificate management.

While the security and data privacy offered by blockchain are widely accepted, there are some concerns about the capabilities of future quantum computers to successfully attack such platforms by leaking the keys of individual nodes [18]. Such concerns have led to the generation of publications that review novel approaches for blockchain cryptography that are resistant to quantum computers [19]. Specifically, the authors of [20] propose an algorithm based on distributed key management that can alleviate such security concerns in a post-quantum world. This approach has immense potential for every application domain of blockchain.

2.2. Issuance, Management, and Validation of Qualifications

Other than official certificates, such as university degrees, blockchain can also facilitate the issuance, management, and validation of skills, qualifications, and micro-accreditations. In such a system, teachers, professors, and trainers could issue digital badges that represent specific skills and learning outcomes and award these badges to students that successfully complete a course. Additional badges could be issued and awarded to students with exceptional performance during a course, exam, or assignments. At the application level, this would allow students and job seekers to have a personal wallet where they can showcase their badges. The transparency and security of the data ensure that students’ skills and learning outcomes are accurately recorded and can easily be shared with potential employers or other educational institutions. Allowing the wallet to also store badges from open or online courses and other training programs would allow people to seek more avenues of lifelong learning, collect various digital badges, and improve their skill profile. The majority of research papers in this subdomain propose methodologies and tools that can not only store and validate micro-accreditations but can also set learning paths and goals, thus motivating users to keep learning and earning badges [21,22,23,24,25].

Specifically, in [26], the authors propose a blockchain system that employs public key cryptography and the Elliptic Curve Digital Signature Algorithm (ECDSA) to form digital signatures for various educational accreditations that can also be used for their verification. The system is developed with a microservices architecture, making it scalable and extendable to other applications. In [27], the authors provide a comprehensive review of competency-based education through micro-accreditations and explore its implementation in various blockchain frameworks and applications. In [28], the authors propose a blockchain approach for decentralized autonomous learning environments that allow learners to receive valid micro-accreditations from various learning/training institutions. Finally, in [29], the authors explore the potential of a blockchain system to be applied in Sweden for digital micro-accreditations received from vocational training. The authors also propose the characteristics that the system should have, taking into account all the related stakeholders. The concept of micro-accreditations in education and lifelong learning has gathered a lot of research interest in the last few years, as showcased by several bibliographic reviews on the subject [30,31,32]. As a final point, we must also note that the ability to safely and confidently validate a certificate or micro-accreditation is also very useful during a hiring process, alleviating a lot of effort and uncertainty for employers [33].

2.3. University Process Optimization

Finally, concerning the optimization of university processes, blockchain can be combined with data analytics and decision support to improve data security, allowing interoperability and transfer of courses, skills, and learning outcomes between different HEIs [34,35]. QualiChain was one such platform, as it combined the computational intelligence offered by data analytics and decision support with the security and transparency offered by blockchain to assist HE stakeholders to analyze educational and labor market data, receive personalized suggestions, perform analyses and decision support processes, and ensure that certificates, skills, and other educational data can be securely stored and transparently validated [3].

3. QualiChain Architecture, Pilot Concept, Research Questions, and Execution Methodology

3.1. QualiChain Architecture

The QualiChain project combined blockchain and ontologies with the computational intelligence provided by data analytics and decision support to develop a value-adding platform for education and labor market stakeholders. In the QualiChain architecture, blockchain and ontologies constitute the baseline services and facilitate the storing and verification of education credentials, as well as the creation of smart badges. Data analytics and decision support constitute the value-adding services and are combined with the baseline services to provide intelligent profiling, CV customization, course recommendations, and career advisory applications. One of the key objectives of the project was to showcase the potential of the aforementioned technologies in higher education, pilot the platform in different settings, and generate lessons learned and policy recommendations towards further application of blockchain in the domain. In the following sub-sections, we present the high-level technical architecture of QualiChain and expand on some key concepts, which are the selection of blockchain and the issuance of smart badges in QualiChain.

3.1.1. QualiChain Technical Architecture

In Figure 1, an overview of the QualiChain technical architecture is presented, including the layers, the modules, and the provided services within the layers, along with the communication among the different modules.

Specifically, the platform is composed of six main layers (Academic Verification and Accreditation, Knowledge Graph, Knowledge Extraction, Identity and Access Control Management, Analytics and Decision Support, and the Front End) that are summarized below.

• The Academic Verification and Accreditation layer is at the top of the architecture and includes services for access to the blockchain and the Distributed File Storage. This layer only communicates with the Knowledge Graph that is described next.
• The Knowledge Graph layer is the central data repository of the QualiChain platform. The QualiChain Ontology is used to store the data, while the QualiChain Knowledge Graph Query Engine enables querying the QualiChain Ontology. The QualiChain Knowledge Graph layer communicates with every other layer of the QualiChain Platform.
• The Knowledge Extraction layer includes services for data acquisition, annotation, and knowledge extraction from data (e.g., skill extraction from job posts). It interacts only with the Knowledge Graph layer for storing the extracted knowledge.
• The Identity and Access Control Management layer communicates with all other layers of the QualiChain platform to ensure that only authenticated users will have access to the requested resources.
• The Analytics and Decision Support layer includes an analytics module that consists of the following set of services: CV customization, Profiling, Course Recommendation, Curriculum Design, Recruitment and Internal Allocation, and Career Advisor, as well as a module for Multi-Criteria Decision Support with several available methods. This component communicates with the Knowledge Graph to read and write data, as well as with the Front End.
• Finally, the Front End includes the QualiChain platform user interface, as well as a Visualization Engine that enables complex data visualizations. It communicates with all services, providing the end user with an intuitive way to take advantage of all the offered services.

All these layers were deployed independently and communicate with each other through a Microservices architecture implemented in the QualiChain Mediator Component. Finally, the Front End communicates with the other components through APIs.

3.1.2. Selection of Blockchain

Towards maximizing the impact of the QualiChain platform over time, the blockchain service was decided to be developed in a future-proof manner. In practice, regarding the blockchain aspects of the platform, this translated to a requirement to abstract over the specific blockchain implementation, and to support the storage and verification of qualifications on multiple blockchains. This approach allows for easy adaptation of the platform to new developments in the future. The project reference implementation was built on Ethereum, as a widely used blockchain with support for smart contracts, but the processes were designed in such a way as to offer extensibility in the future. Moreover, the selection of Ethereum means that there is no encryption employed for data transactions. However, public/private key cryptography is used in the production of digital signatures for data transaction authentication (e.g., certificate verification), which was the main goal of QualiChain [36].

Blockchain-agnosticism and multiple blockchain support also offer other practical benefits. Given that different blockchain technologies have different trust features, we cannot assume that all stakeholders will trust the same blockchains equally. An educational institution may, as a matter of policy, select the blockchain platform that it prefers to use to issue qualifications. A graduate of that institution may have a different opinion. If we require that, e.g., students simply have to trust an institution’s choice, we replicate the current scenario of requiring centralized trust. If, however, the system is designed from the outset to allow the use of multiple blockchains for a single certificate, all stakeholders can have their requirements met with no conflict.

3.1.3. Semantic Blockchain for the Issuance and Validation of Accreditations

The Semantic Web is a family of technologies designed to enable interoperable and easy integration of data on the Web and to support a decentralized network of machine-readable data connected by links. As such, the concept of Linked Data was integral for the semantification of the QualiChain blockchain. Linked Data are RDF (Resource Description Framework) representations of data, in which data are represented as semantic triples that also create links between different datasets that are also machine-readable.

In QualiChain, the LinkChain platform was utilized to support the publication, sharing, and validation of Linked Data using a blockchain-backed verification model. By verification, we mean that, given a data item (or data items), a user can determine, in a trustworthy manner, when it was published, by whom, and whether or not it has been altered since publication. One of the key requirements for LinkChains is that it is designed around individual privacy and control of data. A user in LinkChains ought to be able to control their own data fully, but, where data have external value of some form, it should be possible to demonstrate their integrity. Qualifications are an ideal example of such valuable data, being both personal, and also, given their relation to jobs and income, with an incentive for users to modify and embellish them. LinkChains provides guarantees to data consumers that such modifications have not taken place. The implementation of LinkChains in QualiChain supports the publication of verifiable data using the Ethereum blockchain, making use of ERC721 tokens. ERC721 [37] is a specific Ethereum standard for non-fungible tokens, meaning tokens that cannot be spent. By making ERC721 tokens that can neither be spent nor traded, we achieved the equivalent of a badge, which can be given to a user as a permanent asset. More specific information about LinkChains can be found in [38,39].

Furthermore, the OpenBadges specification [40] was employed in QualiChain as a method of representing machine-readable educational qualifications with blockchain-backed verification. In fact, most of the ongoing initiatives into the blockchain validation of qualifications use OpenBadges as a standard format. Conceptually, a badge consists of a set of linked objects, typically represented in JSON-LD (the Linked Data extension of the widely used JSON format). As JSON-LD, a badge is a piece of semantic data in the Semantic Web’s Resource Description Framework (RDF) data model and can therefore be connected to any relevant data in the wider Semantic Web. In QualiChain, a badge consists of the following attributes.

• Badge Class: a general description of the qualification it represents.
• Issuer: the individual or institution giving the badge to a learner.
• Recipient: the individual receiving the badge
• An object representing an assertion that a badge of the given Badge Class has been issued to that recipient (by the issuer).

To summarize, in QualiChain, OpenBadges are used to represent the QualiChain smart badges, and LinkChains is used to verify them. Badge-specific wrappers around LinkChains were created to give developers APIs to create, issue, and verify a badge, and to input the information into a PNG file (i.e., OpenBadge JSON-LD is embedded into the image as metadata). This service also uses the platform authorization system, except for badge verification, as we wanted anyone to have the capability to verify badge data, and because verification should not rely on trusting a centralized service. Badge issuing specifically within QualiChain is kept as an access-controlled web service, because this anchors a badge to the blockchain, and it is important to ensure that we can control access to the ability to issue badges anchored with the QualiChain private key.

3.2. Pilot Concept and Stakeholder Needs

The pilot that was executed in the context of the present publication focused on the optimization of the teaching process, the curriculum, and the university operation as well as the acknowledgment and verification of skills and qualifications of undergraduate and Ph.D. students in a university setting. The pilot was applied in the ECE School of the NTUA, which is one of the oldest academic institutions for technical education in Greece. The ECE School of the NTUA is currently home to more than 2000 undergraduate students, 800 Ph.D. students, and 75 professors. As such, this pilot use case allowed us to test the QualiChain platform in an institution that includes technology-related courses from manifold fields, and hence offers the alumni the possibility to follow multiple potential career paths. Specifically, this pilot revolved around the following actions.

• Providing students (undergraduate and Ph.D.) with the tools that can help them build their academic/professional profile. Analytics and decision support functionalities provided recommendations for the improvement of a student’s profile.
• Provision of analytics and recommendations for the restructuring of the ECE School’s curriculum.
• Leveraging a trusted, immutable, and secure blockchain ledger to award students and lecturers with smart badges to accredit them for their performance and recognize their work and experience.

This pilot leveraged the computational intelligence stemming from QualiChain’s analytics and decision support tool [2] to suggest modifications to specific courses and by extension the ECE School’s curriculum as a whole, so that the curriculum can be modernized and include the skills that are required to succeed in the current labor market. In addition, both the undergraduate and Ph.D. students at the school were able to receive personalized recommendations that helped them effectively select university courses, open courses, and extracurricular activities. Those recommendations were based on students’ completed courses, preferences, QualiChain profile, and desired career paths. More importantly, this pilot utilized the platform’s blockchain component to provide student and lecturer accreditation and validation through the endorsement of smart badges that validate specific skills with verifiable micro-accreditations. QualiChain, through this pilot, enabled the verification of not only hard skills (e.g., “capacity to code in Python”), but also soft skills (e.g., “communication skills”, “teamwork”, “leadership potential”), which are admittedly very difficult for employers and recruitment agencies to validate. As of now, such skills are assessed via interviews and tests that assess the emotional intelligence and soft skills of candidates. By providing proof of such qualifications in a verifiable manner (through their addition in the smart badge ecosystem that was created), this pilot brings out the potential to significantly facilitate and expedite the evaluation of prospective employees. The actors that were involved in this use case are undergraduate and Ph.D. students, and professors.

• Undergraduate students: One of the students’ main needs revolves around the selection of courses that are the most interesting to them and can facilitate the improvement in their skills and knowledge so that they will be better prepared to enter the labor market upon graduation. As of now, there is a lack of technical infrastructure, organized databases, and recommendation systems that could help students at the university choose suitable courses. In addition, there is currently no way to verify specific skills of students, such as advanced knowledge of a specific programming language, which is a significant problem when it comes to proving their qualifications to potential employers. As such, QualiChain provided students with the opportunity to have their own personal profile that enables the quick verification of skills and qualifications gained from completing courses and projects.
• Ph.D. students: Most Ph.D. students perform tasks such as teaching and correcting tests, in addition to carrying out research. However, oftentimes, there is no way for Ph.D. students to be recognized for their work. Similar to the case of undergraduate students, QualiChain leverages smart badges to enable the recognition and verification of the tasks that Ph.D. students perform. As a result, they can build a professional profile and enhance it with qualifications that they gain during their work at the university, and which are currently not verifiable.
• University professors: Currently, updating a course is a very time-consuming process since professors do not have a structured way to review their courses and compare them with the job market requirements. The analytics and decision support functionalities of QualiChain enable professors to add skills and knowledge quickly and effectively to their courses. In addition, professors can award smart badges to both undergraduate and Ph.D. students to recognize their skills and contributions.

3.3. Research Questions, Test Cases, and KPIs

This section presents the main research questions that this pilot investigated, the test cases that have been designed, and the KPIs that will be used to measure the performance and usability of the smart badge accreditation functionality.

As previously mentioned, most students do not have a structured way to select courses and make informed decisions that could help them follow their desired career path. Therefore, this pilot tried to understand whether the analytics and decision support tools of the QualiChain platform can simplify students’ choices and help them make informed decisions about their future careers. In addition, there is a growing gap between the skills that are being taught in HEIs with the ones that are required by the respective labor market. Therefore, this pilot investigated whether the QualiChain platform could bridge the gap between academic organizations and the job market by providing recommendations for restructuring the ECE School’s curriculum. Moreover, this pilot investigated the importance and added value of a smart badge ecosystem in a university setting. Finally, a research question that was relative to this pilot but was tackled under the context of other QualiChain pilots (public sector-related pilots) was whether blockchain has the potential to provide a new solution for decentralized certificate verification. The aforementioned research questions helped us shape three main test cases that guided the pilot-specific experiments. These test cases are presented in

Table 1. Test case scenarios of the pilot.

Case	Scope	Actors
A	Students create personal profiles and use analytics and decision support to improve their skills and knowledge and select suitable courses/extracurricular activities.	Undergraduate students
B	Professors use analytics and DSS tools to update their courses.	Professors ECE School
C	Undergraduate and Ph.D. students receive smart badges from professors to validate their skills and qualifications.	Undergraduate students Ph.D. students Professors

below.

Based on the test cases, the following two main technological offerings were tested: data analytics and decision support, and blockchain for certificate verification and issuance of micro-accreditations. While the students of the ECE School tested the full functionality of the platform, the results and evaluation of the data analytics and decision support functionalities in the context of this pilot have been presented and sufficiently explained in [2]. As such, the current publication will focus on the blockchain part of the platform and pilot (Test Case C) and delve deeper into the execution methodology of the pilot, as well as lessons learned and policy recommendations that can be derived from the available results. The pilot execution methodology is presented in Section 3.1, while the lessons learned and policy recommendations are discussed in Section 5.

The KPIs that will be examined in the context of this pilot concern the usefulness and satisfaction of students and professors towards the blockchain micro-accreditation that the QualiChain platform offers. In

Table 2. Blockchain-related KPIs.

Name of KPI	Criteria for KPI Selection	How Was Data Collected?	Pilot Stakeholder
Student skillset improvement (measured from the student’s personal profile before and after the pilot case application) both in terms of number of skills/smart badges and specific skills levels.	This KPI allowed us to see the improvement in the declared skills of a student’s profile before the execution of the pilot and after the smart badge functionality was implemented and tested by the students	From platform databases	Undergraduate students
Number of smart badges/students	This KPI was selected to evaluate the interest of both students and professors to be part of a smart badge ecosystem	From platform databases From questionnaires	Undergraduate students Ph.D. students
Average percentage of verified skills in a student’s profile (number of smart badges/number of skills in CV)	This KPI was selected to show the value of showcasing a skill versus showcasing a verified skill in a student’s profile	From platform databases	Undergraduate students Ph.D. students
Number of validated Ph.D. student tasks	This KPI was selected to evaluate the interest of professors in creating Ph.D.-specific smart badges	From platform databases	Undergraduate students Ph.D. students
Ph.D. student work recognition and profile improvement through smart badges accreditation	This KPI was selected to evaluate the participation of professors and Ph.D. students in the smart badge ecosystem	From platform databases From questionnaires	Undergraduate students Ph.D. students
Number of smart badges awarded by professors for each course	This KPI was selected to evaluate the interest of professors in creating course-specific smart badges	From platform databases From questionnaires	Professors

below, these KPIs are presented.

3.4. Pilot Execution Methodology

This section describes the methodology that was followed for the execution of this pilot case. It provides an overview of every step that was completed before the beginning of the pilot’s operation and the involvement of pilot users in the platform. The following Figure 2 presents the process that was followed from the very early stages of the project for the preparation and completion of this pilot.

Furthermore, in

Table 3. Timeline for pilot implementation.

Months	Pilot Activities
[M1–M6]	Identified popular professions for the school’s graduates and performed data crawling on job posting sites to formulate the respective job market requirements. Furthermore, we generated the consent form and information sheet that were necessary for CV gathering.
[M7–M12]	Presented the QualiChain concept to students at the school and started the process of CV gathering. Moreover, we mapped the curriculum of the school and generated the user requirements for the QualiChain services. These activities also helped us to generate pilot-specific test cases and KPIs.
[M13–M20]	Having gathered a sufficient number of CVs, we analyzed the current skill level of an average student at the school. We also assessed the gap between the school’s curriculum and the job market requirements. Finally, we completed the first round of development for the QualiChain services.
[M21–M28]	We started organizing the focus groups with the early participants and received a first round of feedback, which we used to improve upon the QualiChain services (second round of development).
[M29–M36]	Finalized the development of the services and the focus groups with the students and professors. Finally, we organized the received feedback and used it as input to address the KPIs and generate lessons learned and policy recommendations.

, we present a timetable for the implementation of the pilot that includes all preparatory steps, data gathering, the pilot execution, and the evaluation of the platform by the participants. The QualiChain project had a total duration of 36 months.

In the following paragraphs, we expand on the timeline presented in Table 3 and offer more details about each individual activity. To begin with, the labor market requirements of popular professions among the school’s graduates had to be analyzed and assessed with the current skill/knowledge profile of the average student at the school. As such, the data analytics module of the platform produced several indicators concerning popular careers among the NTUA School’s graduates. Following that, the pilot team charted the school’s current curriculum to identify gaps and insufficiently addressed knowledge fields as well as the courses in which those gaps may be addressed by updating/enriching the course’s syllabus. All the above-mentioned data were used as input for QualiChain’s Analytics and Decision Support tools that provide personalized suggestions for students, recommendations to professors who want to update their courses, and overall guidance and decision support for the curriculum update of the entire school. Based on this methodology, the distinct steps that were followed throughout the execution of this pilot are as follows:

• To identify popular professions for the school’s graduates, the project team defined five target sectors (i.e., Computer/Data Science, Electrical Engineering, Management, Telecommunications, and Teaching) that constitute an amalgamation of potential career trajectories that can be followed based on the knowledge offering of the school’s curriculum. Each target sector represents various specific professions and was used for the next step, that of data crawling from job posting sites.
• Data crawling was performed on job posting sites and helped formulate the job market requirements for each of the target sectors identified.
• To analyze the current skill level of the average ECE student, data gathering activities were initiated, in which students at the school were requested to send their CVs to the project team. The project team managed to gather around 90 CVs from students (undergraduate and Ph.D.). It is difficult to estimate how many students were asked to send their CVs compared to how many actually sent them. The reason for that is that the requests to the students were made in the context of several university courses, as well as through e-mails and word of mouth from already participating students. As such, we can estimate that the number of students that received the request could be a few hundred. To ensure the legality of the process, an informed consent form and information sheet were drafted, validated by legal representatives, and given to the students to sign. The students’ CVs could not be used without a signed consent form from the data owners.
• The curriculum mapping was completed for over 140 specialization courses (which are different from the main courses that all students must complete) and the data helped in the training and development of the platform’s recommendation tools.
• After gathering CVs and analyzing the requirements of the labor market, several analyses were performed to assess the gap between the school’s curriculum and the job market requirements. This step yielded several skills that are in high demand by the labor market but are not taught in any of the courses of the school’s curriculum. Such skills include windows, docker, CSS, APIs, react, JSON, GIT, android development, iOS development, JavaScript, HTML, Linux, and PHP, among others.
• QualiChain’s analytics and recommendation tools, considering the mapping of the curriculum and the missing skills, provided recommendations concerning the courses in which those skills could be covered.
• The tool that provides personalized recommendations to students regarding courses and activities was developed.

These seven steps constituted the preparatory stage of the pilot. After that point, students and professors were engaged and introduced to the QualiChain solution, and the actual execution of the pilot with early users was initiated. Before introducing new users to the platform, the pilot approach was presented and validated in more than six impromptu workshops in the classroom with over 70 students each time. Students responded very positively to the functionalities that QualiChain can offer and concurred that it indeed meets their needs.

For the evaluation of the QualiChain technical solution, several workshops, focus groups, and platform exhibitions took place with undergraduate and Ph.D. students as well as professors at the school, so that all three of the pilot’s scenarios could be covered. A total of 50 undergraduate students were engaged through the various classes that are taught by the professors of the Decision Support Systems (DSS) lab. A total of 40 Ph.D. students were engaged through the DSS lab as well as other research labs of the ECE School. Finally, 11 professors were engaged via e-mails. For the selection of pilot participants, as mentioned in step 3 of the methodology, we asked students during courses, through e-mails, as well as by asking participating students to disseminate QualiChain through word of mouth. At the beginning of the project, we were ideally aiming for at least 50 participants (including undergraduate students, Ph.D. students, and professors). The fact that we managed to run the pilot with over 100 end users gives credence to the potential of QualiChain and solidifies the conclusions, lessons learned, and policy recommendations.

Each of those groups of stakeholders were introduced to the platform separately. Undergraduate student workshops consisted of around 5–10 students each time. Ph.D. student workshops consisted of 10 students. Finally, professors were introduced to the platform one at a time. Each activity lasted around 1.5–2 h.

The first 15 min of every platform exhibition activity were spent in introductions among the participants, the introduction to QualiChain and the specific pilot case, as well as the goals of the project and the affected stakeholders. Following that, participants were given the link to the platform and were requested to create an account. For every following action, the pilot representative spent 5–10 min introducing and explaining the various functionalities that the participants were going to use. This was to ensure that students and professors became familiar with the tools so that they could further use them after the workshop in their everyday lives in the university.

For undergraduate students, the next action entailed the creation of their profile and more specifically filling in their CV in the platform. After that, students were requested to select courses from the course tab as well as fill in the courses that they have completed. The students were then introduced to the Multi-Criteria Decision Support System (MCDSS) and were requested to perform an analysis concerning the selection of a course among several of the options that were suggested to them via their profiles. Finally, they were introduced to the smart badge functionality and shown how to award their colleagues and professors with one or more of the available smart badges. Then, the participants were given the questionnaires for the evaluation of the QualiChain platform and were requested to fill them out after using the platform for at least one semester.

The workshops with Ph.D. students were quite similar with a few key differences. First of all, after creating their account and profile, participants were shown how to declare the Ph.D. role on the platform. Following that, instead of courses, they were introduced to the jobs tab and were shown how to apply for a job offer that is available on the platform. The MCDSS was similarly introduced but the analysis performed concerned the selection of a skill among various alternatives so that they can enrich and advance their academic and professional profiles. The last actions entailed the tutorial for awarding smart badges and the evaluation questionnaires.

Finally, professors were introduced to the platform in a similar way. The first things they were asked to do were to select the courses that they teach in the courses tab and transfer some of the theses they have published in the thesis tab. After that, they were asked to visit the curriculum redesign part of the platform and check whether they teach any of the courses for which additional skills were recommended. In a few cases that a skill was suggested for several of the professors’ courses, they were introduced to the MCDSS function and performed an analysis to select the most suitable course to update. Finally, professors were introduced to the smart badge functionality and shown how to create and award smart badges to their undergraduate and Ph.D. students.

Smart badges could be awarded by both professors and students. On the one hand, professors could award their undergraduate and Ph.D. students with smart badges. On the other hand, students could award other students and also lecturers. Each smart badge has a counter that signifies how many times a specific smart badge has been awarded to someone, as can be seen in Figure 3.

Smart badges were also showcased in the users’ profiles. Students, graduates, and job seekers had the choice to make their profiles public, or give access to specific users, in case they are looking or have applied for a job and want their prospective employer to be able to validate the skills in their profile (Figure 4).

The names of students and smart badge issuers have been hidden to preserve their anonymity.

4. Platform Evaluation and KPIs Analysis

4.1. Platform Evaluation

Το verify that the QualiChain platform V1.0 is fully functional and works properly, not only as individual services but also as an integrated solution, a significant amount of effort was dedicated to integration- and system-level testing. Integration testing is the phase in software testing in which individual software modules are combined and tested as a group and system testing is a level of testing that validates the complete and fully integrated software product.

Given that most QualiChain components are independent of each other, the developed integration tests focused mostly on verifying that access to different services can be granted only for authenticated users that are authorized to access the requested services and data. Specifically, eight integration tests were developed for testing the access to different resources of QualiChain. These tests included the following.

• Test smart badges: tests the creation and awarding of badges for authorized and unauthorized users.
• Test courses: Verifies that the creation of courses cannot be performed by unauthorized users but is feasible for authorized ones (professors). Also, it verifies that only the user that created a course can update or delete it. Moreover, only students can enroll to a course.
• Test course recommendations: verifies that only authenticated users can receive course recommendations.
• Test curriculum designer recommendations: verifies that only professors and academic organizations can receive curriculum design recommendations.
• Test CVs: Verifies that a user can create, update, or delete his/her own CV and no other user’s CV. Furthermore, it verifies that only authorized users can view another user’s CV. This case applies for professors that intend to view the CV of a student enrolled on their courses, as well as for recruiters that intend to view the CV of a job candidate.
• Test jobs: Verifies that the creation of jobs cannot be performed by unauthorized users and is feasible for authorized ones (recruiters). Also, it verifies that only a user that created a job can update or delete it. Moreover, a recruiter can view the profiles and CVs of users that have applied for his/her jobs.
• Test profiles: verifies that a user can create or update his/her own profile and no other user’s profile.
• Test thesis: verifies that only professors can create a thesis, and only professors that created a thesis can update or delete it.

System-level testing on the other hand was conducted by utilizing an open-source record and playback test automation tool, Selenium IDE [41], which simulated the execution of the following use-case scenarios by the corresponding types of user accounts.

• Enrolling on course (student);
• Applying for job (job seeker);
• Applying for thesis (student);
• Creating a smart badge (professor);
• Viewing thesis candidates (professor);
• Creating a course (professor);
• Creating a new job object (recruiter);
• Awarding and validating a smart badge (professor/student);
• Uploading a certificate (academic organization);
• Validating a certificate (recruiter);
• Revoking a certificate (academic organization).

Initially, each scenario was executed manually by the platform testers. Selenium IDE recorded every step required to complete each task, automatically converting them to executable commands and using multiple locators for each interface element the users interacted with. Thus, each scenario was finally described by a script of commands that, when reproduced, should always successfully complete the scenario. These scripts constituted the end-to-end tests for the QualiChain platform.

Both the integration- and system-level tests ran successfully 100% of the time, thus proving that the platform works securely and as intended.

We must also note that the QualiChain platform was not integrated into any existing systems. Any interaction between QualiChain and existing systems (e.g., a university’s system) was facilitated through API calls that ran successfully. Moreover, the standardization of the QualiChain solution was facilitated from the start of the project when the decision to develop QualiChain to be blockchain-agnostic was taken. This practically means that QualiChain can be integrated in any other system without such considerations. Concerning the platform’s scalability, there were no bottlenecks observed during testing, given that the blockchain is used only for verification purposes. This means that the underlying processes of the platform are scalable regardless of the number of verification requests. The same is true for the data analytics components of the platform that deal only with textual data that do not pose significant load to the system. As an example, during the piloting of QualiChain to students, around 1400 course suggestions took place and the average time/course suggestion was measured as 0.236 s. Finally, the cost-effectiveness of the QualiChain platform can be proven when compared to the current manual processes for qualification verification. In the As-Is situation, the verification of a university certificate (e.g., a degree) requires time from the university’s secretariat (which can be up to a week) as well as a small expenditure by the student requesting the service. However, in the tests performed by one of the other pilots of QualiChain [42], 100 certificates were verified in QualiChain. The results of the tests showed that the cost for verification is 0 ETH, and the latency is around 1.3 s, which is orders of magnitude better than the current situation.

4.2. KPIs Analysis

The results of the aforementioned questionnaires, coupled with evidence from the platform’s databases, allowed us to assess the level to which every KPI of the pilot was achieved. The results for the six KPIs that were presented in Section 3.2 are presented below and they show an overall positive evaluation of the smart badges functionality by both students (undergraduate and Ph.D.) and professors.

Student skillset improvement: This KPI was measured from the QualiChain platform, based on the average number of smart badges that were awarded to students. The 50 undergraduate students that participated in this pilot case were awarded on average about seven smart badges each. These smart badges only represent the skills that students can learn from the courses of professors who also participated in the pilot. In addition, many of those smart badges represent soft skills such as leadership, team spirit, communication, and ability for presentations, among others. Before the application of the pilot, students would not be able to prove that they possess these skills other than with a recommendation letter from a professor. Of course, students would still possess those skills in any case but with the help of QualiChain, they can now prove so to prospective employers. In addition, the existence of smart badges in the QualiChain platform can give students the incentive to participate in the activities that will give them the opportunity to earn additional skills and qualifications. As such, it can be surmised that QualiChain can lead to an increase in a student’s skillset.

Number of smart badges/students: All in all, the 90 students (40 Ph.D. and 50 undergraduates) who participated in the platform were awarded on average about seven smart badges with some small variations among undergraduate and Ph.D. students. It can be said that the smart badge accreditation functionality was one of the most well received by the participants. This can also be proven by the answers that participants gave to the question “state your level of agreement with the following statement: The smart badges that I have received from QualiChain will help me build a stronger professional profile”. In total, 1/90 (1%) answered disagree, 8/90 (9%) answered neutral, 30/90 (33%) answered agree, and 51/90 (57%) answered strongly agree. As such, and given that 90% of students agreed with the significance of smart badges in an academic/professional profile, it can be said that the value of smart badges was evident to the pilot participants.

Average percentage of verified skills in a student’s profile: On average, the 50 undergraduate students that participated in the pilot had about 10 skills on their CVs. Based on the previous KPI, the average number of smart badges for students was about seven. This leads to 70% of skills being verified. The smart badges that the students received do not necessarily validate skills already existing on their CVs. This fact makes sense since most of the skills that students have on their CVs might have been earned by courses whose professors did not participate in the pilot or through other means (online courses, work experience, etc.). Despite that fact, these numbers prove that smart badge accreditation has the potential to revolutionize the way that skills are validated, especially if the QualiChain platform included more professors, employers, etc.

Number of validated Ph.D. student tasks: The 11 professors who participated in the pilot created 25 distinct smart badges in total for their Ph.D. students. Coupled with the answers from the questionnaires, 0/11 (9%) created no smart badges, 6/11 (55%) created 1–3, and 4/11 (36%) created 4–7. The fact that some professors created fewer or no smart badges can be explained by the fact that professors can award to their students smart badges that have been created by other professors (e.g., thesis completion) and in that case, most of their needs are being covered by the earlier users. The smart badges created by professors for their Ph.D. students validate various tasks performed by Ph.D. students such as lectures, research, grading exams, authoring scientific papers, and creating presentations among others.

Ph.D. student work recognition and profile improvement through smart badges accreditation: The 40 Ph.D. students that participated in the pilot received on average a little under seven smart badges each. This functionality was considered very important by both Ph.D. students and professors at the school since at the moment, there is no official way to validate the tasks that Ph.D. students undertake for the university. This fact led to a notable improvement in their profiles, which is also proven by the questionnaires that they filled in. In the question, “state your level of agreement with the following statement: The smart badges that I have received from QualiChain will help me build a stronger professional profile”, 4/40 (10%) answered neutral, 13/40 (32.5%) answered agree, and 23/40 (57.5%) answered strongly agree. Overall, work recognition for Ph.D. students via smart badges is one of the most well-received functionalities of QualiChain, especially given that the validation is performed by a professor at the school.

Number of smart badges awarded by professors for each course: Overall, the 11 professors that participated in the pilot created 38 distinct smart badges for 15 courses. On average, the professors awarded about 23 smart badges per course, which is deemed sufficient given that there were 50 student participants and that they did not all select every course. In addition, in the question “state the level of agreement/disagreement with the following statement: “I believe that the smart badge accreditation functionality is integral in a university setting””, 1/11 (9%) answered neutral, 5/11 (45%) answered agree, and 5/11 (45%) answered strongly agree. In conclusion, it can be said that this KPI was not only sufficiently addressed but that the smart badge accreditation functionality was very well received by professors who were convinced about its usefulness.

5. Discussion

In the current section, the results and findings that were generated after the completion of the pilot are discussed. The discussion is divided into two sub-sections. In the first sub-section, general lessons learned that were gathered throughout the pilot process are presented and discussed to guide future researchers and developers to overcome potential barriers, while also suggesting pathways that can increase blockchain technology penetration and end-user participation. In the second sub-section, the generated knowledge and lessons learned are shaped into targeted policy recommendations that can assist policy and decision makers in the fields of blockchain, HE, and the labor market.

5.1. Lessons Learned

5.1.1. CV Gathering from Students and Data Protection

In the first step of this pilot’s execution, the pilot leader requested through various channels (classroom, social media, university websites, e-mails) CVs from undergraduate and Ph.D. students that would be analyzed to draw conclusions on the average skill level of students. Since the CVs that were requested represented personal data, according to the legal strategy of the project that is compliant with the General Data Protection Regulation (GDPR), they had to be accompanied by a consent form and information sheet. Based on the legal obligations of the project, this consent form required a signature from an official administrative body (such as a police precinct), which was a tedious process for students. For that reason, at the beginning of the pilot’s operation, participation of students was low with regard to CV gathering. During the second year of QualiChain’s operation and during the first wave of the COVID-19 pandemic, additional digital services were developed for public administrations that allowed this process to be performed online. This shift in administrative processes greatly increased the number of CVs received from students. It is evident that in our modern society, in which ICT technologies provide novel solutions for various everyday problems and tasks, the effort to make a shift from physical to digital administrative processes needs to be intensified. Nowadays, citizens are always “connected” either via a personal computer or through a smartphone and national administrative processes need to reflect and leverage that fact to create new and innovative tools. This need was also magnified by the COVID-19 pandemic and the quarantine situation, which made it very difficult to receive a physical signature. In addition, this activity validated QualiChain’s legal strategy and provided some perspective regarding the GDPR. It was proven that the GDPR should not be viewed as a barrier when it comes to developing technical solutions but rather as a set of guidelines to protect personal data and inform end users about their rights.

5.1.2. QualiChain Integration in the ECE School

While the QualiChain platform was successfully piloted in the ECE School, it is far from being considered officially integrated in the school’s processes. Like most HEIs, NTUA also operates by following strict administrative processes that are set out by its various administrative bodies. This creates many barriers when it comes to considering and implementing updates and changes in the school’s operation, even in the cases in which the changes will have a positive impact, as shown by the pilot results. For example, the introduction of new courses in the school’s curriculum needs to be agreed upon by the council of professors. While this council meets several times a year, the thematic is usually overflowing with administrative issues. In addition, any changes in the school’s processes must be agreed upon by the council of professors as well as the school’s secretariat. QualiChain would bring about innovative, but also radical changes, when it comes to the everyday operation of the school, such as alleviating overhead burdens for certificate issuance and verification and introducing a new evaluation system for students and professors via smart badge accreditation. The current situation in the university is considered to be counterproductive when it comes to implementing changes and introducing innovative practices in the school’s operation. It is strongly believed, as concluded by the pilot’s operation, that university administration in Greece should become simpler, more agile, and less resistant to change.

5.1.3. Personalized Services for Students

The QualiChain platform offered, among others, several functionalities that are personalized to students of the ECE School, such as course and thesis selection, personalized recommendations regarding skills and courses, and smart badge micro-accreditations. While most students had a positive opinion regarding these personalized services, we also received feedback from them regarding their expectations from such a platform. Several students were disappointed that the platform was not connected to the school’s administrative systems and as a result they had to select the courses that they are attending twice: once on the QualiChain platform and once on the university’s official system. Moreover, they would like to see other functionalities in QualiChain regarding their everyday life such as a calendar for courses and assignments, connection of the platform with the school’s forum (that is being managed by students), and more notifications concerning school announcements. In addition, some students thought that QualiChain should be linked with other platforms, such as LinkedIn, so that their CV and skills could be transferred from one platform to the other (while also transferring the verification of skills through the smart badges). While, overall, students were quite satisfied with QualiChain and the combination of innovative technologies that it offers, the lack of connection with similar platforms, combined with the fact that QualiChain’s UI is not as intuitive as they would like in some functionalities, could result in some users losing interest in the platform. The implementation of this feedback in the form of platform improvements and connections with other systems would require less resistance to technological change from university administrative bodies and increased collaboration of research projects with established solutions, thus creating an ecosystem in which both research and existing solutions can improve and learn from each other.

5.1.4. Smart Badge Accreditation

The smart badge accreditation functionality of the QualiChain platform was the most well received by the pilot’s stakeholders. Smart badges offer verifiable and immutable validation for skills gained and tasks completed and as such, they were positively evaluated by both students and professors. However, the integration of the smart badge functionality in the everyday life of the university could potentially lead to some resistance and complaints by some professors and groups of students that do not agree with this type of student evaluation. Some students thought that such a process could lead to the creation of unfair leaders in the cases of exceptional or popular students. Of course, such an issue would be easy to solve by setting an upper limit to how many instances of a smart badge can be awarded to a student by other students. An additional functionality that would solve this issue is a filtering mechanism that would allow a user to only see smart badges in a profile that have been validated by professors, as they can be trusted to be more impartial. In addition, based on the received feedback, it seems that despite the explanations provided during the workshops, students and professors still had some misunderstandings concerning the role of blockchain in the platform. Some thought that the right to data deletion could not be respected even though no personal data are stored in the platform’s blockchain. In fact, the blockchain was used for the verification of transactions in the system, while students’ personal data were stored off-chain to respect the right to data deletion that is imposed by the GDPR. This solution is widely used in the research community. Others failed to see the merit of having blockchain-verified smart badges in their profiles, given that their understanding of the technology is lacking. It is strongly believed that knowledge around blockchain should be increased via the inclusion of blockchain in the curriculum, special lectures, and platform exhibitions, as well as cooperation of HEIs with national and European blockchain initiatives. On another note, the smart badge functionality provided the opportunity to verify soft skills as well. Skills like teamwork, ability to communicate, leadership, etc., can manifest during presentations and collaborative assignments. While these skills are also requested by the labor market, job seekers are having difficulties in proving them to prospective employers and thus, a smart badge ecosystem that includes students, universities, job seekers, and job offerors would have a positive effect on such processes.

5.1.5. QualiChain Platform as a Whole

Overall, it can be said that the implementation of the QualiChain platform in the context of this pilot was successful. QualiChain presented to the stakeholders of higher education a combination of technologies that they have not seen before in the context of education. It can be said that leveraging innovative technologies can lead to new and original platforms and tools that meet real needs and provide out-of-the-box solutions. From a technical standpoint, combining those innovative technologies does not always lead to the most robust solutions. However, it is imperative that such approaches are explored to develop the services of the future. On another note, research projects like QualiChain are critical in bringing people in contact with some of those technologies, which can lead to increased trust. For example, while blockchain is still considered by most as synonymous with cryptocurrencies, QualiChain proved that it is largely underutilized in other domains as a solution for dealing with private and sensitive data. Especially in the post-COVID-19 era, with a new reality being shaped when it comes to learning and working, functionalities like certificate verification, smart badge accreditation, and decision support could and should be at the forefront of future solutions.

5.2. Policy Recommendations

The elaboration of a set of practical recommendations and valuable insights for European policy makers (indicatively targeting future research and innovation strategic plans), but also other interested stakeholders (e.g., national policy making instruments, public sector representatives, education stakeholders, researchers), constitutes a core value-added output of the pilot’s operation and the project at large, targeting the acceleration of the project’s results and crowning project endeavors to promote state-of-the-art concepts and technologies. The policy recommendations that are presented in the current section build on the knowledge created throughout the project and were generated by following a highly collaborative methodological approach, as depicted in Figure 5 below.

The first step was the design of a short, yet complete, template to aid contributors to provide their input in an effective, efficient, and user-friendly way. The survey was distributed to all consortium partners, and a timeframe of two weeks was given for completion, an activity that constitutes the second step of the methodological approach. The survey template can be seen in Figure 6 below.

The third step was the preliminary analysis (e.g., merging of inputs, homogenization in terms of language used, grouping of similar/identical inputs) of the results. The fourth step was the identification of stakeholders per result. Although the survey itself asked for “Impacted Stakeholder(s)”, a brainstorming exercise on additional stakeholders’ groups that could be targeted and/or affected by each result took place. The fifth step was a 45 min workshop for further enriching the findings. The list of analyzed preliminary results was distributed a priori to all participants, and the activity focused solely on fine-tuning existing results, since no additional results were extracted. The sixth (and final) step of the collaborative exercise was a 30 min workshop, realized two weeks after the previous step for the validation of the results. Each policy recommendation is accompanied by additional attributes (besides the description), namely:

• Thematic: a particular subject relating to the policy recommendation.
• Timeframe: the time horizon that the policy recommendation is expected/proposed to be applied (taking the distinct values “Short”, “Med”, and “Long”).
• Impacted Stakeholder(s): the group(s) of stakeholders that the policy recommendation is expected to mainly impact. The list of identified stakeholders includes policy makers and regulators, public administrations, funders and research and innovation activities, accreditation organizations, researchers, students, lifelong learners, job seekers, recruiters, employers, and technology providers.

After implementing the described approach, a list of 21 policy recommendations was distilled. To present these recommendations, a clustering per reference level was selected as the most efficient and effective way. For the needs of this task, “reference level” is defined as the main driver in the context of the business target; indicative examples include “Education”, “Labor market”, and “Generic/Horizontal”. A visualized overview of the taxonomy described above can be found in Figure 7 below.

Of the 21 generated policy recommendations, not all are directly related to the specific thematic of the current publication that relates to leveraging blockchain to facilitate the validation and verification of certificates, skills, and other qualifications. As such, the 12 most relevant ones have been distilled and are presented in

Table 4. Policy recommendations.

Policy Recommendation	Thematic	Timeframe	Impacted Stakeholders
Emphasize the concept of micro-accreditation	Methodological Procedural	Short	Policy makers and regulators Accreditation organizations Students and learners
Connect education institutions and their curricula with job market requirements	Societal	Mid/Long	Accreditation organizations Students and lifelong learners Job seekers Recruiters and employers
Establish dialogue among educational institutions around qualification standards	Societal	Mid	Policy makers and regulators Public administrations Accreditation organizations
Adopt an official CV standard such as Europass2	Methodological Procedural	Mid	Public administrations Students and lifelong learners Job seekers Recruiters and employers Policy makers and regulators
Strengthen continuous training and skills enhancement	Methodological Procedural	Long	Public administrations Funders and R&I activities’ supporters Job seekers Recruiters and employers
Ask for clear, comprehensible, and easy-to-use interfaces	Technological	Short	Technology providers Public administrations Students and lifelong learners Job seekers Recruiters and employers
Ensure access to and training on digital infrastructures for targeted stakeholders	Societal	Short	Public administrations Students and lifelong learners Job seekers
Leverage blockchain to address data source reliability and data integrity issues	Technological	Mid/Long	Funders and R&I activities’ supporters Researchers Technology providers
Clarify the legal framework around GDPR and create a roadmap to ensure compliance	Legal	Mid	Policy makers and regulators Funders and R&I activities’ supporters Researchers Technology providers
Establish a reliable way to connect people and institutions with an identity	Technological	Short/Mid	Policy makers and regulators Accreditation organizations Students and lifelong learners Job seekers Technology providers
Encourage European governments to formulate comprehensive blockchain adoption strategies	Methodological Procedural	Long	Policy makers and regulators Public administrations Funders and R&I activities’ supporters
Encourage European countries to establish coordinating authorities for public sector blockchain based projects	Methodological Procedural	Long	Policy makers and regulators Public administrations Funders and R&I activities’ supporters

below:

6. Conclusions and Next Steps

This section presents the conclusions that were drawn from the operation and execution of this pilot use case as well as the next steps for future research. Overall, it can be observed from the previous sections that this pilot was successful in showcasing the QualiChain technical results in a critical group of stakeholders belonging to a university. The main research question of this publication concerned the added value of a smart badge ecosystem in a university setting. The evaluation results showed that the smart badge accreditation functionality was one of the most successful functionalities of the platform. Undergraduate students were excited to see that through attending classes and completing assignments, they could improve their academic and professional profiles through validation of the skills they gain. The fact that this accreditation is officially performed by professors is also a strong point of QualiChain. Ph.D. students were also very satisfied with having a way to be validated for the various tasks that they perform for the university. Finally, professors agreed that smart badge accreditation could be integral in a university setting and were very active both in creating smart badges as well as awarding them.

When it comes to blockchain for certificate verification, while this functionality was not explored under the context of the pilot, the functionality was developed and used in some of the other pilot cases with successful results. However, it can be concluded that the process of certificate verification through blockchain requires the participation of the school in its entirety, connection of databases between the school and QualiChain, and the provision of specific APIs so that QualiChain can be connected with the official processes of the school. Future steps of the pilot regarding this will concern the verification of students’ certificates also for post-graduate education programs offered by the university, or for a Ph.D. position. This action will provide the opportunity to validate this functionality (that was already validated under the context of public sector pilots) in a university setting.

Participation of the school’s stakeholders in the execution of the pilot was very satisfactory with 90 students and 11 professors. Several workshops were organized and as such, the pilot was successful in engaging early users and showcasing the offerings of QualiChain. Based on the answers of the questionnaires, most users were satisfied with the functionalities catered to them and excited to see the application of innovative technologies, such as blockchain and data analytics, in a technical solution that is specifically catered to their needs.

However, one question that needs to be answered is whether society at large is ready to accept and use blockchain solutions. Based on our research, the integration of blockchain into education and human resource management seems to face several obstacles. Specific barriers we identified are technological immaturity, lack of knowledge and understanding, concerns for privacy, limited scalability, and lack of interoperability. Apart from these, there are also challenges that relate to cost, lack of infrastructure, need for training, security issues, and difficulty integrating blockchain with existing and often deprecated systems. At the organizational level, challenges such as lack of skills, financial barriers, and lack of commitment need to be tackled, especially in public universities that are characterized by a resistance to change. To address these challenges, one solution could be the creation of programs, courses, and vocational training activities so that learners can acquire the skills to develop blockchain applications and the knowledge to use them properly. Furthermore, cooperation between educational institutions, employers, and public authorities is essential to create a single certificate verification system. Despite the initial cost, implementing blockchain technology is considered to be a strategic investment with potential benefits in security, transparency, and efficiency. As a new technology, financial resources are required for research, development, and training, while funding is essential to create awareness and effectively promote the technology.

As far as the next steps regarding this pilot are concerned, the following actions will be followed:

• We will continue to promote the QualiChain project among students of the ECE School so that more students understand how the platform could benefit them. It is also believed that word-of-mouth dissemination of QualiChain from the early users of the platform will also help with its promotion.
• We will approach more professors that could be willing to test the component that provides recommendations for the restructuring of the school’s curriculum.
• Additional research will be performed to align the technological offerings of QualiChain and the results of the platform with current technological advancements and policy documents that are published by the European Commission.

Furthermore, employers and recruiting organizations are currently looking for indications of skills that reveal the ability to efficiently work remotely. The same is true for green skills that represent a growing need for future workers and towards combatting climate change. The QualiChain platform could facilitate the assessment of candidates regarding such skills since it supports the award of smart badges. It is believed that QualiChain, through this pilot, was successful in showcasing the needs of higher education and the labor market and has provided a sufficient solution to address such issues.