A Methodological Framework Based on a Quantitative Assessment of New Technologies to Boost the Interoperability of Railways Services

Abstract

Concerning the increase in the number of trips and tourists after the COVID-19 pandemic, TSPs (Transport Service Providers) and transport organizations are trying to improve their operability to answer the needs and expectations of passengers. This paper presents a methodology to assess and evaluate to what extent innovative technologies meet the needs of tourists and TSPs involved in the digital ecosystem for door-to-door trips in Europe, making railways and public transport more attractive and consequently encouraging people to use more intermodal solutions in public transport. In this study, two kinds of quantitative data are used: operational KPIs (Key Performance Indicators) and USI (User Satisfaction Index) surveys. The Effectiveness concept, as a metric of the capacity to meet these needs and expectations by the innovative technology, is calculated by merging both types of quantitative data. The method considers tourists’ socio-demographic profiles, allowing comparisons among TSPs and profiles for a specific technology, and it is extended to figure out correlations among variables through regression and Bayesian Networks analysis. In addition, specific socio-demographic data relevant to the needs and expectations were studied through the ANOVA test. This work belongs to the methodological framework of the IP4MaaS (Innovation Program 4 Mobility as a Service) project, which sets six demo sites on which this assessment method will be applied in a further stage. The concept of Effectiveness is applied in all the above-mentioned demo sites for the final assessment. Some IT innovations such as Location-Based Experience and Journey Planning have shown high Effectiveness. This work could be interesting for TSPs and IT (Information Technology) developers, researchers, policymakers, and organizations in the transport sector.

1. Introduction

While cities are increasing in their number of inhabitants, some rural areas are suffering from depopulation. According to the United Nations, around 75% of Europe’s population currently resides in urban areas, and the projection is that by 2050 this figure will be almost 84% [1]. Moreover, the population is becoming older, especially in developed countries: In 2018, 19.7% of the EU’s (European Union) population was aged 65 or over, representing a 2.6% increase since 2008 [2].

Changes in the living environment and age of the population, and effects on travel patterns, will need innovative solutions to fit the current needs better. Concerning the recent development and advancement of technologies, using clean public transport has become one of the main focuses of current environmental issues [1].

Digitalization and innovative technologies (IT) are boosting the use of public transport and MaaS (mobility as a service) solutions by citizens and tourists, and consequently it is becoming critical to work on assessment methodologies able to measure in a quantitative way the effectiveness of these IT solutions to reach the goal of increasing the use of public transport. On the other hand, the knowledge about what specific factors would increase this effectiveness, and what kind of socio-demographic profiles are more accepting of these innovations, is valuable information for continuous improvement.

This study aims to present a methodological framework to assess, in a quantitative way, how innovative technologies can respond to the needs of tourists and Transport Service Providers (TSP) involved in the digital ecosystem for door-to-door travel in Europe, making railways and public transport more attractive [3].

This paper assesses the needs of travelers belonging to different socio-demographic profiles and TSPs based on rail transport in a methodological way, addressing societal trends such as a reduction in Greenhouse Gas (GHG) emissions and road congestion [4].

The concept of the attractiveness of rail and public transport depends on complex psychological factors from a scientific and technical perspective [5]. This assessment methodology consolidates the concept of “user profile” [6] and the ability of the system to respond to needs and expectations, including sociodemographic-related aspects such as aging, reduced mobility, impairments, and other specific conditions [7].

Described as the Effectiveness of IT (Information Technology) solutions, the adaptability of data to satisfying the requirements of both travelers and operators, the equity of ITs deployed in society [8], and the potential acceptance by the market and, in particular, by railway operators and their ecosystem [9] are considered in the assessment methodology introduced in this work.

To assess the effectiveness of new technologies for boosting interoperability, a methodological framework for a quantitative assessment is needed. Such a framework should include indicators to measure the performance of different modes of transport, especially railway services, such as punctuality, reliability, capacity, and the satisfaction level of users.

In this paper, for the assessment of USIs concerning new IT services, online survey forms have been used to explore and identify what are the real needs from the users’ point of view, and what are valuable investments for these services to encourage people to use more transport services—especially railways.

This assessment methodology belongs to the methodological framework developed in the first phase of the H2020 Shift2Rail IP4MaaS project (2020–2023), which sets six demo sites (Barcelona, Padua, Athens, Liberec, Osijek, and Warsaw) on which this methodology will be applied in the second phase [10,11].

2. Literature Review and State of the Art

The Shift2Rail (S2R) Innovation Programme 4 (IP4) aims to build a digital ecosystem for door-to-door travel in a seamless, multimodal, and European-wide transport system based on railways. IP4 is expected to radically change the way people travel in Europe, making railways and public transport more attractive, and addressing key societal trends such as reductions in greenhouse gas (GHG) emissions and road congestion [12].

To make travelling around Europe easier, S2R IP4 Call for Members (CFM) projects such as ATTRACkTIVE [13], CO-ACTIVE [14], MaaSive [15] and CONNECTIVE [16], and Open Call (OC) projects ST4RT [17], My-TRAC [18], SPRINT [19], and RIDE2RAIL [20], developed (and are developing) a technical framework of sophisticated Information Technology (IT) building blocks that can be flexibly combined in multiple configurations into solutions that adapt naturally to multiple scenarios. In this context, IP4MaaS takes up the challenge of providing the individual IT solutions developed in IP4 CFM and OC projects, and consolidated by COHESIVE [21], by combining them into solutions for specific demonstration scenarios in multiple real environments across Europe.

The IP4MaaS project aims to advance the uptake of Mobility as a Service (MaaS) schemes by analyzing and testing the technologies developed under the Innovation Programme 4 (IP4) of the Shift2Rail Joint Undertaking in six demonstrations across Europe (Barcelona, Athens, Warsaw, Osijek, Liberec, and Padua) [12]. This paper introduces the methodology used for these assessments, and the results and discussions relevant to one of them (the Athens one).

As a first step, a literature review was conducted to evaluate the best assessment methodology [22]. Santarremigia et al. (2021) [22] combine several decision-making methodologies to provide a comprehensive and robust approach to evaluating new digitalization technologies. The use of the AHP method and the incorporation of multiple decision criteria, as well as the SWOT analysis and risk management plan, make the approach systematic and effective. The case study demonstrates the practical application of the methodologies and highlights the value of a structured decision-making approach for organizations looking to embrace digitalization. On the other hand, Molero et al., 2019 [23] utilized a qualitative research approach, with a case study methodology, applied to investigate the key factors that enable the successful implementation and integration of innovative information and communication technology (ICT) solutions in small and medium-sized enterprises (SMEs) and large companies involved in the multimodal transport of dangerous goods. The authors conducted interviews with experts in the field, and analyzed relevant literature and documentation.

In addition, Lopez-Lambas and Alonso (2019) [24] made a scientific methodology focused on the FGs (Focused Groups) approach to evaluate and assess the public perceptions and acceptability of driverless buses. In this research, they have tried to determine the level of acceptability of innovative technology among individuals, focusing on means of transport such as buses, and analyze the ideas and feedback of a group of individuals. Merton and Kendall (1946) [25] tried to develop a Focus Group technique as a way to study human behavior, undertake market analysis, assess product acceptance, and evaluate the structure and organization of companies and social issues. The analysis of focused groups is capable of generating qualitative information and exploring the perceptions, thoughts, needs, and expectations of individuals in terms of travel behavior in different modes of public transport [26]. Moreover, in the focused group approach, for the sake of diversity and equity, different socio-demographic profiles such as age, gender, and income were considered. It is highly advisable to analyze more than one Focus Group in more than one location to get more accurate data.

Furthermore, Korkmaz et al. (2017) [27] conducted a methodology to investigate the acceptance of autonomous public transport by tourists. This assessment was undertaken through an online survey based on a five-point Likert scale, which consisted of three parts: the first part was an introductory text about the topic of the survey, “autonomous public transport”, the second part consisted of survey questions to assess the experience of tourists with autonomous public transport, with two aims (intention and willingness to use different autonomous means of transport), and the third part collected socio-demographic data at the end of surveys.

The methodology introduced by this paper involves quantitative (operational KPIs) and qualitative data (User satisfaction surveys), which combine the strengths of previous methodologies applied to assess the acceptability of new IT solutions. The combined computation of operational KPIs gathered from the performance of the IT solution while being used by users (tourists and operators), jointly with satisfaction indicators gathered through surveys filled in by these same users, provides a final quantitative metric regarding the benefits offered by this IT solution applied to public transport, and especially railways [23]. Increasing the possibility of carrying out specific assessments focused on users’ profiles, such as age, disability or gender, based on their particular needs and expectations, so as to build equity via future improvements of IT solutions based on the results of this assessment, is one of the main objectives of this study. The previous literature has focused on expert scoring (AHP and VOLERE) and operational KPIs handled through data analysis methods such as the Bayesian network [22,23], but these do not take into account the satisfaction levels of real users.

This updated methodology brings novelty by including a User Satisfaction Index (USI), which is based on the needs and expectations of real users with the aim of assessing the degree to which there are equal and fair opportunities for everyone to access and use these new technologies.

This new approach enables the assessment methodology to mirror reality, as it is not only a “desk” assessment, but instead operates in the real world. In other words, this methodology subjects the evaluation process to real-world conditions, as it involves on-site investigations and not just an analysis based on theoretical information.

3. Methodology

The methodology used in this paper sets out the concept of a “demonstration scenario” [10], which is the intersection of a new technology offered to travelers and the TSP that is offering it. “Effectiveness” is a metric that reflects how this new technology meets the needs and expectations of these travelers and TSPs (Transport Service Providers), from the perspective of an aggregated analysis of the profiles of travelers via intersectional analysis [28].

To calculate the “Effectiveness” of new technology in a specific “demonstration scenario”, two kinds of quantitative data, one objective and another subjective, are required [29].

3.1. Objective Data: Operational KPIs

The concept of Operational KPI is defined as a quantitative and objective operational indicator that measures the benefits of new IT via the services provided by new technology for a specific TSP and a specific traveler profile. These operational KPIs will be automatically collected through the application of this new technology in the field.

Each KPI is measured in a specific unit, and it will be converted into a dimensionless KPI, with a value between 0 and 1, by dividing each KPI linked to a specific IT service by the maximum value measured for this KPI among all TSPs. The KPI is defined such that that the higher the value, the better, so a dimensionless value close to 1 will always be better than a dimensionless value close to 0 [11].

For each demonstration scenario “D”, several User Journeys “i” can be considered with a different travel solution [30] for traveling from an origin to a destination through the combination of several means of transport (TSP “k”) [31]. A list of KPIs is proposed based on the technological capabilities of the TSP “k” for the integration of the new IT service “j” [10], as depicted in Figure 1. The process of following the flowchart (Figure 1) will start from the travel solution” which is offered in a user journey “i” from its origin to its destination. In the next step, the technological capabilities of TSP “k” with IP4 functionality "j” will be checked. As it is shown in Figure 1, if the TSP “k” has the capability of handling and performing functionality “j”, it will be integrated and two conditions will occur. If the answer to the aforementioned sentence is “Yes” a list of operational KPIs for Demonstration scenario “D”, User journey “i”, TSP “k” and IP4 functionality “j” will be prepared. However, on the other hand, if the answer to the initial sentence is “No”, no KPIs for the TSP “k” and functionality “j” will be produced. The process of this flowchart is shown in the following figure (Figure 1).

3.2. Subjective Data: USI Surveys

Two USIs, one for travelers and another for TSPs, measure opinions about a specific new technology’s utility [4]. These indexes are calculated based on answers to surveys.

The methods of the assessment and evaluation of satisfaction levels in USI surveys considering user needs and expectations are divided into six levels: strongly disagree (lowest satisfaction), disagree, neither agree nor disagree, agree, strongly agree (highest satisfaction) and no opinion (lack of opinion) [8,28]. In addition, this evaluation collects the socio-demographic information (illustrated in Appendix C) of the traveler to conduct an intersectional analysis [8,32].

The USI questionnaire for TSPs contains questions about the satisfaction of the TSP with the new integrated technology [33].

The USI questionnaire for travelers is structured into two sections:

Part A—Questions about needs and expectations. This questionnaire includes a specific section with questions for all profiles and a second section with profile-based questions, which will only be answered by users that select this profile in the sociodemographic questionnaire (see Appendix C).

Part B—Sociodemographic questionnaire (Appendix C) for travelers that will support the analysis of satisfaction for each profile vector “r”, as a set of sociodemographic characteristics of the traveler [10]. This sociodemographic questionnaire is divided into eight characteristics: age, gender, professional status, residential area, income, travelling with a dependent person, having a problem, disability or impairment, and familiarity with technology, especially mobile phones.

The USI surveys of TSPs and travelers and the sociodemographic questionnaire are shown in Appendix A, Appendix B, Appendix C.

The option “no opinion” was offered in the survey because respondents unfamiliar with a new IT service are likely to choose a neutral response [34], and this may be confounded with those respondents who are familiar with and/or informed about the new IT service choosing a neutral response.

In the USI surveys, all questions have a score. A question will not be given any score only in cases in which the traveller has not used the IT service or has answered the question with “no opinion”.

The satisfaction index of travelers belonging to profile vector “r” with the new IT service “j” offered by the TSP “k” is calculated as:

$${\mathrm{USI}}_{{\mathrm{Traveller}}_{\mathrm{rjk}}}=\frac{{{\displaystyle \sum }}_{\mathrm{w}=1}^{{\mathrm{m}}_{\mathrm{rjk}}}{{\displaystyle \sum }}_{\mathrm{v}=1}^{{\mathrm{n}}_{1\mathrm{jk}}+{\mathrm{n}}_{2\mathrm{jk}}{}^{\mathrm{r}}}{\mathrm{Score} \mathrm{question}}_{\mathrm{wv}}}{{\mathrm{m}}_{\mathrm{rjk}}·\left({\mathrm{n}}_{1\mathrm{jk}}+{ \mathrm{n}}_{2\mathrm{jk}}{}^{\mathrm{r}}\right)·5}$$

(1)

where Score question_wv is the score given to the question “v” by the respondent “w”; n_1jk is the number of questions applicable to all the profiles measuring satisfaction with the new IT service “j” offered by the TSP “k”; n_2jk^r is the number of questions applicable only to profile “r”, measuring satisfaction with the IT service “j” offered by the TSP “k”; m_rjk is the number of respondents to the USI questionnaire belonging to the profile “r”, measuring the satisfaction with the IT service “j” offered by the TSP “k”.

The satisfaction index of a TSP “k” regarding an IT service “j” is calculated as:

$${\mathrm{USI}}_{{\mathrm{TSP}}_{\mathrm{jK}}}=\frac{{{\displaystyle \sum }}_{\mathrm{v}=1}^{{\mathrm{n}}_{\mathrm{j}}}{\mathrm{Score} \mathrm{question}}_{\mathrm{v}}}{{\mathrm{m}}_{\mathrm{jk}}.{\mathrm{n}}_{\mathrm{j}}·5}$$

(2)

where “Score question” is the score given to the question number “v” and “n” is the number of questions in the USI questionnaire belonging to a specific IT service “j” offered by the TSP “k”; m_rjk is the number of respondents to the USI questionnaire measuring the satisfaction with the IT service “j” offered to the TSP “k”.

In both equations, a 5 is used to normalize and obtain a value between 0.2 and 1, because the answer to each question has a value between 1 (representing the minimum satisfaction) and 5 (representing the maximum satisfaction) [11].

Regarding the satisfaction level found in USI surveys, one of the options is N/A (not applicable). The N/A option will be applied to those respondents (travelers or TSPs) with no experience of testing the specific IT service. If the traveler or TSP has not tested a specific IT service, the “$Score questio{n}_{wv}$” value from “$v=1$” to “$n_{1jk}+n_{2jk}{}^r$” for the traveller or the “$Score questio{n}_v”$ value from “$v=1$” to “$n_j$” for the TSP will be set as zero.

3.3. The Concept of Effectiveness

The Effectiveness of an IT service “j” offered by a TSP “k” for a specific profile “r” in demonstration scenario “D” is calculated through the following equation:

$$Effectivenes{s}_{rjk}=\frac{{{\displaystyle \sum }}_{n=1}^{N}KP{I}_{n_{jk}}+US{I}_{Travele{r}_{rjk}}+US{I}_{TS{P}_{jk}}}{N+{\delta }_{Traveller}+{\delta }_{TSP }}$$

(3)

where

$$\begin{gathered} \{\begin{array}{c}{\delta }_{Traveller}=0 \mathit{\text{if}} US{I}_{Travele{r}_{rjk}}=0\\ {\delta }_{Traveller}=1 \mathrm{if} US{I}_{Travele{r}_{rjk}}\ne 0\end{array} \\ \{\begin{array}{c}{\delta }_{TSP }=0 \mathit{\text{if}} US{I}_{TS{P}_{jk}}=0\\ {\delta }_{TSP }=1 \mathit{\text{if}} US{I}_{TS{P}_{jk}}\ne 0\end{array} \end{gathered}$$

“N” is the number of operational dimensionless KPIs linked to the IT service “j” offered by the TSP (Transport Service Provider) “k” (N can be zero for some IT services),$KP{I}_{n_{TS{P}_{jk}}}$ is the value of the KPI “n” belonging to the IT service “j” offered by the TSP “k”,$US{I}_{Travele{r}_{rjk}}$ is the value calculated in Equation (1)$, and US{I}_{TS{P}_{jk}}$ is the value calculated in Equation (2).

The comparison of effectiveness can only be performed after grouping based on what parameters are considered in the Effectiveness formula: KPIs, USI Travelers, USI TSPs, or combinations among them [11]. For example, Effectiveness could be grouped in the following way:

Group 1—KPIs.

Calculating Effectiveness for IT service (J) considering only the value of operational KPIs;

Group 2—KPIs + Travelers.

Calculating Effectiveness for IT service (J) considering the value of operational KPIs and USI travelers;

Group 3—KPIs + TSPs.

Calculating Effectiveness for IT service (J) considering all values of operational KPIs and USI TSPs;

Group 4—KPIs + travelers + TSPs.

Calculating Effectiveness for IT service (J) considering all values of operational KPIs, USI travelers, and USI TSPs.

Given that the Effectiveness is dimensionless, with a value between 0.2 and 1, the higher the score, the better, and different demonstration scenarios “D” can be compared to analyze how the needs of travelers in different locations or demo sites are matched by the same innovative technology “j” offered by different TSPs.

The three elements in the nominator are summed in a linear way and with equal weight because an innovative technology with no good operational KPIs, no good acceptance level by tourists, or no good acceptance level by the TSP would not be implemented in practice, or would not remain in use for a long time, as it would not be meeting users’ needs. However, this linear distribution of weights among KPIs and USIs in the Effectiveness formula could be a limitation of the methodology, due to the different amounts of data available for each term, which will necessitate some refinements in the future when these data are available and handled in the IP4MaaS project (2020–2023) [35].

Effectiveness will be used in practice as a quantitative metric assessing the benefits that new technologies bring to users, tourists, and TSPs. Low values of Effectiveness for specific IT services, when compared between several TSPs or different traveler profiles, will show weaknesses in the degree to which these new IT services meet the needs and expectations of tourists and TSPs, which will allow IT developers to implement improvements based on these results, and boost the interoperability of railway services by creating more equitable IT services.

3.4. Extension of the Methodology

In addition, this methodology based on quantitative data (USIs and operational KPIs) could be extended to other data analysis tools, such as the following.

Regression analysis, used to identify pairwise correlations between variables. These pairwise correlations between couples of variables let us define fixed connections in Bayesian network analysis to derive more accurate results [18].

Bayesian network analysis. This analysis is used to figure out correlations among variables in a network based on USI questionnaires and operational KPIs [36,37].

To determine if some sociodemographic profiles are relevant to specific criteria, an ANOVA analysis was performed as a statistical way to compare different groups. The ANOVA test for this case study will be done using the data collected from the USI questionnaire for travelers, while the TSPs USI surveys will not be considered as they do not depend on sociodemographic profiles. Moreover, the ANOVA test can help identify the significant differences in the satisfaction level associated with each sociodemographic characteristic, and it can be developed in Excel version 2019 [36].

4. Results and Discussions after the Application of the Methodology to a Case Study in Athens

The demonstration scenarios in this study are AS-IS and TO-BE scenarios, and they have been used to highlight the benefits provided by the technologies available in the IP4 ecosystem. AS-IS scenarios describe current travel experiences in a demo site. The scenarios are defined through a user journey template, supporting the identification of flow, steps, actions, personal preferences, and decisions users make during the travel. The collected information supports the drawing of a user journey map, providing a clear picture of existing problems and areas of improvement. On the other hand, TO-BE scenarios show how the technologies available in the IP4 ecosystem could be used to overcome the current limitations identified in the AS-IS scenarios. Additionally, in this case, the scenarios are defined by filling in a user journey template to highlight the flow, steps, actions, and touchpoints of the new travel experience. The collected information provides: (i) the definition of one or more new user journey maps drawing a clear picture of new traveler experiences, and (ii) the identification and detailed description of specific travel experiences included in the user journey map that represent potential use-cases for IP4MaaS demonstrations. After the identification of user journey scenarios, the most common travel solutions (description of the most common travel solutions currently adopted by users in an itinerary considering both private vehicles and services offered by all the TSPs operating in the demonstration site) and travel solutions supported by IP4MaaS (description of the different travel solutions available, considering private vehicles and services offered by the TSPs involved in IP4MaaS) can be identified [38]. This methodology was applied to assess future IT services in a demo site in Athens, considering four TSPs (OASA, MIRAKLIO, Taxiway, and Brainbox).

The demonstration in Athens took place in an urban environment, and included multiple modes throughout July 2022. The PTOs (Public Transport Operators) and TSPs involved in the IP4MaaS Athens demonstration site were OASA (which is the planning authority responsible for coordinating and financing the public transport system in the Athens metropolitan area, covering buses, trams, trolleys and the metro (three lines)), MIRAKLIO (which is the public transport operator responsible for the buses operating within the Municipality of Heraklion, Attica), BRAINBOX (which is a company offering bike and car-sharing services) and TAXIWAY, which provides taxi services [39].

After employing a User Engagement Strategy based on incentives, the authors were able to collect 17 travelers willing to test these IT innovations for tourists, who thus provided the sample in this data analysis.

4.1. IP4MaaS Functionalities for Travellers and TSPs

The functionalities that are listed in Section 4.2 and Section 4.3 belong to the Travel Companion Application, and they were generated in the framework of the IP4 program in Shift2Rail. Future functionalities and operational KPIs were considered for the use case of Athens via the methodology.

4.2. Functionalities for Travellers

• Journey planning function: The function of finding routes involving different modes of transport.
• Booking function: This function reserves and purchases both specific tickets and multimodal tickets [40].
• Issuing function: The function providing online tickets that can be validated and inspected through a mobile application.
• Location-based experience (LBE) function: A function used to discover entertainment services, such as quiz games or commercial offers provided during the trip.

4.3. Functionalities for TSPs (Transport Service Providers)

• Asset manager: The tool used to insert TSP services and facilities into the IP4 platform [35].
• Location-Based Experience tool: The tool that allows for building Location-Based Experiences for the user.

4.4. List of Operational KPIs

The list of operational KPIs measuring the increase in the benefits accrued by the TSP and the traveler caused by integrating innovative technologies with the TSP’s services [41] is given in

Table 1. List of operational KPIs per functionality.

Number	Innovative Technology (IP4)	KPI	Units
1	Journey Planner (JP)/Offer Builder	Number of modes of transport used in the trip (multimodality)	Number per day
2	Journey Planner (JP)/Offer Builder	TSP web services acting as JP integrated into the IP4 ecosystem	Number of TSPs integrated
3	Journey Planner (JP)/Offer Builder	A successful proposal for customers (due to the integration of transport modes)	Number of travel solutions shown per day
4	Journey Planner (JP)/Offer Builder	Available travel solutions for customers (due to the integration of transport modes)	Number of travel solutions shown per day
5	Booking	Number of offers booked per day	Number of offers booked per day
6	Issuing	TSP web services for the issuing process integrated into the IP4 ecosystem	Number of TSPs integrated
7	Issuing	Successful issuing of multimodal travel solutions	Number of issues per day
8	Location-Based Experiences (LBE)	Number of users using the entertainment services	Number of users per day
9	Location-Based Experiences (LBE)	Time using the entertainment services	Number of seconds per connection

. The innovative technologies listed in this table are new functionalities developed in IP4 according to the Shift2Rail program [42]. These operational KPIs were automatically collected from the cloud system of the IP4 functionalities.

4.5. USI Travellers’ Survey

The quantity of each type of question was set by the following coding method.

1

“r” refers to the profile vector in travelers’ USI:

• (r = 1) refers to all profiles;
• (r = 2) refers to unemployed people, low-income people, retired people, and students;
• (r = 3) refers to disabled or impaired people, people with physical or mental illnesses, people in wheelchairs, people with reduced mobility, and people with visual impairments and hearing impairments;
• (r = 4) is the elderly;
• (r = 5) is women.

These profiles were addressed in this case study as the sentiment analysis showed that these profiles were especially sensitive to satisfaction regarding these functionalities [43].

2

The functionality assessed by the USI survey is shown as “j”;

3

“k” stands for the TSP (Transport Service Provider), assessed in each demo site;

4

“q” is the number of questions belonging to the functionality “j.

The USI survey for travelers used in this case study can be consulted in Appendix A.

In total, the number of responding USI travelers was seventeen.

Table 2. Definitions and values of each code relating to USI travelers.

Code	Definition	Value
USI_Traveller_r1_J1_K1	Calculation of USI traveler regarding journey planning taking OASA as the TSP for all profile vectors	0.6333
USI_Traveller_r2_J1_K1	Calculation of USI traveler regarding journey planning taking OASA as the TSP for low-income people	Undefined due to lack of data
USI_Traveller_r3_J1_K1	Calculation of USI traveler regarding journey planning taking OASA as the TSP for disabled people	Undefined due to lack of data
USI_Traveller_r4_J1_K1	Calculation of USI traveler regarding journey planning taking OASA as the TSP for elderly people	Undefined due to lack of data
USI_Traveller_r5_J1_K1	Calculation of USI traveler regarding journey planning taking OASA as the TSP for women	0.65
USI_Traveller_r1_J8_K1	Calculation of USI traveler regarding Location-Based Experience taking OASA as the TSP for all profile vectors	0.7533

addresses the USI for travelers [36] calculated in this use case, along with codes and their meanings and values. Considering the results extracted from the data analysis, the tourists in the “general” profile vector were the most satisfied with the Location-Based Experience functionality provided by the TSP OASA.

The USI surveys in this methodology apply several questions to all profiles (r = 1 to 5), and other specific questions that are applicable only to specific profiles, such as low-income people (r = 2), disabled people (r = 3), the elderly (r = 4) and women (r = 5). As a result, the User Satisfaction Index (USI) values for all profiles are similar to the USI values for the aforementioned specific profiles, but with slight differences, which show changes in satisfaction when tourists belong to these specific profiles. Even though the methodology has the ability to consider different specific profiles (low-income people, disabled people, the elderly, and women), no data regarding low-income (r = 2) and disabled people (r = 3) were collected through the surveys. As a result, in the following table (Table 2), there is no value assigned for these two profiles.

The USI survey for travelers used in this case study can be consulted in Appendix A.

4.6. USI TSPs’ Survey

As regards the USI TSPs, a total of seven responses were gathered.

Table 3. Definitions and values of each code regarding USI TSPs.

Code	Definition of Variables	Value
USI_TSP_8_1_8	Calculation of USI TSP regarding Location-Based Experience for OASA	0.72
USI_TSP_8_1_13	Calculation of USI TSP regarding asset manager for OASA	0
USI_TSP_8_2_8	Calculation of USI TSP regarding Location-Based Experience for MIRAKLIO	0.76
USI_TSP_8_2_13	Calculation of USI TSP regarding asset manager for MIRAKLIO	0
USI_TSP_8_3_8	Calculation of USI TSP regarding Location-Based Experience for Taxiway	0
USI_TSP_8_3_13	Calculation of USI TSP regarding asset manager for Taxiway	0.9

introduces which USI for TSPs were calculated in this use case, along with codes, their meanings and their values [36]. The results extracted from the data analysis suggest that the TSP Taxiway were the most satisfied with the asset manager tool.

The USI survey for TSPs used in this case study can be consulted in Appendix B.

4.7. Calculation of Effectiveness

In

Table 4. Results of Effectiveness for general profiles.

Name of Variable	Definition of Variables	Value
Effectiveness_r1_J1_K1	Effectiveness value for all profile vectors considering journey planning functionality provided by OASA	0.51127
Effectiveness_r1_J1_K2	Effectiveness value for all profile vectors considering journey planning functionality provided by MIRAKLIO	0.02450
Effectiveness_r1_J1_K3	Effectiveness value for all profile vectors considering journey planning functionality provided by Taxiway	0.05392
Effectiveness_r1_J1_K4	Effectiveness value for all profile vectors considering journey planning functionality provided by BrainBox	0.02941
Effectiveness_r1_J2_K3	Effectiveness value for all profile vectors considering booking functionality provided by Taxiway	0.33333
Effectiveness_r1_J8_K1	Effectiveness value for all profile vectors considering Location-Based Experience functionality provided by OASA	0.92476
Effectiveness_r1_J8_K2	Effectiveness value for all profile vectors considering Location-Based Experience functionality provided by MIRAKLIO	0.82285
Effectiveness_r1_J13_K1	Effectiveness value for all profile vectors considering asset manager functionality provided by OASA	0.33333
Effectiveness_r1_J13_K2	Effectiveness value for all profile vectors considering asset manager functionality provided by MIRAKLIO	0.11111
Effectiveness_r1_J13_K3	Effectiveness value for all profile vectors considering asset manager functionality provided by Taxiway	0.33333
Effectiveness_r1_J13_K4	Effectiveness value for all profile vectors considering asset manager functionality provided by BrainBox	0.22222

, the results of implementing the effectiveness formula considering the USI TSPs, USI Travelers, and KPIs value are illustrated [18].

The abovementioned table (Table 4) introduces the Effectiveness values derived for the profile vector, functionality, and TSP. As regards Effectiveness, the most effective profiles are “general” profiles, with the Location-Based Experience provided by TSPs “OASA” and “MIRAKLIO” taking values 0.92 and 0.82, respectively, out of 1. Regarding the comparison of Effectiveness groupings, which has been discussed in Section 3.3, the Effectiveness value r1_J8_K1 was given to the group of operational KPIs, TSPs, and travelers, whereas the Effectiveness value r1_J8_K2 was given to the group of operational KPIs and TSPs.

4.8. Results Related to the Extension of the Methodology

4.8.1. Regression Analysis

The regression analysis in this case study was developed in Julia’s programming language software version (1.7.0), and was based on scores for variables collected through USI questionnaires launched online [36].

All variables showing a high correlation with others are listed in the first column of

Table 5. Analysis of the correlation level between each pair of variables.

Variables	Variables Highly Correlated with (p < 0.05)
J1K1q1	J1K1q3, J8K1q3
J1K1q3	J1K1q1, J8K1q3
J8K1q1	J1K1q1, J8K1q2, J8K1q3
J8K1q2	J1K1q3, J8K1q1
J8K1q3	J1K1q1, J1K1q3
J8K1KPI2	J1K1KPI7
J1K1KPI7	J8K1KPI0, J8K2KPI0, J8K1KPI1, J8K2KPI1, J8K1KPI2, J8K2KPI2, J8K1KPI3, J8K2KPI3, J8K1KPI4, J8K2KPI4, J13K1KPI3, J13K2KPI3, J13K3KPI3, J13K4KPI3, J1K1KPI4, J2K3KPI8

, while the list of variables correlated with each of them is listed in the second column.

The abovementioned table indicates that these variables are highly correlated (the p-value is less than or equal to 0.05). In the regression analysis, overall, 23 variables and 17 observations (responses from USIs) were analyzed.

Table 6. Definition of variables’ codes.

Variable Name	Definition of the Variable Name
J8K2KPI4	Total number of connections in the evening regarding Location-Based Experience for MIRAKLIO
J13K1KPI3	Number of services integrated with the pilot regarding asset manager for OASA
J8K1KPI1	Number of experiences launched during the demo regarding Location-Based Experience for OASA
J8K2KPI0	Number of entertainment services offered during the demo regarding Location-Based Experience for MIRAKLIO
J8K2KPI3	Total number of connections in the morning regarding Location-Based Experience for MIRAKLIO
J8K2KPI2	Average time of connection (in seconds) regarding Location-Based Experience for MIRAKLIO
J8K2KPI1	Number of experiences launched during the demo regarding Location-Based Experience for MIRAKLIO
J8K1KPI3	Total number of connections in the morning regarding Location-Based Experience for OASA
J8K1KPI0	Number of entertainment services offered during the demo regarding Location-Based Experience for OASA
J1K1KPI7	Average number of shopping offers regarding journey planning for OASA
J2K3KPI8	Average number of booking offers regarding booking for Taxiway
J8K1KPI2	Average time of connection (in seconds) regarding Location-Based Experience for OASA
J8K1KPI4	Total number of connections in the evening regarding Location-Based Experience for OASA
J13K3KPI3	Number of services integrated with the pilot regarding asset manager for Taxiway
J1K1KPI4	Average number of modes involved in the journey regarding journey planning for OASA
J13K4KPI3	Number of services integrated with the pilot regarding asset manager for Brainbox
J13K2KPI3	Number of services integrated with the pilot regarding asset manager for MIRAKLIO
J8K1q1	Question about general satisfaction regarding Location-Based Experience provided by OASA
J1K1q3	Question about cost-saving regarding journey planning provided by OASA
J8K1q2	Question about time-saving regarding Location-Based Experience provided by OASA
J8K1q3	Question about cost-saving regarding Location-Based Experience provided by OASA
J1K1q1	Question about general satisfaction regarding journey planning provided by OASA

shows all variables handled by this regression analysis and the meaning of each code. To make each variable more easily identifiable in the data analysis, a unique code has been applied to each of them (Table 6). The code “j” identifies functionality, the code “k” specifies the name of TSPs, and the code “q” refers to the question associated with each functionality. Questions related to the USI survey for TSPs were not considered, as insufficient data were available for appropriate data analysis.

Focusing on services offered by OASA,

Table 7. Graphical representation of the correlation between the tourist’s desire to save and the desire to receive high-quality services.

Variables	J1K1q1	J1K1q3	J8K1q1	J8K1q2	J8K1q3
J1K1q1	0	1.00 × 10−13	0.7184	0.7184	1.00 × 10−13
J1K1q3	1.00 × 10−14	0	0.9316	0.9316	1.00 × 10−13
J8K1q1	1.00 × 10−14	NaN	0	1.00 × 10−13	1.00 × 10−13
J8K1q2	NaN	1.00 × 10−13	1.00 × 10−13	0	NaN
J8K1q3	1.00 × 10−13	1.00 × 10−13	NaN	NaN	0

shows a graphical representation of the correlation between the tourist’s desire to save, through the variables (i) time-saving with journey planning (J1K1q3) and (ii) comfort with Location-Based Experience (J8K1q3), and their desire to receive high-quality services, through the variables (i) general satisfaction with journey planning (J1K1q1), (ii) general satisfaction with Location-Based Experience (J8K1q1), and (iii) time-saving with Location-Based Experience (J8K1q2), highlighting strong correlations in grey (p < 0.05).

From what has been discussed above (Table 5 and Table 7), it can be concluded that there is a high correlation between the general satisfaction resulting from journey planning amongst travelers (J1K1q1), cost-saving by journey planning (J1K1q3) and cost-saving by Location-Based Experience (J8K1q3). The reason for these high correlations might be found in the importance of the costs of the trips for tourists, or the commercial offers that they receive during their journey. These high correlations can be observed in another example, where the general satisfaction of travelers related to Location-Based Experience (J8K1q1) is correlated with the general satisfaction related to journey planning (J1K1q1), as well as the time-saving and cost-saving associated with Location-Based Experience (J8K1q2 and J8K1q3). In addition, in the case of the correlation between KPIs, it can be seen that there is a high correlation between the average number of shopping offers related to journey planning (J1K1KPI7) and the variables of Location-Based Experience, such as the number of entertainment services (J8K1KPI0), the number of launched experiences (J8K1KPI1), the average time of connections (J8K1KPI2), the total number of connections in the morning and evening (J8K1KPI3 and J8K1KPI4), the number of integrated services (J13K1KPI3), the average number of modes involved in the journey (J1K1KPI4) and the average number of booked offers (J2K3KPI8). This correlation illustrates that there is a high interest amongst tourists in using commercial offers, entertainment, or quiz games while purchasing their tickets during journey planning.

4.8.2. Bayesian Network Analysis

For the Bayesian network analysis, the K2 metric was used to achieve the network with the maximum likelihood, according to Equation (4).

$$Likelihood=P\left(B_S,D\right)=P\left(B_S\right)·{\displaystyle \prod }_{i=1}^n{\displaystyle \prod }_{j=1}^{q_i}\frac{\left(r_i-1\right)!}{\left(N_{ij}+r_i-1\right)!}·{\displaystyle \prod }_{k=1}^{r_i}{N}_{ijk}!$$

(4)

where $\mathit{P}\left({\mathit{B}}_{\mathit{S}}\right)$ is the probability of obtaining the network ${\mathit{B}}_{\mathit{S}}$ (initially all ${\mathit{B}}_{\mathit{S}}$ values are considered to have the same probability, so $\mathit{P}\left({\mathit{B}}_{\mathit{S}}\right)$ is 1/number of possible networks); $\mathit{n}$ is the number of variables; ${\mathit{r}}_{\mathit{i}}$ is the number of possible values for each variable ${\mathit{x}}_{\mathit{i}}$; ${\mathit{q}}_{\mathit{i}}$ is the number of possible configurations (given the network ${\mathit{B}}_{\mathit{S}}$) of the variable ${\mathit{x}}_{\mathit{i}}$; ${\mathit{N}}_{\mathit{i}\mathit{j}}$ is the number of times variable $\mathit{i}$ adopts a specific configuration $\mathit{j}$—for example, if we consider a network ${\mathit{B}}_{\mathit{S}}$ where the variable $\mathit{i}$ has two parameters, ${\mathit{N}}_{\mathit{i}\mathit{j}}$ will be the number of times these parameters have the configuration $\mathit{j}$ or specific values (e.g., 1 and 5) [44]; ${\mathit{N}}_{\mathit{i}\mathit{j}\mathit{k}}$ is the number of times that variable $\mathit{i}$ adopts a specific value (e.g., 2) when the variables adopt the configuration $\mathit{j}$(e.g., 1 and 5). For example, when the two parameters’ variables adopt values 1 and 5, variable $\mathit{i}$ adopts 2 times the value 2. In this case, ${\mathit{N}}_{\mathit{i}\mathit{j}\mathit{k}}$ = 2. The Bayes network with the maximum likelihood according to Equation (4), determined after considering 1500 possible Bayes networks, is shown in Figure 2. Calculations were done in the Julia program, version (1.7.0).

The results of this analysis of correlation between variables in a network are shown in Figure 2.

The aforementioned figure (Figure 2) clarifies that the main variable is the total number of connections in the evening by Location-Based Experience (J2K2KPI4), and the average number of modes involved in journey-by-journey planning (J1K2KPI4 and J2K3KPI4) is directly dependent on the main variable. Furthermore, the variable cost-saving by journey planning (J1K1q3) is directly dependent on the cost-saving by Location-Based Experience (J8K1q3), general satisfaction, and time-saving by journey planning (J1K1q2 and J1K1q1). In addition, the number of modes involved in journey-by-journey planning (J1K4KPI4) is directly linked to the average number of shopping offers by journey planning (J1K4KPI7) and the number of services integrated with the pilot by the asset manager (J13K2KPI3 and J13K4KPI3). Additionally, the number of services integrated with the pilot by the asset manager (J13K4KPI3) is linked to the number of experiences launched in the demo by Location-Based Experience (J8K2KPI1), the average number of booked offers regarding booking (J2K3KPI8), the number of entertainment services offered in the demo by Location-Based Experience(J8K2KPI0), the total number of connections in the morning (J8K2KPI3) and the average time of connection (in seconds) by Location-Based Experience (J8K1KPI2).

4.8.3. ANOVA Analysis

The results of the ANOVA test, which was developed using Excel, are shown in

Table 8. Results of the ANOVA test concerning the significance of sociodemographic variables.

	Variables
Socio-Demographic Variables	J1K1q1	J1K1q2	J1K1q3	J8K1q1	J8K1q2	J8K1q3
Age
Gender
Residential area
Income
Travelling with dependent person
Professional Status
Disability
Familiarity with technology

, where differences per profile regarding satisfaction with each coded variable are analyzed. If the p-value is less than or equal to 0.05, the difference in the scoring between the ranges of a specific sociodemographic variable is significant (grey color); alternatively, if the value is higher than 0.05, it is not significant (white color) [36].

The abovementioned table (Table 8) indicates that the satisfaction related to the benefit linked to the functionality Location-Based Experience offered by the TSP OASA is significantly different for the people who live in urban areas in comparison with the people who live in suburban or rural areas (shown in grey color in the Table 8). This could be because of the performance of this functionality, since Location-Based Experience involves presenting commercial offers or entertainment services to the travelers.

On the other hand, the satisfaction regarding the benefits linked to the functionality of journey planning offered by the TSP OASA is significantly different between people with and without disabilities(shown in grey color in the Table 8). This could be related to the aim of this functionality, which is to make the trip more convenient and comfortable for disabled people by allowing them to choose the most accessible route from the available options, which will increase their level of safety. As a result, the variable of time-saving with the journey planning functionality provided by OASA has a high significance for disabled people.

4.9. Limitations of the Study

Achieving a sufficient response sample size for different sociodemographic profiles strongly affects the results of the data analysis carried out in this study. Sometimes, insufficient and limited participation can have negative impacts in the assessment phase, and may lead to a decrease in the accuracy of the data analysis. Generally, the required sample size for each module of the regression analysis, Bayesian network analysis, and ANOVA test should be considered in the quantitative analysis. It should be taken into account that, to derive accurate and precise results in data analysis, a sufficient sample size for both general and specific profiles should be achieved.

5. Conclusions and Further Developments

A methodological assessment framework for quantitatively measuring to what extent a specific new technology meets the needs of tourists and TSPs is presented in this work.

With this aim, a new basis for selecting demonstration scenarios on which the assessment will be conducted was introduced, using two quantitative types of data, Operational KPIs and USIs, which allow for the calculation of the Effectiveness of a specific innovative technology offered by a TSP to a profile group of tourists.

The Effectiveness index is dimensionless, and ranges between 0.2 and 1 (the higher the score, the better); it measures how an innovative technology matches the needs and expectations of its users, travelers, and TSPs. According to this research, the top three values of Effectiveness were those relating to the Location-Based Experience functionality provided by OASA and MIRAKLIO, and the journey planning functionality provided by OASA, respectively.

This methodology allows further data analyses based on the quantitative data gathered during each demo, such as regression analysis, Bayesian networks, and ANOVA.

The methodology has been applied to a case study in the Athens demo site, focusing on four TSPs (OASA, MIRAKLIO, Taxiway, and Brainbox); several new functionalities for TSPs and tourists were assessed for their Effectiveness, and compared.

In this use case of Athens, three analyses were carried out.

The regression analysis aimed to identify the pairwise correlation between the variables discussed in this study, and provided more accurate results in the case of correlation among variables. In the regression table, if two variables are highly correlated, for example J1K1q1 (general satisfaction with journey planning provided by OASA) and J8K1q3 (cost-saving of Location-Based Experience provided by OASA), it means the value of this correlation is less than or equal to 0.05, which means a 95% confidence level, or less than 5% chance of being wrong. On the other hand, if two variables are not correlated, such as J8K1q2 (time-saving via Location-Based Experience provided by OASA) and J1K1q2 (time-saving via journey planning function provided by OASA), it means the value of this correlation is more than 0.05, which mean a less than 95% confidence level, or more than 5% chance of being wrong. As a result, the highest correlation in the regression analysis was found between the general satisfaction with the journey planning function provided by OASA and the time-saving offered by journey planning with OASA, and the general satisfaction and comfort offered by the Location-Based Experience provided by OASA.

Concerning the BN analysis, the most important variables have been identified. These variables are questions focused on finding more secure routes in off-peak hours to improve journey planning, questions about a safe trip from a COVID-19 perspective to improve journey planning, questions about general satisfaction regarding Location-Based Experience, questions about time-saving regarding Location-Based Experience, questions about cost-savings regarding Location-Based Experience, questions about cost-savings regarding journey planning, questions about general satisfaction regarding journey planning, and questions about time-saving regarding journey planning. According to this study, the top three variables in the BN analysis were the total number of connections in the evening by Location-Based Experience and the average number of modes involved in journey-by-journey planning provided by MIRAKLIO and Taxiway. As an additional conclusion of this Bayesian network analysis, we can forecast that the selection of safe and non-crowded routes has been a priority for tourists since the COVID-19 pandemic.

ANOVA analysis was performed as a statistical approach to comparing different groups, and thus discovering whether sociodemographic profiles are relevant to specific criteria. This analysis tried to identify significant differences in the satisfaction level per each sociodemographic characteristic, and it has been found that out of eight sociodemographic variables, two of them (“disability” and “residential area”) are significant variables. This actively demonstrates that any changes in the two variables time-saving by journey planning provided by OASA and the average number of modes involved in the journey-by-journey planning provided by OASA will have a significant effect on people with disabilities in their sociodemographic profiles. On the other hand, for the characteristics related to residential area, the most important variable is cost-saving regarding Location-Based Experiences provided by OASA.

Future works will apply this quantitative assessment methodology to other demo sites with multiple demonstration scenarios set by the H2020 Shift2Rail IP4MaaS project (2020–2023), so as to validate its benefits, proceed with the required refinements, and explore its potential. This will enable us to analyze the impact of these innovative IT functionalities on the number of tourists and travelers using railways in a post-COVID 19 environment.