On the Usability of a Modeling Language for IoT-Based Public Transportation Systems

Abstract

Internet of Things (IoT)-based public transportation systems face distinct challenges within the broader realm of IoT. Developers of such systems encounter a notably intricate development environment compared to general IoT systems, which are inherently characterized by elevated levels of complexity and heterogeneity. As successfully applied in other domains, domain-specific modeling languages (DSMLs) can also be employed to facilitate the development of IoT-based public transportation systems and address the challenges mentioned. Hence, in this study, a novel model-driven engineering (MDE) methodology is presented, comprising the steps of using a DSML, called DSML4PT, for the development of a wide-range of IoT-based public transportation applications. Moreover, the usability evaluation of DSML4PT within this MDE methodology during the real applications of IoT-based public transportation systems is also provided, which is missing in similar studies. For this purpose, we investigated the usability of DSML4PT within a systematic evaluation approach in which the features of DSML4PT are assessed both quantitatively and qualitatively in eight different real public transportation applications with the participation of experienced developers. Comparative analysis revealed that approximately 80% of IoT-based public transportation systems could be automatically generated through modeling exclusively employing DSML4PT. In contrast to the conventional software development methodologies, the novel DSML4PT approach also decreased the time required for the development of public transportation applications by almost 50%. In addition, according to a questionnaire-based assessment, the general evaluation rating of the language was measured as 4.44 over 5-point Likert scale. Feedback from the developers corroborated the practicality of this language and its widespread adoption across diverse perspectives.

1. Introduction

Currently, various systems, including fare collection, pollution measurement, passenger counting, vehicle system management, passenger information, advertisement management, and vehicle tracking systems are extensively utilized in public transportation. These systems may operate independently or in conjunction with one another. Public transportation networks are present in diverse settings such as buses, subways, trains, and trams. Within these contexts, the systems are subjected to harsh environmental conditions such as extreme temperature differences, continuous vibrations, rain, and sunlight. Moreover, the majority of devices within public transportation are inherently mobile systems.

The Internet of Things (IoT) is being used in various domains, including smart cities, public transportation, industry, manufacturing, agriculture, and healthcare [1]. IoT entails a network wherein entities are connected with one another and with the broader Internet. Objects in this context encompass any unit capable of connecting to the Internet and exchanging information. In IoT, these entities are interconnected and capable of data exchange. Key advancements facilitating the evolution of IoT include the miniaturization of circuits, reduced energy consumption, and enhanced processor performance [2].

Nowadays, the utilization of IoT systems within public transportation has become increasingly prevalent. The challenges encountered in IoT-based public transportation systems closely resemble those observed in general IoT contexts. Heterogeneity stands out as a significant challenge in both public transportation IoT systems and their general IoT counterparts. Hardware and software components exhibit a high degree of interdependence, often resulting in platform-dependent code development, a common occurrence within these systems. Adapting developed code to run on a different platform performing the same function, even when employing the same programming language, proves to be notably challenging. This inherent issue is a common thread across both public transportation IoT systems and general IoT systems.

Furthermore, IoT systems deployed within public transportation settings face additional challenges specific to this domain. These challenges encompass disparities among IoT systems, systems operating within dynamic physical environments, exposure to harsh environmental conditions, utilization of domain-specific System-on-Chip (SoC) technologies, adoption of domain-specific hardware, and adherence to domain-specific design standards [3].

Across all IoT-based public transportation systems, domain-specific standards govern communication, management, and control mechanisms among constituent units. Foremost among these standards is ITxPT, which fosters a standardized framework across public transportation systems [4]. The adoption of such standards distinguishes IoT-based public transportation systems from their general IoT counterparts.

IoT-based public transportation systems encompass highly mobile infrastructures situated within urban environments. Given this mobile nature, the selection and management of wireless network systems have paramount importance. During system design, myriad challenges emerge, including connectivity issues inherent to mobile structures, as well as the management of system disconnections and reconnections. The establishment of solutions to address these challenges renders IoT-based public transportation systems distinct from general IoT systems [5].

Designing public transportation IoT systems necessitates consideration of operation within harsh temperature and vibration conditions. To ensure seamless long-term operation under such conditions, the use of domain-specific System-on-Chips (SoCs) and platforms becomes indispensable. While easily accessible platforms like Raspberry Pi and Arduino are prevalent in the market, they are unsuitable for operation within harsh environments. Instead, semiconductor manufacturers advocate for SoC platforms such as ARM core, iMX [6], and Sitara [7], which are tailored for deployment in public transportation and automotive applications.

Public transportation systems may involve various subsystems including IoT components. Notably, fare collection systems represent one of the critical subsystems wherein passengers are charged for transit services. Fare collection methods vary, including Radio Frequency (RF) card, credit card, Quick Response (QR) code, and smart mobile phone payments. For payment security, IoT systems integrate security features such as the Secure Access Module (SAM) card. The design of these payment systems requires specialized structures distinct from those of general IoT systems.

The Controller Area Network (CAN) line structure is employed in vehicle management systems, which constitute a subset of public transportation IoT subsystems. While in a general IoT system, data can be sourced from sensors in analog or digital form, in the context of public transportation systems, data are received from sensors via the CAN line. Various sensor data, such as engine speed, fuel level, and vehicle velocity, are transmitted over the CAN line. Administrators use this received data for analysis and display purposes [8].

Light-emitting diode (LED) panels are extensively utilized in IoT systems deployed in public transportation, notably in applications such as passenger information and advertising management. These panels serve to display information regarding upcoming stops and display advertisements to passengers. Specific protocols tailored for public transportation govern the operation of LED panels [9]. The implementation of these protocols differs from that of general IoT systems.

IoT-based public transportation systems, exemplified here, exhibit numerous distinctions from general IoT systems. Consequently, the utilization of model-driven IoT development tools (see Vorto [10], IoTDraw [11], SoML4IoT [12] as examples and [13] for an extensive review of their features) designed for general purposes directly in the context of public transportation can be inefficient and incomplete due to the nature of the IoT-based public transportation systems as discussed above. Addressing this gap necessitates research efforts aimed at developing vehicle models specialized for public transportation [3]. Several investigations, albeit in the nascent stage and often confined to case studies, have commenced to address public-transportation-specific challenges [14,15,16].

As successfully applied in many other domains, model-driven engineering (MDE), largely supported by the use of domain-specific languages (DSLs) [17,18] and domain-specific modeling languages (DSMLs) [19,20], may also help overcome the above-mentioned difficulties of IoT-based public transportation applications and facilitate the system development by leveraging the abstraction level during design and generating many components automatically for the complete implementation, i.e., software models of the public transformation systems can be automatically manipulated and transformed to the executable and deployable artifacts [3]. For this purpose, a modeling language, called DSML4PT, was recently introduced in [5].

DSML4PT is the pioneering and currently the sole modeling language that facilitates model-driven engineering (MDE) of IoT-based public transportation systems by considering different modeling perspectives [5]. It also enables the generation of necessary system software for deployment and execution on various IoT-based public transportation devices. However, the usability evaluation of this language, considering the real applications of IoT-based public transportation systems, is lacking. Within this context, the main contributions of our study can be summarized as follows:

• A modeling language with a complete set of syntax and translational semantics definitions is presented for the MDE of IoT-based public transportation applications.
• DSML4PT is supported with an Eclipse-based open-source Integrated Development Environment (IDE) to design and implement such public transportation applications composed of various SoC and hardware components running on different embedded operating systems.
• A novel MDE methodology is defined, comprising the steps of using DSML4PT modeling language in conjunction with its IDE for the development of public transportation systems.
• A systematic evaluation, covering various quantitative and qualitative usability aspects, is provided to determine how MDE with DSML4PT facilitates system development compared to traditional development approaches.

To assess how this language impacts application development, including its potential to expedite the process and its value for developers, we conduct a study. We systematically evaluate DSML4PT’s usability using both quantitative and qualitative measures. This assessment involves examining the language’s features in eight real public transportation applications with the assistance of experienced developers. This paper presents the systematic evaluation in its entirety, including the application methodology, the achieved results, and their discussion.

The remainder of this paper is structured as follows: Section 2 provides a review of related work on the MDE of IoT-based public transportation systems. Section 3 and Section 4 briefly introduce DSML4PT language features and its development environment and describe the MDE process based on DSML4PT. Section 5 presents the comprehensive multi-case evaluation of DSML4PT, while the conclusions are given in Section 6.

2. Related Work

Related work is discussed under two sub-sections. First, the general challenges encountered within the realm of IoT systems and the use of model-driven techniques to address these issues are explored. Subsequently, the research efforts focusing on the MDE of IoT-based intelligent transport and public transportation are discussed.

2.1. General Model-Driven IoT Studies

The design and implementation of IoT systems incorporating interconnected objects capable of remote monitoring and control are naturally complex [2]. Specifically, developers of such systems encounter several common challenges, including heterogeneity [21], device availability [22], data volume [23], and security and privacy [22,23].

Model-driven approaches offer potential solutions for addressing these challenges in IoT system development [24,25,26]. Numerous studies employing model-driven techniques are documented in the literature, focusing on various problems and proposing corresponding solutions [13].

One notable approach that targets fundamental IoT challenges such as distribution and heterogeneity is the Internet of Things Modeling Language (ThingML) approach [24,27]. ThingML supports visual modeling constructs like state charts and component diagrams, enabling the modeling of IoT applications from diverse perspectives using a platform-dependent modeling language, ranging from architectural to device behavior levels. It supports platform-dependent code generation for various platforms, including Arduino, Raspberry Pi, Intel Edison, and popular programming languages (C/C++, Java, JavaScript). Sharaf et al. [28] present the CAPSml framework, which can translate any CAPS-SAML model into the ThingML language. Similarly, the Vorto project [10] creates the Vorto DSL, which is employed to define manufacturer-independent abstraction layers describing the functionalities and characteristics of devices at different levels of granularity [29].

There are also studies aimed at ensuring self-adaptation during system runtime. For instance, in [30], software development automation and reusability are explored using MDE techniques. Additionally, a different study [23] specifically addresses the engineering of mission-critical IoT systems, focusing on requirements such as reliability, safety, and security. These requirements present further challenges, which can be addressed through model-based verification techniques [31].

The model-driven development framework known as UML4IoT, designed for industrial IoT production systems, is introduced in [32]. This approach, grounded in the Unified Modeling Language (UML), leverages IoT components within production systems. Various model-driven IoT approaches in the industrial domain are also documented in the literature [33,34,35,36,37,38]. For instance, Muthukumar et al. [33] present an architecture for the Industrial Internet of Things (IIoT) along with a MDE approach for designing, validating, and automatically generating code for control applications in the manufacturing sector. Liebel et al. [34] evaluate the current state and challenges encountered in applying model-driven techniques to embedded IoT systems through a survey study. In [35], an IoT platform and associated prototypes developed under a project are described, with a focus on industrial applications. Albers et al. [36] introduce a novel method for incorporating IoT features based on the system model of a reference product, utilizing an ‘IoT canvas’ approach to create an IoT-specific view of the reference product’s system model. Conzon et al. [37] present a framework and methodology supporting fully decentralized, aggregative, and dynamically intelligent collaboration among heterogeneous IoT platforms in the industrial sector. Ahmed et al. [38] propose a general, modular, and extensible interoperability architecture based on modeling principles suitable for industrial applications.

COMFIT [39] encompasses an application development paradigm utilizing a model-driven infrastructure, where the cloud interface of an application management and execution module is employed for developing automated IoT applications. The ENACT DevOps (EDO) framework [15,40], rooted in the DevOps concept, aims at fostering close collaboration between developers (Dev) and operational teams responsible for deploying and operating software systems (Ops). In [41], a solution to address model-driven communication heterogeneity through abstraction layering is discussed. This approach relies on a networking language and a rule-based language for modeling and controlling the behavior of the IoT network.

Several studies propose a methodology for a leveraging model-driven and service-oriented architecture (SOA) to develop software components for IoT [42]. They offer varying levels of abstraction for constructing IoT software components using SOA. In IoT, network resources are dynamically allocated using application programming interfaces (APIs). Various studies in the literature adopt the SOA structure [11,12,16,43]. Studies in [11,12] aimed to provide an executable modeling language and development environment for SOA-based IoT systems. For the SOA structure, a technique consisting of four stages with different levels of abstraction, perspective, detail, and service orientation is proposed in [43].

The conceptual development framework introduced in [14] facilitates the separation of IoT applications into different topics, with support for stakeholders and developers. This framework abstracts the complexities of scalability and heterogeneity. Brambilla et al. [44] proposed a model-driven approach for defining user interface components and design patterns specific to the IoT domain.

In the realm of wireless sensor networks (WSN), operating systems like TinyOS and Contiki have gained widespread adoption, with hardware nodes and software platforms becoming standard. The utilization of model-driven techniques in WSN has proven more effective compared to other IoT domains, with numerous studies addressing this aspect documented in the literature [45,46,47,48].

Highly specialized metamodels are employed in systems engineering, particularly in IoT and embedded systems, including firmware development. Kühlwein et al. [49] introduce a novel associative modeling language designed to integrate numerous custom metamodels, enhancing firmware synthesis for ultra-thin IoT devices through model integration.

The literature also encompasses studies focusing on IoT system management [50,51,52,53]. GeneSIS [50] offers a tool-supported framework for continuous management and deployment of IoT systems. It addresses heterogeneity across IoT, edge, and cloud levels, facilitating management and continuous deployment of software systems on corresponding infrastructures. The extended version of GeneSIS, incorporating security and privacy concerns [52], is introduced with risk control mechanisms [54] to enhance reliability. Centurión et al. [51] present a model-driven approach to systems management, bridging visualization and automation requirements, offering a successful solution to IoT network and system management challenges.

In [55], a MDE tool named CHESSIoT is introduced, which combines high-level visual design languages, software development, security analysis, and deployment approaches for multi-layered IoT systems. Boutot et al. [56] propose UCM4IoT as a modeling environment for uncovering and specifying the requirements of IoT systems. UCM4IoT takes into account the heterogeneity of IoT systems and provides DSL constructs to model various aspects of IoT. Erazo-Garzón et al. [57] suggest using a DSL called Monitor-IoT, supporting developers to design multi-layer monitoring architectures for IoT systems with high abstraction, expression, and flexibility.

A model-driven development approach is defined in [58] for simulating, generating code, and deploying IoT systems. This approach enables the design of complex IoT simulation environments and their deployment without coding. Another DSL-based simulation approach is proposed in [59] for designing, generating code, and simulating IoT systems deployable to the FIWARE infrastructure.

Boutot and Mustafiz [60] propose IoTMoF as a model-driven framework for designing and generating code for IoT systems. IoTMoF supports platform-independent modeling with the ReqMIoT environment, covering use-case modeling and creating IoT ARM-compliant domain models. The use-case model addresses exceptional behaviors alongside processing functionality.

Cloud and fog computing concepts are commonly utilized in IoT systems, playing a crucial role in determining the computational placement within IoT architectures. Moreover, these systems facilitate abstraction levels and diminish hardware dependencies. Model-driven approaches have been proposed for both cloud systems [61,62,63] and fog systems [64] to address these requirements.

Furthermore, various model-driven IoT studies are conducted across different domains. These include rapid prototyping [65], smart home applications [66], civil healthcare [67], real-time operating systems (RTOS) [68], smartphone alarms [69], smart city solutions [70,71], door access control systems [72], and automotive systems [73]. Additionally, developments in complex event processing (CEP) programs [74], machine learning [75], personal data privacy [76], flexible communication using MQTT [77], fleet management with numerous devices [78], edge device computation systems [79], reactive approaches [80], and distributed smart applications [81] are also available in the literature. Studies utilizing high-level modeling tools such as Papyrus [82], and SysML and Capella [83] for the model-driven IoT systems also exist.

Table 1. General model-driven IoT studies.

Study	Target Domain	IDE/Tool Provided	Use Case Included	Evaluation
ThingML [24,27]	General IoT Systems	Yes	Yes	No
CAPSml [28]	General IoT Systems	No	No	No
Vorto [10]	Digital Twins	Yes	Yes	No
MDE4IoT [30]	General IoT Systems	Yes	Yes	No
MC-IoT [20]	Mission Critical IoT Systems	No	No	No
UML4IoT	Industrial IoT Production Systems	Yes	Yes	Yes
Muthukumar et al. [32]	Manufacturing	No	Yes	No
Khaleel et al. [35]	Car Manufacturing	Yes	Yes	No
Albers et al. [36]	Industrial IoT Systems	Yes	Yes	No
Ahmed et al. [38]	Smart Gas Distribution	No	Yes	Yes
COMFIT [39]	General IoT Systems	Yes	Yes	Yes
CyprIoT [41]	IoT Networks	Yes	Yes	No
Sosa-Reyna [42]	SOA for IoT	No	No	No
SoaML4IoT [11,12]	SOA for IoT	Yes	Yes	No
Patel and Cassou [14]	General IoT Systems	Yes	Yes	Yes
Brambilla et al. [44]	General IoT Systems	Yes	Yes	No
DSML4TinyOS [45]	Wireless Sensor Networks	Yes	Yes	Yes
Özkaya and Örs [46]	Wireless Sensor Networks	No	No	No
Vorapojpisut [47]	Wireless Sensor Networks	No	No	No
Asici et al. [48]	Wireless Sensor Networks	Yes	Yes	No
Kühlwein et al. [49]	Firmware Development	Yes	Yes	No
GeneSIS [50,52]	IoT System Management	Yes	No	No
Muntes-Mulero et al. [54]	IoT System Management	No	Yes	No
Centurión et al. [51]	IoT System Management	Yes	Yes	No
CHESSIoT [55]	Multi-layered IoT Systems	Yes	Yes	Yes
UCM4IoT [56]	General IoT Systems	Yes	Yes	No
Monitor-IoT [57]	Multi-layered IoT Systems	Yes	Yes	No
SimulateIoT [58]	IoT Simulation	Yes	Yes	No
SimulateIoT-FIWARE [59]	IoT Simulation	Yes	Yes	No
IoTMoF [60]	General IoT Systems	Yes	Yes	No
Alulema et al. [53]	IoT System Management	Yes	No	No
Cai et al. [61]	IoT Cloud Systems	Yes	Yes	No
Luo et al. [62]	IoT Data Systems	No	No	No
Pusztai et al. [64]	IoT Fog Systems	Yes	No	No
Xiao et al. [65]	Rapid Prototyping	Yes	Yes	Yes
Kölsch et al. [66]	Home Applications	No	Yes	No
Kotronis et al. [67]	Civil Healthcare	No	Yes	No
Gomes and Baunach [68]	Real-time Operating Systems	No	Yes	No
Tufail et al. [69]	Smart Phone Systems	Yes	Yes	No
García-López et al. [74]	Complex Event Processing	Yes	Yes	No
ThingML+ [75]	Machine Learning	No	No	No
Michael et al. [76]	Personal Data Privacy	Yes	No	No
Dalibor et al. [77]	Cyber-Physical Systems	Yes	No	No
Dautov and Song [78]	Fleet Management	Yes	Yes	No
Betancourt et al. [79]	Edge Device Computation	Yes	No	Yes
Dias et al. [80]	Reactive Approaches	No	No	No
Jahed and Dingel [82]	High-level Modeling	No	No	Yes
Sundharam [83]	High-level Modeling	Yes	Yes	No

lists all above-mentioned general studies for model-driven development of IoT applications. As can be seen, although they provide various noteworthy approaches for modeling IoT, they do not address the problems inherent in modeling IoT-based public transportation systems and primarily offer MDE methods and/or tools only applicable to the domains other than public transportation. In addition, many of them do not provide an evaluation of their languages/tools/approaches, and the use of the supported features is mostly exemplified just with case studies.

2.2. MDE Studies for IoT-Based Intelligent Transportation and Public Transportation

In the domain of intelligent transportation, which may have co-working spaces with public transportation, only a very limited number of studies exist for proposing modeling or discussing the case studies of applying MDE for IoT-based intelligent and/or public transportation systems.

The work by Rafique et al. [16] combines feature-driven design with model-driven SOA frameworks. They present a system capable of generating platform-specific code, exemplified through a case study conducted on a smart transportation system. However, the focus primarily revolves around traffic management and road safety, with no consideration given to public transportation systems, and the developed framework lacks specialized structures tailored to address issues inherent to such systems.

A model-driven systems engineering approach within the context of smart city IoT systems is discussed in [71]. The approach covers modeling the entire system, including elements like traffic lights, sensors, and lighting systems as well as addressing concerns such as complexity management and synchronization between subsystems. It is underscored that IoT plays a crucial role in public transportation, and leveraging model-driven methodologies can streamline the development of IoT systems.

Iovino et al. [72] present a case study involving an integrated card-based access control system employing near-field communication (NFC) tags for user authorization. A model-driven methodology is proposed, facilitating semi-automatic code generation within an IoT infrastructure. The focus of this approach lies in addressing pass authorization within the NFC domain. However, the NFC system under consideration is strictly viewed as a means for access control at entry points, lacking specific adaptation for integration into public transportation systems. Notably, NFC systems akin to those discussed can potentially be employed in fare collection systems within public transportation. Yet, the structure described in this study fails to cater explicitly to the needs of public transportation systems. Within public transportation, NFC deployment often encompasses fare reduction mechanisms alongside specialized hardware configurations such as secure access modules (SAM), ensuring the integrity of the fare collection process. Moreover, NFC payment constitutes merely a fraction of the broader fare collection ecosystem.

Judvaitis et al. [15] highlight the challenges encountered while employing the ENACT tool within a testbed environment, primarily attributed to the heterogeneity prevalent across IoT platforms. Offering a DevOps approach, ENACT addresses various IoT challenges not only during development but also in operational settings. Although the testbed utilized in this study finds potential application in intelligent transportation systems, the focus remains on train control via IoT technologies, with no explicit mention of public transportation concerns.

Henze et al. [63] introduce a tool designed to fulfill cloud-based privacy requisites, primarily focusing on general IoT challenges. While the study mentions potential applications within intelligent transportation systems, no specific development endeavors or problem delineations are considered. Similarly, the study conducted by [14] introduces a system addressing general IoT challenges. It is suggested that this system could potentially be adapted for wireless sensor network (WSN) applications within the transportation domain. However, no specific details are provided, leaving it as a mere conceptual proposal. There is a notable absence of discussions pertaining to transportation or public-transportation-specific issues.

Cueva-Fernandez et al. [73] present Vitruvius, a platform enabling users without programming expertise to design and swiftly implement sophisticated web applications leveraging real-time data sourced from interconnected vehicles and sensors. The system offers a model-driven development environment tailored for deployment on vehicles, aimed at centrally collecting and monitoring sensor data such as CO₂ levels, speed, and location, irrespective of vehicle type (e.g., automobiles, trucks). Nonetheless, there is a lack of focus on addressing challenges within any particular facet of public transportation. This study primarily revolves around the collection of sensor data from mobile devices via a platform.

Neisse et al. [31] propose a model-driven toolkit for IoT devices, integrated into a management framework, supporting the specification and efficient evaluation of security policies to safeguard user data. An IoT framework is introduced for smart cities, emphasizing user data protection. However, the primary objective remains the introduction of a generic IoT development framework. Although a smart city case study encompassing general concepts is included, transportation is only mentioned as a component within the broader system. Modeling specifically pertaining to the transportation or public transportation domains is not covered.

An innovative model-driven toolkit, BRAIN-IoT, for general IoT development is presented in [37]. This toolkit utilizes a semantic discovery service to dynamically select and locate available resources or devices, offering developers a flexible environment with a graphical interface for constructing hybrid applications. Although the toolkit is primarily developed for industrial applications, it is suggested that it could potentially be applied to public transportation. However, no specific details or examples are provided, and there is a notable absence of discussions concerning transportation-specific issues.

A model-driven approach is outlined in [81] to design effective and efficient protection mechanisms, encompassing real-time, non-functional features such as security and performance within IoT system components. Although it is mentioned that this new methodology can also be applied to the issues of verifying data distribution in intelligent transport systems, there is no discussion of how the modeling capabilities exactly support eliminating these issues and uncertainties. Finally, Moadad et al. [84] introduce a Quality of Experience (QoE) framework with a primary focus on IoT-enabled devices within smart cities. However, there is a lack of specific investigation dedicated to public transportation systems in detail.

Considering all these studies on the MDE of IoT-based public transportation systems (summarized in

Table 2. MDE studies of IoT-based intelligent transportation and public transportation.

Study	IoT-Based Public Transportation Modeling Features	Supporting Platform/OS/Language	IDE/Tool Provided	Use Case Included	Usability Evaluation
Rafique et al. [16]	Traffic and Road Safety	Cloud, REST Services	Yes	Yes	No
Hause et al. [71]	Traffic Lights, Lighting Systems	Cloud, Scada Systems	Yes	Yes	No
Iovino et al. [72]	Fare Collection	Secure Access Modules, Near-Field Communication, Tinkerforge	Yes	Yes	No
ENACT [15,40]	Train Control	EDI Test Bed	Yes	Yes	No
Henze et al. [63]	Cloud and Privacy	Cloud	Yes	No	No
Patel and Cassou [14]	Wireless Sensor Networks	Android, MySQL, Java	Yes	Yes	Yes
Vitruvius [73]	Sensor Tracking	Web	Yes	Yes	Yes
Neisse et al. [31]	Security	Raspberry Pi, PandaBoard, Java, C/C++	Yes	Yes	No
BRAIN-IoT [37]	Security and Privacy	Cloud	Yes	Yes	No
Mazzini et al. [81]	Mission-Critical Systems	CHESS	Yes	No	No
Moadad et al. [84]	Quality of Experience	-	No	No	No
Our Study	All above	Wide range of platforms, languages and OS including above via model trans.	Yes	Yes	Yes

), it can be seen that different modeling viewpoints and public transportation modeling features supporting high mobility, standards and protocols specific for public transportation, various SoC and hardware components being used, as well as different underlying operating systems, platforms, and languages, are not covered altogether, which in fact limits the variety of the implementation platforms and hence rarely supports the automatic generation of the executables for the wide range of applications that will be deployed on different IoT-based public transportation systems. More importantly, the practical evaluation of the proposed MDE solutions and/or modeling approaches is not covered by most of these studies. Usability is mostly exemplified with simple case studies without any systematic evaluation of how MDE facilitates the design and implementation of such public transportation systems.

3. Overview of DSML4PT Language

As previously introduced, DSML4PT is a modeling language whose syntax is based on a metamodel that enables modeling IoT-based public transportation applications according to different viewpoints and facilitates the MDE of such applications for different public transportation platforms. This section briefly discusses the main components and features of this language. Complete specifications of DSML4PT’s syntax and semantics definitions, as well as its modeling and code generation features, can all be found in [5].

The metamodel, which constitutes the abstract syntax of DSML4PT, provides the connections and main elements of public transportation systems, including IoT technologies. Related metamodel entities and their relations are derived based on an examination of various public transportation specifications (such as [4,85,86]) and the requirements of developing IoT software for public transportation systems discussed in [3]. The metamodel is divided into three different viewpoints, named IoT Core, Public Transport, and Service/Cloud.

DSML4PT’s IoT Core viewpoint includes elements (such as Controller, Port, Actuator, Sensor) that define a generic IoT device, regardless of the characteristics of public transportation devices. The Public Transport viewpoint includes elements that describe the unique features of a public transportation system. In this viewpoint, the difference between public transportation devices from general IoT devices is defined at the metamodel level. It contains elements and relationships that enable the modeling of the domain. As indicated in [5], DSML4PT’s Public Transport viewpoint supports special hardware, standards, and protocols created for general use starting from the processor level. Microprocessor families working with the operating system are reflected in the metamodel in great detail from this point of view. Finally, the Service/Cloud viewpoint of the language defines the elements such as ServiceApp, MonitorApp, SmartPhone for service and cloud-based mass transit. They are specialized for using the services of simple microcontrollers. In this perspective, there are smart phone and computer application elements that provide the monitoring of public transportation IoT devices.

The graphical concrete syntax that supports modeling public transportation systems according to the DSML4PT’s metamodel was created based on the well-known Sirius open-source software (Version 11.7) project of the Eclipse Foundation [87]. Sirius allows users to easily create a graphical modeling environment using Eclipse modeling technologies, mainly Eclipse EMF and GMF. A modeling environment built with Sirius consists of Eclipse editors that allow users to create, edit, and view EMF models. Hence, DSML4PT model instances can be created visually for each of the required viewpoints that match the DSML4PT concrete syntax and stored as Ecore files that can be further parsed and interpreted for the model-to-text transformations.

Figure 1 shows a screenshot from the integrated development environment (IDE) of DSML4PT. Developers can drag and drop the required meta-entity instances listed in the palette on the right-hand side of the editor. The palette provides all modeling elements for each of the language’s viewpoints. Modeling in this IDE is also empowered with automatic checking of model constraints and validation rules according to the DSML4PT language semantics. Hence, wrong modeling element relations, restrictions on the number of relationships between elements in the instance models, and confirmation on the source and target of a modeling relation can be controlled. Moreover, DSML4PT includes some editorial constraints that assist the user in the design while creating the system models, such as keeping the unity of the whole system and its components by switching between editors of different viewpoints and keeping the integrity of the elements and the relationships in case of any addition or removal. It is worth indicating that all of these constraint checks are made in real time during system modeling without any intervention from the developers, i.e., appropriate error messages are shown to the user when a constraint check fails while the developer works on modeling the system. In addition to the restrictions and features described above, more than 60 rules were defined for the model validation according to the DSML4PT language’s semantics, and execution of these rules inside the IDE indicates an “Error” that must be fixed or a “Warning” that needs attention. The complete set of all these model validation rules can be found in [5].

The translational semantics of DSML4PT was provided by creating various model-to-text (M2T) transformation rules written in the Acceleo language. Acceleo is a pragmatic implementation of the OMG MOF M2T Language (MTL) standard. It also provides an Eclipse plugin where Acceleo transformations can be written, parsed, checked, and executed directly inside the Eclipse environment. DSML4PT’s M2T rules are applied at runtime on the public transportation system models conforming to the DSML4PT syntax, and hence the code and/or files required for the system implementation are generated automatically. A developer does not need to know the whole generation process including the details and the execution mechanism of the transformation rules, i.e., the whole code generation process is abstract and there is no need for any human intervention.

4. MDE with DSML4PT

In this section, we first describe the MDE methodology in which developers can design and implement IoT-based public transportation systems by using both the DSML4PT language and its IDE. Then, we discuss a case study on applying this methodology to develop a vehicle management system.

Figure 2 shows the steps followed in the proposed MDE methodology. In Step 1, a developer creates DSML4PT instance models according to the IoT Core, Public Transport, and Service/Cloud viewpoints by using DSML4PT concrete syntax. As discussed in Section 3, DSML4PT IDE does not only offer a graphical design for system modeling, but also supports various real-time automatic controls on modeling elements and relations which guide designers in creating accurate models. The main outcome of this step is DSML4PT models of the public transportation system under development. These models conform to the DSML4PT metamodel specifications, so each can be also referenced as an instance of the DSML4PT metamodel.

The next step (Step 2) of the methodology is the model validation process, in which the definitions of the system components and the construction of the relation items in the system models are all checked and validated according to the semantics of the DSML4PT language. For this purpose, the model validation rules are applied to the DSML4PT model instances automatically when the developer selects the model validation function via the menu provided inside the DSML4PT IDE. Errors and warnings (if they exist in the model) are shown for further modifications on the designed models.

Step 3 consists of the code generation from DSML4PT instance models. Validated public transformation system models (which are the output of the previous step) are converted into the code for the targeted applications. M2T transformation rules written in Acceleo are automatically executed on the instance models and the corresponding code is obtained for implementing the public transformation system. As indicated in Section 3, the whole process in this step is automatic and abstract from the developer. The only thing required from the developer is to initiate this code generation process by selecting the appropriate function from the menu provided again in the IDE.

Based on the specifications of the system to be implemented, auto-generated code obtained as the output of Step 3 can be further completed with manually added delta code in Step 4 to achieve the full implementation of the targeted public transportation application. Both models and code of the final application are all available in the project repository of DSML4PT IDE.

To give some flavor of applying the above MDE methodology, we now consider the design and implementation of a vehicle management application using the DSML4PT IDE. This application controls and monitors some critical sub-functions of a public transportation vehicle such as global positioning, door operations, and fuel consumption. The application will be installed on a device having an iMX8 series ARM core microprocessor (NXP Semiconductor, Eindhoven, The Netherlands). The related system also has 8 GB of main memory and 16 GB of hard disk space. Noteworthy components modeled as sensor inputs within the system include the ignition key, GPS location input, and CAN communication line. Additionally, there exists a MAX9768 audio output circuit, interfaced via inter-integrated circuit (I2C), and a 15″ liquid crystal display (LCD) output, operated through low-voltage differential signal (LVDS) interface, serving as actuators within the system. Furthermore, the GSM module utilized in this setup is the HE910 module (Telit Cinterion, Irvine, CA, USA).

Conforming to Step 1 of our methodology, the device’s public transportation system models are created graphically inside the DSML4PT IDE. The palette is used by the developer to create the models of the components and their relations required for the vehicle management application. Figure 1, previously given in Section 3 to illustrate DSML4PT IDE, also shows the model instance for the IoT Core view of our example system including the fundamental input, output, sensor, and actuator components and their relations. Moreover, specialized CAN hardware tailored for public transportation is used, and required elements are also included in the model. Figure 3 shows the Public Transport viewpoint modeling, encompassing all structures tailored for public transportation, and adopts the BalenaOS operating system, featuring the Wayland window system with C++ language. The Vehicle Management system transmits data to the cloud system via Azure Cloud Services. Within the cloud environment, a Vehicle Management Admin Application to be coded in Java language and an Android smartphone application facilitating system management are housed. Figure 4 portrays the system viewpoint that encapsulates the Service/Cloud modeling aspects.

After modeling the application according to the different system viewpoints, model instances are validated by executing the second step of the methodology. When the developer selects the “Validate Diagram” menu item inside the DSML4PT IDE, the model is automatically checked according to the semantics rule of the modeling language, and any errors and warnings are shown to the developer. Figure 1 shows a validation error found in the public transportation model of the vehicle management device. According to the PubTrans_1 rule of the DSML4PT semantics (see [5] for the definition of all rules applied for the validation of DSML4PT model instances), the Architecture node within the Public Transport diagram should have a name but it does not exist in the current model. Hence, DSML4PT IDE promptly identifies this deficiency within the model element and notifies the software developer accordingly. All such validation errors should be fixed prior to completing the model design.

By following Step 3 of the MDE methodology, code generation from the designed models of the public transportation system is possible. So, model instances for our case study are given as input to the M2T transformations inside the IDE, and the corresponding source code is automatically generated. In the context of the Vehicle Management IoT system, a comprehensive set of three code files is generated: (1) an application written in the C++ language tailored for execution on the device utilizing the Wayland window system; (2) the codebase for a system administrator desktop application written in Java; and (3) the codebase for an administrative application designed for Android smartphones, coded in Java as well. Figure 5 illustrates the view of the generated code files (red frames) within the project interface of the DSML4PT IDE.

The creation of the public transportation application starts with the Architecture element. This element should be present in the public transportation system and its properties should be specified. Hence, the generation of artifacts from the designed DSML4PT models starts from instances of the Architecture element. Figure 6 shows an excerpt from the Acceleo rules written for M2T transformations to generate code in our study. If the condition in Line 01 is met during parsing of a DSML4PT model instance, the part between the [if] lines will be generated as the corresponding code. Figure 6 is the Acceleo code example to produce a GPS module function from the corresponding model. Figure 6 shows that if the condition where the Controller Port type is not empty is satisfied, the corresponding “if” block is triggered. Subsequently, upon fulfillment of the condition, the values of variables such as GPS module type, GPS module model, SoC type, port, baud rate, and pin retrieved from the model are seamlessly integrated into the pertinent sections of the code. Figure 7 includes an excerpt from the code generated when the M2T rule given in Figure 6 is automatically applied to the instance model of the Vehicle Management application.

Figure 7 presents a C language code snippet exemplifying the generation of GPS initialization code from the Public Transport viewpoint of a DSML4PT instance model. The code generated here allows entering the parameters that enable the GPS module in the system to be initialized. For example, the communication port connected to the GPS module is UART3 line while the communication speed is 115,200 bauds. Likewise, code can be generated for the other viewpoints of the modeled public transportation application. Finally, in the fourth step, generated code can be completed by adding code in, e.g., C++ or Java for the full implementation.

5. Evaluation

In this study, a comparative evaluation was performed with IoT software developers who have experience in public transportation systems to determine how MDE using DSML4PT facilitates system development compared to traditional methods. Related evaluation aimed at determining how MDE using DSML4PT facilitates system development instead of the traditional development of IoT-based public transportation systems.

The following sub-sections describe the evaluation methodology, provide a discussion on the quantitative and qualitative assessments of DSML4PT, and present the evaluation results. Additionally, threats to the validity of this evaluation study are outlined.

5.1. Multi-Case Evaluation Methodology

We followed the Multi-case Evaluation Methodology which was introduced in [88] and previously applied in the assessment of various DSLs/DSMLs and MDE processes such as [45,89,90,91,92,93,94,95,96] for different research domains. Before we discuss how this methodology is used for the evaluation assessment of DSML4PT, evaluation dimensions and multi-case study protocol of the adopted methodology are discussed in this section briefly due to space limitations. A complete definition of the methodology can be found in [88].

Multi-case Evaluation Methodology proposes the usability evaluation of DSLs/DSMLs according to two main dimensions, namely Execution Dimension and Quality Dimension. Please note that there exists one remaining dimension, called Language Dimension, but it was not considered in our evaluation since it is not related to DSML usability and it covers the number analysis of both modeling elements and model transformation rules, which are out of our evaluation’s scope.

Execution Dimension aims to measure the output performance and time savings achieved by using the DSML during system development. Output Performance considers measuring the ratio of the generated artifacts (e.g., computer programs, configuration files, etc.) over these artifacts plus delta (manually produced) artifacts, while measurement of Development Time leads to determine how using DSML shortens the overall system development as its name suggests.

Quality Dimension covers the user perspective where a DSML is qualitatively assessed based on the feedback of the users who experienced using the DSML while developing various systems. User feedback is gathered by a questionnaire comprising two parts: (1) scoring the DSML features based on predefined quality characteristics and (2) answering a set of open-ended questions. For the scoring part of the questionnaire, use of the quality characteristics defined in the Framework for Qualitative Assessment of Domain-specific Languages (FQAD) is recommended in [88], and was also preferred in our study.

FQAD [97] defines the following ten quality characteristics, each composed of various sub-characteristics and their descriptions, to qualitatively assess DSLs: Functional Suitability, Usability, Reliability, Maintainability, Productivity, Extensibility, Compatibility, Expressiveness, Reusability, and Integrability.

Functional Suitability represents a DSL’s ability to meet the requirements of the application domain while Usability aims at evaluating whether a DSL can be used to achieve the goals specified for the construction of software systems. Reliability is defined in [97] as the property of a DSL that aids in producing reliable programs. Maintainability of the languages should also be provided, e.g., by modifying the language (if needed) without degrading its functionality. Productivity needs to be considered for the language evaluation to determine whether both the development time and the amount of human resources decrease by using the language. Extensibility considers whether the DSL leads users to add new features, while Compatibility measures the degree to which a DSL is compatible with the domain and development process. As its name suggests, the Expressiveness quality characteristic is defined in [97] as the degree to which a problem-solving strategy can be mapped into a program naturally. Another desirable feature is that the syntax and semantics definitions of DSLs can be used to create other languages or development platforms. That feature is evaluated in FQAD’s Reusability quality characteristic. Finally, Integrability of a DSL measures how the language can be integrated with other languages, e.g., used in the development process. In the following sub-section, we discuss how FQAD characteristics were adopted and interpreted during the qualitative assessment of DSML4PT and the formalization of the questionnaire (given in Appendix A).

The multi-case evaluation methodology includes a study protocol that can be followed to evaluate the usability of a DSML, taking into account the above execution and quality dimensions. The study protocol consists of three major phases: Preparation, Execution, and Analysis [88]. In the Preparation phase, research questions are defined, case studies are specified, and the evaluator teams participating in the evaluation studies are determined. The Execution phase covers the design and implementation of the software systems required for the use cases with and without using the DSML in question. Evaluator teams perform analysis, design/modeling, implementation, and testing tasks for each case study during the Execution phase of the methodology. Following the system development, all participants answer questions to measure the features of the DSML according to the FQAD characteristics, in addition to a set of open-ended questions. Finally, in the Analysis phase, all data collected from the execution phase are used to prepare quantitative and qualitative assessment reports, tables and charts.

5.2. Multi-Case Evaluation of DSML4PT

5.2.1. Preparation

As discussed in Section 5.1, research questions, use cases and the team of the evaluators are prepared in this phase.

The conducted evaluation aims to answer the following research questions (RQs):

RQ1: To what extent does the use of DSML4PT allow the automatic generation of the software components of an IoT-based public transportation system?

RQ2: Does using DSML4PT reduce system development time?

RQ3: What are the pros and cons of using DSML4PT from the perspective of the system developers?

To find the answers to the above RQs, our evaluation has two parts: (1) quantitative analysis including production performance and development time assessments, and (2) qualitative assessment from the user perspective.

In this study, ten software developers, who are currently working or previously worked in the domain of public transportation, voluntarily participated as the evaluators. All evaluators had at least a B.Sc. degree in computer engineering or electrical engineering. Five of these ten evaluators had a M.Sc. degree in computer engineering, and two of these five evaluators were also pursuing Ph.D. degrees in information technologies at the time that this study was conducted. All of the evaluators were experts in developing public transportation IoT systems based on embedded software and IoT components, with varying experience from 3 to 16 years in the IT companies producing intelligent transportation system solutions for automated fare collection, vehicle tracking, real-time passenger information, etc. Specifically, the evaluators had an average of 4.5 years of experience in the design and implementation of IoT-based public transportation systems. Although they were mostly familiar with the UML and software modeling, five of them had previously used DSMLs and/or MDE techniques before this study.

Development of the following eight different public transportation applications with varying complexities were taken into consideration as the use cases:

• Validator: Validators are one of the main IoT devices used in the public transportation systems. A validator is essentially a system for collecting fees from customers. Payment transactions can be made using media such as RF card, smart phone, or QR code. The system generally consists of a validator device software and a server software in the cloud used for one fee collection. Mobile phone applications can also be integrated into the system.
• Advertising Management System: This is an IoT system where advertisement broadcasts are made in public transportation vehicles. It consists of an ad management software and cloud software. The ad management software is used by an admin to manage all system activities, while the cloud software holds interactions and ads in the system.
• Air Pollution: This IoT system is used to monitor air pollution outside of public transportation vehicles. It consists of software that reads data from sensors and server software on the cloud, which handles all the information and statistics.
• Passenger Count: This IoT system tracks the number of people in public transportation vehicles. It consists of sensors for passenger counting and software that reads data from the Global Positioning System (GPS) module, and a server software on the cloud. The server software manages location and passenger count information depending on the stops.
• Passenger Information: These IoT systems provide updates on new stops and announcements for public transportation passengers. They consist of software that uses LED panels and LCD monitors, as well as cloud software that holds the required information and manages the interactions.
• Temperature Sense: This application is used to send internal or external temperatures measured in public transportation vehicles to a cloud system. The cloud system stores the information and prepares statistical data representations.
• Vehicle Management: This IoT system provides management for public transportation vehicles. It consists of onboard software and server software on the cloud, receiving data from sensors via CAN line and performing necessary system management. Information such as vehicle speed, engine speed, and fuel level is processed and managed in this system.
• Vehicle Track: This application consists of software that reads data from the GPS module and cloud software for the global location information of the vehicles used in the systems. It is especially useful for managers to track the location of vehicles spread throughout the city.

5.2.2. Execution

At the beginning of the evaluation study, all evaluators received a brief training course to familiarize themselves with the MDE methodology and using DSML4PT and its IDE. A presentation on the case studies and their requirements was also made for all evaluators.

Evaluators were divided into two groups. In the first group, half of the participants developed the required public transportation applications for all case studies using traditional methodologies and familiar software development tools. During this process, coding was performed manually. Then, the same group designed and implemented the same applications using DSML4PT and the accompanying MDE methodology. In the second group, the order was reversed. The evaluators first developed all required public transportation applications using DSML4DT and then utilized the traditional methods and tools to which they were accustomed. Elapsed times were recorded for each developer and each step. Moreover, the created or auto-generated applications for all evaluation sessions were collected. Finally, all participants answered the questionnaire to provide their feedback on using DSML4PT.

5.2.3. Analysis

All data, programs, and documents achieved from the previous execution phase were used for assessment. For this purpose, quantitative analysis of generation performance and development time performance, along with questionnaire-based qualitative assessment, were provided, considering the evaluation dimensions introduced at the beginning of Section 5.1. Discussion of these results is given in the following sub-sections.

Quantitative Analysis

To find answers for RQ1 and RQ2, this analysis has two parts: measuring the generation of the public transportation applications (called generation performance), and the time saved during the entire development process (development time performance), respectively.

Generation Performance: Since we aim to determine whether DSML4PT is feasible to create public transportation applications considering RQ1, comparisons were made between the automatically generated code and the delta code added to complete the applications for all case studies. Performance evaluation was performed by comparing the percentage of automatically generated and manually developed software code.

Figure 8 shows the bar chart of the Lines of Code (LoC) from the case studies perspective. The LoCs listed for each case study are the average of all developers in the evaluator group. For example, all evaluators developed 79.11% of the final code for the Validator use case by using DSML4PT. After this, evaluators added manual code, 20.89% of the final code. All measurement results for each developer and for each case study can be found in [98].

As can be seen from Figure 8, the overall average LoC is 79.56%, i.e., 4 out of every 5 LoCs are automatically generated with DSML4PT. It allows us to say that this rate of code generation performance facilitates software development, and a significant part of the code is produced just by using DSML4PT.

The rate of generated code for the case studies ranges from approximately 72% to 84%. This variation primarily arises from the differences in both the complexity and the representability of the public transportation systems inside the DSML4PT IDE for each case study. System components, including IoT elements and their relations, required to construct the system differ for each case study. For instance, most of the software required for the passenger count system was automatically generated from all developers’ models since design with DSML4PT succeeded in creating rich models for IoT, public transport, and cloud service elements and their relations. On the other hand, generated LoCs for the advertising management (case study #2) and passenger information (case study #5) applications were relatively lower since they were dependent on using external libraries and a set of hardware configuration files that cannot be modeled with DSML4PT.

Moreover, variety in both the domain knowledge and experience of the developers could have affected the quality of the models they created as well as the output of the code generation process. However, to keep this effect to a minimum, we conducted this multi-case evaluation with varying public transportation development complexities instead of only one use case.

Development Time Performance: Time measurements and analyses were made to see the development time effects of using DSML4PT. To measure the times recorded during development efforts, MDE of the applications using the DSML4PT and its IDE (Experiment A) and development of the same applications without using MDE and DSML4PT (Experiment B) were considered. The experimental results were compared and analyzed. As previously indicated, during Experiment B, the developers applied the current software development methodologies with which they were familiar. Development times were recorded for all developers, as were all steps of software development processes (discussed in Section 5.2.2) required for each case study of IoT-based public transportation systems development. The mean time for these steps was calculated separately for both experiments. The results from Experiments A and B are presented in Figure 9 for the comparison. Moreover, average times of system development for each specific use case elapsed for these experiments are also listed in

Table 3. Times elapsed during system development for each specific use case.

Use Case	Experiment A (with MDE and DSML4PT)	Experiment B (w/o MDE and DSML4PT)
Validator	68.1 min	152.3 min
Advertising Management	68.7 min	104.2 min
Air Pollution	29.2 min	73.7 min
Passenger Count	60 min	131.8 min
Passenger Information	58 min	97.1 min
Temperature Sense	52.5 min	52.5 min
Vehicle Management	50.1 min	85.7 min
Vehicle Track	52.6 min	89.5 min
Average	54.9 min	98.35 min

. Times measured for each developer’s software development processes can be found in [98]. Based on the average results given in Figure 9, the following inferences can be made for all development steps.

Analysis: The analysis step is not tied to any tool or platform. For this reason, Experiment A and Experiment B have similar analysis times. In fact, this step only depends on the structural complexity of the IoT-based public transportation systems. This step is independent of the used language, IDE, or methodology. Therefore, the difference between the times is negligible—times are already measured quite close to each other.

Design/Modelling: Almost 25% more times elapsed in Experiment B at this step. Completing system design more quickly using DSML4PT may originate from the language’s domain entities specific to the IoT and public transportation systems, including more detailed elements and attributes that can be used during production. Additionally, DSML4PT has static semantic checks that allow designers to avoid semantic errors that take more time to fix later. Despite such additional features that are thought to require more time, modeling with DSML4PT reduced design time by around 25% on average. Although this kind of detailed modeling was not performed by the developers during Experiment B, it can be supposed that the time was shortened due to the high ability to model the IoT-based public transportation with the IDE of DSML4PT.

Implementation/Generation: This is the step in which the developers achieved the most time saving by using DSML4PT. In Experiment A, most of the application code for the desired public transportation system was automatically obtained with the built-in M2T transformations provided by DSML4PT. On the other hand, in Experiment B, the developers wrote all the code necessary for implementing the system. It is worth indicating that although use of DSML4PT led them to achieve a significant portion of the public transportation applications from instance models, the developers still needed to complete the generated application by manually adding delta code to have the full implementation. Hence, times elapsed for completing the auto-generated code during Experiment A were also measured and compared. Consequently, the implementation time required in Experiment B is about three times more than the generation and implementation time elapsed in Experiment A. Here we see that using DSML4PT is most beneficial in this step.

Test: Testing time also significantly decreased when the developers followed MDE using DSML4PT. All the developers tried to find the syntactic and semantic errors of the programs in the testing/debugging step. In Experiment B, the developers searched for and fixed bugs in the entire code. However, in Experiment A, the developers mostly dealt with the problems in the delta code manually added, which was much less than the generated code. The generated code consists of many of the error-free and almost ready-made architectural code since it is auto-generated from the system models already validated by the DSML4PT constraint checks. Therefore, the time spent for testing in Experiment A was reduced to one-third compared to Experiment B.

Total Time: The total average time elapsed during Experiment A was approximately 55 min when all case studies were considered. On the other hand, the total average time elapsed during Experiment B was approximately 99 min. Thus, application of MDE and using DSML4PT led to decreasing the total system development time by almost half on average, considering ten participants’ performed system design and implementation processes for eight different system development cases studies for IoT-based public transportation.

Qualitative Assessment

To answer RQ3, a questionnaire having two parts was used (see Appendix A). In the first part, the same participants answered 26 questions to evaluate their experience using DSML4PT from different perspectives. To prepare this scoring part of the questionnaire, the FQAD framework [97] discussed in Section 5.1 was adopted and customized according to both DSML4PT specifications and the public transportation domain. These questions were scored by the evaluators on a Likert scale ranging from 1 to 5, where 1 means “Very Bad”/“Absolutely not agree” and 5 means “Very Good”/“Absolutely agree”. All questions in this section are categorized into ten different quality characteristics of the FQAD framework, which are also briefly introduced in Section 5.1. It is worth noting that these questions are prepared to retrieve the evaluators’ feedback on the usability of DSML4PT considering the sub-characteristics also defined in FQAD [97] (in Appendix A, see these sub-characteristic names given in parenthesis at the end of each question text (if applicable)). In the second part of the questionnaire, six open-ended questions were asked of the evaluators to obtain their feedback on using DSML4PT. The answers received from all evaluators for this questionnaire are found in [98].

Evaluation Based on the Scored Questions: The average scores obtained from the evaluators for the first part of the questionnaire are illustrated in Figure 10. As can be seen, DSML4PT received scores ranging from 4.3 to 4.9 across all evaluation categories. The overall average score for all responses is 4.44, quite high. Thus, this result can be considered a success indicator of the designed language.

While the average scores for all categories were high, the highlights of DSML4PT for this assessment are integrability, maintainability, and expressiveness. In this context, evaluators largely agree that DSML4PT offers a comprehensive model for the IoT and public transportation system structures and that it is mostly suitable for software development for public transportation systems. The scores also indicated that evaluators believed DSML4PT increased programming productivity. Additionally, DSML4PT was found to be compatible with the public transportation IoT field, and using DSML4PT instance models can facilitate constructing public transportation systems from various viewpoints. Finally, evaluators confirmed the strength of DSML4PT’s syntax and semantics in expressing the varying needs of the IoT-based public transportation systems.

Evaluation Based on Open-Ended Questions: In the second part of the questionnaire, the following open-ended questions were asked to obtain feedback from the participants:

• Does DSML4PT make public transportation software development easier?
• Do you find DSML4PT useful for the development of IoT-based public transportation system software?
• Do you think DSML4PT is strong enough to model overall public transportation systems?
• Do you think DSML4PT IDE is easy to use?
• Are there any difficulties you faced while using DSML4PT? If so, do you have any suggestions to solve it?
• Please write your suggestions and other comments for improving DSML4PT’s features.

Considering the first question, while evaluators generally found DSML4PT feasible, some specifically indicated that visual modeling with DSML4PT makes IoT-based public transportation systems design easier. Two evaluators noted that DSML4PT provides an exceptionally user-friendly structure in which both the public transportation metamodel and the instance models can be developed. Therefore, it is suitable for general use for developing public transportation software. In addition, three other evaluators stated that the well-defined structure of the language made software development easier and ensured the emergence of the quality of the public transportation system software.

Considering the responses to the second question, three evaluators specifically stated that DSML4PT was very successful in guiding developers through restrictions and controls in the development environment. With the public transportation system models created, software quality was increased in every aspect. In addition, five evaluators stated that the defined rules contributed greatly to reducing the error rate. One evaluator stated that although the DSML4PT environment had various features, it was difficult to get used to the modeling environment. Therefore, it was decided to provide detailed information when introducing the development environment to the users and to increase the number of modeling cases.

Looking at the third question, most of the evaluators stated that the built-in modeling brought by the IDE is very powerful in supporting the sustainability of system development, meaning that public transportation IoT models can be easily modified for changing system configurations. Generally, they said that the public transportation IoT domain is modeled comprehensively, including support for cloud and services. However, one evaluator specifically noted that the service/cloud viewpoint of the language was somewhat simple and could be improved, as it could not fully express the details of the service/cloud properties individually. This was because the cloud applications were represented with only one node. However, the same evaluator stated that, according to the results obtained, the applications worked properly and that a change in the metamodel for the service/cloud viewpoint would be sufficient for visual purposes only. This feedback was taken seriously and steps were considered to enrich the service/cloud viewpoint of the DSML4PT’s metamodel.

Considering all the answers to the fourth question, the evaluators generally agreed that the IDE provides a simple and comprehensive design where users do not have to put much effort into preparing IoT-based public transportation system applications. Many evaluators mentioned that the viewpoint approach facilitated the design and was beneficial for the validation process and for ensuring seamless design, including the code generation phase. However, two evaluators reported that some validation rules did not give proper results. They found that even though they entered the correct information into the DSML4PT modelling environment, the validation rules displayed incorrect information on the screen. Upon inspection of the DSML4PT tools, it was understood that these rules were developed incorrectly when DSML4PT was first designed. These bugs have been fixed in the DSML4PT IDE.

The most common answers given for the fifth question considered the difficulty of moving away from traditional development methods to model-driven techniques. Some evaluators, especially those who have not worked with model-driven techniques for a long time, stated that the familiarization period of MDE should be longer. As explained in the answer to the first question, training time and sample cases were increased to overcome this difficulty. Two evaluators mentioned difficulties arising from Java and Sirius installations. To address this challenge, an installation guide was prepared. Two evaluators specifically stated that some icons representing elements were not fully related to the domain and did not reflect the correct elements, making it confusing for modeling. Consequently, the design of the concrete syntax was changed, and visualization of the icons was improved based on user feedback. One evaluator expressed that in two cases, a small part of the generated code was difficult to understand because it conflicted with the modeling relationships. This conflicting situation was examined, and an update was made to these cases specific to the Acceleo code.

Some valuable suggestions were received for the sixth question. For example, three developers suggested presenting the public transportation IoT domain relations defined in DSML4PT inside a document especially for new developers. Considering this proposal, a piece of introduction documentation was prepared. This document explains the domain of IoT-based public transportation systems and specifies the relationships with the DSML4PT metamodel in detail. One evaluator recommended using online modeling technologies to address issues during the migration of the DSML4DT’s IDE features. This suggestion was reviewed, and online modeling tools were studied. The strengths and shortcomings of the examined tools were determined. It was found that current web-oriented modelling infrastructures do not yet offer a complete solution for recreating the IDE of DSML4PT, particularly in terms of meeting the requirements of its MDE stages and language components (modelling features, concrete syntax structures, validation rules, and code generation). Nevertheless, plans are in place to migrate the language’s IDE to the web as soon as deficiencies are addressed. One evaluator mentioned that new SoCs should be added to the related variables in the IoT core viewpoint. According to this evaluation, case studies will be increased, and new SoCs and equipment available on the market will be added to the system.

5.3. Threats to the Validity

There are some threats to the validity of the conducted evaluation, which can be examined according to four well-known main types of validity threats [99]. These are (1) internal validity, (2) external validity, (3) construct validity, and (4) conclusion validity.

5.3.1. Internal Validity

Internal threats to the validity of our experiments are largely related to the selection and grouping of software developers participating as evaluators. We conducted the assessment with ten software developers who design and implement public transportation systems at an industrial scale, which could limit the generalizability of the findings due to the small sample size and lack of diversity. However, the participating developers’ significant experience with leading companies in the public transportation domain enhances the reliability of their feedback and the evaluation of DSML4PT. Their expertise is considered a valuable asset to this study. Using a single evaluator group poses a risk, as evaluators may benefit from their previous experience with DSML4PT when developing the same business application without it, and vice versa. In our previous work on the usability of DSLs/DSMLs for other application areas [45,92,94,95,100,101,102,103], both single and dual evaluator groups have been used. Despite the risk, a single group was chosen to ensure a more efficient qualitative evaluation based on user feedback, as ensuring homogeneity in two separate groups in terms of domain knowledge, experience, and skills is challenging. This threat was mitigated by selecting qualified professionals for the development and evaluation tasks and by randomizing the order of the evaluator groups and the applied case studies: half of the evaluators followed MDE methodology with DSML4PT at first and then switched back to their classical methodologies, while the other half did the reverse. Additionally, developers might become more proficient with the tools as they use them, potentially skewing the results. To balance any learning effects across both development approaches, the order in which developers used DSML4PT and traditional methods was randomized. By carefully selecting experienced professionals, randomizing the order of methodologies, and considering the unique advantages and challenges of using a single group, we aimed to minimize internal validity threats and ensure robust and reliable evaluation results.

5.3.2. External Validity

Generalization of the achieved results should be considered in light of the threats to the external validity, i.e., the degree to which the examined studies are representative of the reviewed topic. We believe that this threat was mitigated in this study by selecting multi-case system design and implementation studies. These studies covered the development of various public transportation applications that can be both directly deployed and executed on real public transportation IoT devices. These applications are nearly ready for industry use, rather than being trivial examples, thus enhancing the representativeness and applicability of the findings.

5.3.3. Construct Validity

Construct validity refers to the extent to which the operational measures and the cases studied are truly representative of what researchers had in mind. We aimed to evaluate DSML4PT by considering its ability to automatically build public transportation system applications and how well the language and its features were adopted by the developers. To accomplish this, experiments were conducted in which the related tasks were performed with and without DSML4PT. LoCs were evaluated according to the experimental outputs from eight different case studies of IoT-based public transportation system development by ten different evaluators with field experience. Additionally, feedback was obtained from the same developers via a questionnaire to identify the advantages and disadvantages of using DSML4PT. Therefore, the application of this multiple-case assessment methodology enabled us to achieve the aim set at the beginning.

5.3.4. Conclusion Validity

Finally, for conclusion validity, we need to consider the reliability of the results obtained. In this context, we believe that selecting multiple case studies of varying complexities helps minimize the risk to the validity of the result as well. For example, with DSML4PT, different cases involving wide ranges of IoT and public transportation devices, microprocessors, and microcontrollers were preferred. Additionally, there is a broad spectrum of software that represents the field well in each case study. Another potential threat to validity is the number of evaluators, as only ten developers participated in this assessment. However, rather than choosing inexperienced users or students, we paid special attention to making this evaluation only with developers who have experience in the relevant field and are actively working in the public transportation domain or have worked there for many years. The length and inclusivity of the multiple case studies also affected the number of these volunteers, as they were asked to develop full-fledged IoT-based public transportation applications from scratch for eight different cases, applying the entire MDE process based on DSML4PT, and repeating the same system development also without using DSML4PT. It was not easy to ensure the voluntary participation of such a large number of people in these experiments, which lasted for hours on different days, especially considering their current workload.

6. Conclusions

In this study, we examined the usability of a domain-specific modeling language (DSML) called DSML4PT, designed for model-driven engineering (MDE) of IoT-based public transportation systems. After introducing the syntax and semantics of the DSML4PT modelling language, we demonstrated its application within an MDE methodology. The language and its integrated development environment (IDE) were assessed both quantitatively and qualitatively through a systematic evaluation. This evaluation involved the use of the language in the design and implementation of eight different real public transportation applications. The evaluation was carried out with the voluntary participation of ten software developers with varying knowledge and experience in the development of public transportation systems. A comparative analysis based on the multi-case studies revealed that around 80% of IoT-based public transportation systems could be automatically generated using DSML4PT exclusively through modeling. Compared to the traditional software development methodologies, this novel approach with DSML4PT also reduced the time required for the development of public transportation applications by nearly 50%. Moreover, the overall evaluation rating of the language, as assessed through a questionnaire-based assessment, was measured at 4.44 on a 5-point Likert scale. Developers who participated in the evaluation studies also provided valuable feedback on DSML4PT. In particular, evaluators confirmed that the language’s strengths lie in integrability, maintainability, and expressiveness. Many highlighted its practicality and its applicability across various perspectives. The DSML4PT language, its IDE, and data and documents from the evaluation results discussed in this paper are all publicly available in [98]. Based on the current structure of DSML4PT and its IDE, the technology readiness level (TRL) of these products is estimated to be between TRL 4 and TRL 5. This assessment is due to the validation of the usability through a systematic evaluation, where functionality was demonstrated using various applications from the public transportation industry.

In our future work, we plan to improve the DSML4PT language constructs and features by taking into account the suggestions of the evaluators. For instance, we will investigate alternatives for developing a web interface to utilize the modelling and code generation features of the language, as suggested by some of the participants. Another area of focus will be refining the graphical syntax currently provided for modelling both IoT and public transportation components within DSML4PT IDE. An additional evaluation can be performed by the participation of DSML4PT users to assess the adoption of the current visual representation of the modelling elements. If necessary, we can improve both the visual expressiveness and the correspondence between graphical symbols and the semantic constructs by revisiting the well-known Physics of Notations [104] principles.