Digital Process Twins as Intelligent Design Technology for Engineering Metaverse/XR Applications

Abstract

The last years have seen propagating Metaverse and Mixed Reality (Extended Reality, XR) technologies into everyday applications. Due to their immersion and digitalization capability, sustainability issues can be addressed to optimize resource consumption before processes are put to practice or products are materialized. In particular, Digital Process Twin technologies can execute behavior models of socio-technical Cyber-Physical Systems. They allow both designing variants of system behavior and validating implementation architectures for system operation. In this contribution, I leverage these capabilities to capture the behavior-centered intelligence of Metaverse and XR applications. The more accurately user roles and cyber-physical components can be captured by behavior models representing Metaverse/XR applications, the more accurately the environmental, social, and economic impact of design alternatives can be experienced.

1. Introduction

Metaverse applications have been introduced as shared virtual spaces where people can interact in real-time using avatars [1]. Thereby, Extended Reality (XR) technology, intertwining Virtual and Augmented Reality, is considered the key component of metaverse applications [2]. Metaverse applications are not device-independent, but enable independent business processes and thus, a virtual economy including digital currencies and non-fungible tokens (NFTs) [3]. NFTs are ‘transferrable rights to digital assets, such as art, in-game items, collectables, or music. The phenomenon and its markets have grown significantly since early 2021′ [4] (p. 216).

As such, Metaverse applications enable exclusive virtual transactions by virtual users. Physical users need to interact with metaverse technologies and applications through head-mounted displays (HMDs) to control exclusive digital processes while acting in the physical world [5]. This duality of being physically and virtually active represents a complex design challenge, both from a functional and interaction perspective [6].

With the development of the Internet of Things (IoT), Digital Twin technology has been widely used in enterprises undergoing a digital transformation as an effective method to address the challenges of cyber-physical convergence, starting in the manufacturing industry [7]. Digital twins (DTs) are digital model representations of physical systems and enable the bi-directional communication between the physical system and its digital representation. Aside from physical components, they can also capture virtual entities or systems, e.g., NFT transactions, and organizational processes (https://www.gartner.com/en/information-technology/glossary/digital-twin, (accessed on 8 November 2023) [8]). Application domains of DTs include product engineering and manufacturing, healthcare, mobility, smart cities, and supply-chain management [9].

Although DTs have become increasingly common in designing, engineering, and operating Cyber-Physical Systems (CPS), the existing variety of approaches so far have inhibited a methodologically grounded and user-centered modeling approach [8,10,11]. However, as Metaverse/XR applications propagate to DT domains, digital design skills become increasingly important for the workforce when organizing collaboration [12], and a human-centered engineering approach to DTs of Metaverse/XR applications becomes mandatory [13,14,15,16].

This demand fits well to the latest sustainable systems engineering approaches: ‘Sustainable development is development that meets the needs of the present without compromising the ability of future generations to meet their own needs’ [17] (p. 2), as it requires stakeholders to be included in system developments. Stakeholder participation ensures design control and how designs are put to practice (i.e., engineering) [18]. In the following section, I detail the research objectives stemming from this problem statement and introduce the structure of the paper for embodying design knowledge into the engineering process when using Digital Twins of Metaverse/XR applications.

2. Research Objectives

The objectives address the need for overcoming an epistemological and abstraction gap in design (Section 2.1), recognizing the evolutionary nature of Metaverse/XR application development (Section 2.2), and enabling continuous stakeholder participation in design and engineering processes (Section 2.3).

2.1. Addressing the Epistemological and Abstraction Gap in Design

The application of DT technologies from a socio-technical perspective has revealed an epistemological and abstraction gap [19]. This gap is likely to widen, when we neglect the dynamics of socio-technical systems as living systems. According to system theory, all living systems are continuously self-reproducing systems in an auto-poietic way [20], which already has an effect when different stakeholders cooperate in the initial development phase, and subsequently, when applications are adapted according to changing needs. Complex adaptive systems carry the ‘burden’ of unpredictability and interaction-based design processes [21], which are at the same time the drivers of self-reproduction.

On one hand, it is the complexity of system architecting and stakeholder interactions—design ‘is practiced inside virtual reality by and on 3D avatars’ [22] (p. 607)—caused by various backgrounds and information exchange requirements in the course of Metaverse/XR application development, e.g., when different stakeholders and experts from various fields like IoT, Cloud Computing, and business case managers, contribute to design intelligence [23]. On the other hand, it is the evolutionary nature of the design process itself that requires an open and dynamic representation space to provide orientation and common ground for human-centered development and design [24]. Such a space is in need of abstractions accessible for the involved stakeholders as they are system-relevant actors [25].

In each step of evolution, a socio-technical system reaches a certain level of complexity. It forms the ground for the next step in system development, likely to increase the complexity. This process has been termed by Luhmann as ‘morphogenesis of complexity’ [26] (p. 415). When looking at the capabilities of Metaverse/XR applications compared to web and AR/VR technologies, the increase in complexity mainly concerns the immersion of users and the resulting interactions. Hence, a representation scheme of a (collaborative) design space has to provide a level of abstraction that keeps humans in the loop of development and evolvement.

2.2. Striving for Sustainability in Terms of ‘Dynamic Stability’

An ecosystem based on an evolving design space ultimately reaches some ‘state of dynamic stability’ [26] (p. 428), meaning that the Metaverse/XR application and their stakeholders form an ecosystem that allows for continuous development according to current technological capabilities and user requirements. This process is determined by variation and selection activities, with the latter aiming for re-installing stability. Once this state has been reached, another system variation exists that creates a subsequent selection process. Since any stage of stability triggers dynamic changes, design representations need to capture those to be of suitable support.

This research aims to overcome the addressed epistemological and abstraction gap by a methodologically grounded design through Digital Process Twins (DPTs). They enrich the data-centric perspective on design representations by taking into account dynamically evolving stakeholder roles, system components and their collaboration, as well as changing technological capabilities on a human-centric level of abstraction [27]. Design intelligence is captured by behavior models representing DPTs and their validation [28,29]. Validation allows stakeholders to execute these models before implementing a respective Metaverse/XR application.

Executable DPTs at design time encode roles and their interactions, which meets the demand for checking the effect of digital transformation on sustainability ([30], https://international.eco.de/jts-solution/jts-facts/ (accessed on 8 November 2023). Sustainability itself has the goal of intelligent design and construction to optimize the use of resources and minimize waste [31]. Through the design process, any organization or product development has the opportunity to investigate design alternatives and to minimize environmental, social, and economic impacts of their product [32]. Designers share the responsibility in this crucial phase of development [33]—they determine the scope of a system at hand [34], i.e., what needs to be designed, with the results that lay ground for optimization [35].

System design requires models to specify a system’s structure and behavior prior to optimization [36]. The closer design models come to reality the better [37,38], with the ontological adequacy as a measure of how close the models produced using a modeling language are to the situations in the reality they are supposed to represent [39]. The selected and presented approach strives for intelligent system design in a stakeholder-relevant ontological adequacy.

2.3. Continuous Stakeholder Participation in Design and Engineering

Aside from bridging the recognized epistemological gap in human-centered design through an adequate modeling technique for the DPT, we need to consider the openness of design representations, and thus, behavior adaptation, important [40]. Complex situations are likely to require variants of system intelligence that need to be tested prior to implementation [41], also for probing the inclusion of machine intelligence [42].

With a proper approach to abstraction, system behavior in complex situations can be simulated and evaluated before actually being implemented and potentially wasting resources or creating harm [27,29,43]. Role-specific modeling as the core of sustainability-relevant activities and interventions provides a stakeholder-centered granularity for adaptive system design through DPTs. It enables to experience behavior, to evaluate it, and control change processes in continuously transforming digitalized ecosystems. The simulation capability allows for optimizing the use of resources and minimizing waste before they are put to physical practice.

Due to the nature of Metaverse/XR applications, stakeholders may participate in design as avatars and/or physically. Hence, the respective technologies provide 3D worlds that have the capability to increase participation through immersive activities of stakeholders. Sustainability-relevant processes and behavior (changes) can be simulated in virtual settings before being put to physical practice.

2.4. Structure of Presented Work

Section 3 introduces the Design Science-based research methodology to capture the evolutionary character of this research, and provides insight into existing findings when addressing human-centered DT modeling and Metaverse/XR application development. Section 4 presents the results based on outcomes reported from existing work and consolidated according to the objective of this paper. They concern the procedure of how to approach design from a task- and role-centered perspective, and how particularities of Metaverse/XR applications can be embodied in process-based DT design specifications. Section 5 of this paper discusses the results in light of bridging the gap between technology specifications and human-centered design of Metaverse/XR applications, and concludes the work with final methodological remarks.

3. Methodology and Materials

This section details the procedure followed for this research, and gives an overview on structuring the design of Metaverse/XR development as well as DT capabilities, and on modeling requirements for human-centered development.

3.1. Design Science-Based Research Procedure

Design Science methodologies [44,45] have been developed with a focus on the evolutionary design of information systems. The Design Science framework [46] is solution-oriented without anticipating a complete set of requirements. Furthermore, it supports searching for empirical solutions to practical problems due to its iterative nature, and thus, facilitates stepwise refinement into validated and traceable results. Finally, it features the structured and traceable integration of previously isolated concepts and practices, e.g., utilizing DTs for sustainable Metaverse/XR application development.

The drawing on other research approaches and methods to create solutions to practical and complex domain-specific problems is of benefit to meet the challenges in Metaverse/XR development. A variety of stakeholders and experts, domain knowledge (on CPS, VR, AR, systems engineering, etc.), and design topics (dual appearance of users, layering of architecture, etc.) are involved, which requires structuring design activities and engineering in an evolving development process. The Design-Science framework provides contextual elements to start iterative design activities (‘Environment’) and preserve their results (‘Knowledge Base’) (Figure 1).

The Relevance Cycle (Figure 1) applied to the objective of this work connects the environment of the Metaverse/XR development with its core activities. The Rigor Cycle relates these activities to a knowledge base informing the research. The Design Cycle iterates between the core development activities (building and evaluating artefacts). This intermediate position ensures on one hand that artefact development remains in the context in which the process started, and on the other hand, that each development cycle is informed by evolving design knowledge and domain-specific practice, and the results are documented in a structured form. Each design cycle result can be traced back to its starting point and related to previous design cycles. In this way, each step of development becomes transparent, and thus, traceable in terms of specifying requirements and evaluating system artefacts.

The Design Science framework has been operationalized capturing the following development stages [44]: (i) identification of the problem, (ii) definition of objectives for a solution, (iii) design and development of artefact, (iv) demonstration of artefact use to solve the problem, (v) evaluation of the solution, (vi) communication of achievements. This research focuses on the initial run-through. The research problem is specified and the value of a solution is justified (Section 1, Section 2 and Section 3 of this paper), as DPTs enable to model Metaverse/XR applications in a human-centered way for reflective and continuous development and adaptation. The artefact is a digital representation of the design process and a modeling space that results in a system architecture and executable DTP models (as a result of system engineering). Its use can be demonstrated and evaluated (see Section 4 of this paper). The paper itself corresponds to the communication of achievements, including the discussion and conclusion reflecting on the achievements and further design cycles following Design Science-based research. It thus becomes part of the Knowledge Base (see Figure 1) on Metaverse/XR development.

3.2. Structuring Metaverse/XR Development Utilizing Digital Twins

Since DTs allow structuring Metaverse/XR applications, we need to know how existing development approaches could fit to DT representations, and how (universally) they could be utilized for design and/or operation. In the following, I analyze work related to that respect. Moreover, the intelligibility for human users (due to domain adaptation and experts involved in development) is a challenge that needs to be met and thus, is also analyzed from the findings so far.

In this paper, Metaverse/XR applications are understood as technology that intertwines virtual and physical world elements with users as immersive parts. In these mixed reality settings, users can interact with physical and digital (3D) objects and manipulate them in real time [2]. As such, the Metaverse and DTs have attracted intense research interest from several perspectives:

• on a conceptual level, leading to architectures, layers, and frameworks, e.g., [47,48,49];
• to develop concrete use cases or when propagating to specific application domains, e.g., [50,51,52,53,54];
• from an implementation perspective, e.g., [55,56], dealing with infrastructures and communication protocols.

The following overview comprises studies focusing on conceptual considerations when relating Metaverse applications to DTs. Due to their abstraction from implementation and/or domain specifics, they help to achieve the objective of this study, namely to develop methodologically grounded, while still universal, design support. Most of the existing concepts also include stakeholder roles that indicate design tasks or development expertise.

DTs have been assigned to various levels of detail. Metaverse objects represented by DTs can be considered at a meso- or macro-/micro-scale [57]. In addition, from the perspective of behavior and content, DT-scaling information has been categorized according to social relations and group behavior, e.g., herding, and according to solid, liquid, gaseous, plasma, and other uncertain states. Encoding patterns of social behavior relates to sustainability-relevant developments, since DTs have been used as design representations for transhumanist settings on an implementation-independent layer. When executing DT models, system behavior can be experienced, and the impact of shifting control to digital components can be evaluated [27].

Ref. [58] defined a three-layer architecture, with a user interface layer linking the physical world to the digital one. The following uppercase terms are those used in the study. The top layer is termed Physical/Real World Layer: it comprises objects from the Real World, Users, Industrial Things, and the Physical Market. From this layer, a Request for Generating DT is issued to the intermediate layer. This layer is termed Link (User Interface) Layer. It contains Simulation/Migration components that Submit Generated DT of NFT on the Blockchain. The Blockchain is the second component of the intermediate layer and comprises Distributed Consensuses, Smart Contracts, Full Nodes, and Miners, whereas the Simulation/Migration component contains Developers, Simulators, Rarible, and Opensea. The bottom layer is the Metaverse (Digital World) Layer and has the following components: Online Market, Digital Industrial Things, User Avatar, and Digital World. This layer issues a Request to Access DTs for Use.

Since the intermediate layer receives requests from both layers, it can be considered core to DT-based Metaverse applications. In addition, the architecture identifies the Developer as the role handling the design and engineering of the Metaverse application. The developed Metaverse DTs take into account several design-relevant engineering properties [58]:

• User control of Metaverse lifecycle: Each Metaverse-based DT and its linked real-world product lifecycle need to be ‘easily’ controllable.
• Transparency and traceability: Any operation, such as buying, selling, or ownership transfer based on DT transactions, can be traced in Metaverse applications.
• Reliability: Metaverse DTs are reliable due to blockchain-based transactions.
• Decentralized management: There is no central control once a consensus protocol is accepted. Peer-to-peer communication features machine-to-user and user-to-machine communication using DTs without intermediaries.

Since XR technologies are core to Metaverse application, a closer look to their design is required. The latest research by [6] reveals: ‘XR technology lacks a fundamental universal design framework targeted at improving accessibility and usability for all’ when looking to the domain of learning support systems, a major target of Metaverse/XR application development. In line with another recent study [59], based on SLR, design-relevant challenges in implementing XR technologies have been identified. They are related to content and adaptation limitations. According to these findings, design should include multimodality in interaction, setting up and adapting XR devices, and handling specifications for interoperable application components. Further research quests included the development of guidelines including examples for increased usability of XR technology, and standardization and ‘solidification of said guidelines into standards’ [6].

A recent study on Metaverse developments based on application-oriented findings [60] concluded with specific design requirements of Metaverse/XR applications. These requirements deal with user-centered and technology-related design issues that are particular for Metaverse/XR applications when utilizing DT technology. The user-centered design issues concern the immersion of users when interacting, the intelligibility of design representations for stakeholders, and adaptation of design representations [61]:

• The concept of immersive user interaction requires users, their avatars, and their interaction capabilities being represented as part of DT models.
• Aside from overcoming interoperability problems, a Metaverse DPT needs to be dynamically adaptable, i.e., in terms of adding and modifying components and functionality.
• The capabilities and effects of Metaverse/XR have to be checked on affecting user/work-related functionality.
• When considering collaboration among users, DTs need to represent interaction mechanisms between user-relevant roles and tasks, and to support interaction between human actors and/or their Metaverse (re)presentations.
• User-relevant data, such as business objects, might need to be secured as digital assets in the Metaverse. DTs should represent such mechanisms in an explainable form. There also may be cross-cutting concerns, affecting several design entities or metaverse components.
• Users may want to integrate social media, and other existing 2D applications with (cyber-)physical interaction modalities, such as bodily immersion and body language—DTs should allow for interaction including body and senses.

Design representations and system models are created and adapted in a specific application context. This also holds for DTs as design support tools. As such, some analysis is required to identify the context of the development process [27]. The objectives and scope of a Metaverse/XR application have to be determined in that phase of development. When the (user) tasks to be supported are specified, transactions between application components, human actors, and their XR representations need to be captured in the course of socio-technical design. In particular, the functionality to switch between (cyber-)physical elements influences the design space [62,63].

4. Results

Metaverse/XR application design, if not being considered as a one-time activity, requires some specification scheme. The proposed notation is used to express and refine information considered relevant for design. It encapsulates the structure and behavior of system components by utilizing subject-oriented process modeling [28]. The resulting Digital Process Twins (DPTs) do not only provide a notation to model Metaverse/XR application designs, they also guide designers to refine their design to architecture specifications relevant for technologies to implement the developed designs. Both aspects are detailed and exemplified in the following subsections.

The first subsection introduces the design space developed to be used along design cycles. The artefacts are design representations that demonstrate a possible solution for a specific application in form of a DPT. It can be developed further either through validation or implementation. The second subsection introduces the notation to express the application designs. The provided examples address the scenario introduced in the first subsection. The final subsection details a layer concept for architecting to finally put DPT designs for Metaverse/XR applications into operation.

4.1. Structuring the Design Space

A sample scenario is described before the structure of the design space for socio-technical Metaverse/XR application design is detailed in this subsection.

Consider a potential Metaverse/XR user wearing an XR device., e.g., XR glasses, enabling him/her to interact in the physical world and in the virtual world. We also assume the user is living in a smart home environment including Internet-of-Things applications allowing for remote control of his/her home. In addition, our user is supported by a Cyber-Physical System (CPS) in home healthcare activities. They include measuring individual blood pressure and connecting to medical services for supply and emergency cases. Both smart home and home healthcare requirements are met by CPS functionalities and interaction features.

With respect to Metaverse/XR activities, the user is interested in NFTs. Non-Fungible Tokens only exist in the virtual words as cryptographic assets on a blockchain. They have unique identification codes and metadata that distinguish them from each other and cannot be replicated. These capabilities enrich the interaction space in terms of processes and data handling that occur in the cyber-world. In that case, the DPT has no counterpart in the physical world with respect to the NFT processes, but user-relevant elements and behavior, including digital components and systems in the virtual world.

Once a human user is involved, NFT processes become connected to the physical reality of users, as they may interact with other users in the physical work on NFT-relevant topics, thus affecting NFT processes. A typical interaction of that kind is newcomer behavior, seeking advice on dealing with NFTs [64]. Hence, our considered user, for the sake of practicability named Aurora in the following, will connect to others via existing channels, including social media, when asking personal contacts about NFT issues. In addition, professional sites or contacts could also be involved via Web2.0 channels, including webinars when it comes to details handling NFTs. From a design perspective, the Metaverse/XR application will have to embed existing 2D formats as part of the immersive interactions.

Figure 2 shows the structure of the design space. It follows a design principle advocated for organizational change management, as these principles are one way to bridge the addressed epistemological gap in developing complex adaptive systems: ‘Let everyone know (and design) the design’ ([65], p. 135). The Metaverse/XR application as a part of the considered socio-technical CPS is structured as a set of task-specific features assigned to interacting roles, as shown in the center of the figure. Since the focus in this paper is on the Metaverse/XR design, other CPS functionalities, e.g., contacting a medical expert in case a home-healthcare emergency occurs, are not detailed and included in the figure; for CPS design based on DPTs, see [43].

In Figure 2 and for illustrating Metaverse/XR applications, the exclusively available digital processes for handling the selection and the acquisition of NFTs are exemplified. Since the NFT functions are performed in the virtual world, they require corresponding Metaverse/XR presentations: They are (i) assigned to a virtual location or space (Rooms) for relevant subtasks, and (ii) to Avatars (re)presenting human users to enable immersive interaction. In the figure, avatars are indicated by the small symbol on the left side of the parallelogram denoting activities supported in the virtual world.

The exemplified Metaverse/XR functionality addressed in the figure comprises: NFT-Search, NFT-Selection, and NFT-Auction. The involved roles of our user Aurora are NFT-interested person assigned to NFT-Search, and NFT-customer assigned to NFT-Selection and NFT-Auction. The figure also shows as part of the Metaverse/XR design space activities put into the context of roles, that either are performed by technology components or humans. The exemplified case in Figure 2 also shows our human user in roles handling smart home and home healthcare CPS devices while being interested in NFTs. They can be detailed and further refined according to the addressed application scenarios that are acquired and analyzed prior to design.

Having set up the application-specific context, the resulting CPS activities concern 3 different domains, with one exclusively occurring in the virtual world (Metaverse), namely handling NFTs through interested person and customer behavior.

All role-specific behavior abstractions represent inputs to the DPT as design representation. Since the focus of the present work is on Metaverse/XR applications, we focus on the respective features in the following. However, the user has to be able to interact in both contexts, the CPS environment and Metaverse/XR application. Moreover, when interacting with other users and/or components, the CPS elements addressing the smart home and home healthcare can be accessed and need to be part of the design space.

On the right side of the figure, the expert for home healthcare is listed as role assigned to CPS home healthcare features, e.g., to be involved in configuration or maintenance tasks. Social media comprise user roles related to social contacts, e.g., peer groups, that are part of the scenario referring to the information exchange of Aurora. The NFT chatbot is included in the role of advising Aurora. It provides machine intelligence for handling NFT requests and deals. The elements on the right side of the figure establish the interaction space that needs to be included as they capture the socio-technical context of the Metaverse/XR application features to be developed.

4.2. Digital Process Twin Modeling

Modeling as activity central to DT-based development requires design items that abstract from the observed physical reality or (envisioned) experiences in virtual reality. When applying subject-oriented modeling, encapsulated behavior abstractions specify which roles actors perform to complete a specific work task or business process. The concerned data or business objects are exchanged between these actors to achieve desired goals. Since subject-oriented design is performed independently of implementation, the specific role carriers or technical systems are specified in the course of the organizational and ICT implementation of the DPT. As such, the development process remains traceable for humans throughout design and implementation.

In case of exclusive digital content and process support, e.g., handling NFTs, virtual role carriers (avatars) represent users in the Metaverse applications (see avatar symbols in Figure 2 and Figure 3). In this context, it needs to be noted that a single human user like Aurora can have several roles, each of them (re)presented as an avatar. In our example, the assignment of avatars has been determined by the process steps performed as an interested person (NFT-Request Specification, NFT-Search), and as a customer (NFT-Selection, NFT-Auction).

CPS specified in subject-oriented DPTs are represented as a network of autonomous behavior entities, allowing for concurrent operation synchronized by the exchange of messages [28,43]. A design entity corresponds to a role or function system unit and is termed ‘subject,’ as is represents an active element that is capable of performing local actions (like a subject in natural language sentences). These actions (like predicates in natural language sentences) concern state transitions that do not require interaction with other subjects, such as checking whether a provided value is above a certain threshold when dealing with NFTs.

In addition to those local actions, subjects can also perform communication actions, i.e., send and receive messages to and from other subjects. The actions are specified in a specific sequence to complete a certain task. For instance, the Home Healthcare subject sends a request to the blood pressure measurement device to transmit the most recent value representing the blood pressure, in order to compare it with a critical threshold. Once it receives that value, it calculates whether a medical expert needs to be contacted, sends a respective message, or completes the task. Hence, a logical bundle of local, send, and receive actions along a corresponding time line represents the procedure how to accomplish a specific task.

Subjects are specified on two layers of abstraction, requiring two types of diagrams: Subject Interaction Diagrams (SIDs) and Subject Behavior Diagrams (SBDs). The SID provides an integrated view of CPS, comprising the subjects involved (rectangles) and the messages (directed links) they exchange. At design time, system components and actor roles need to be specified to set up the design space that needs to be refined for engineering a socio-technical system.

Figure 4 shows a SID for a home healthcare scenario. At the bottom, the Blood Pressure Measurement subject is located, enabling sensing the blood pressure of a client. The Medication Handler takes care of providing the correct medication at any time and location. The Personal Scheduler coordinates all activities wherever a client is located (traditionally available on a mobile device). The Shopping Collector contains all items to be purchased to ensure continuous supply of medical items and thus, quality-of-service in home health care. The scenario addresses a home health care system that handles the measurement device according to a schedule and taking into account actions in case further actions need to be taken. The Shopping Collector receives requests from both the Medication Handler when drugs are required from the pharmacy, physician, or hospital, and the Personal Scheduler, in case further medicine for the client is required.

Figure 5 shows handling NFT requests from the perspective of interested persons and consumers. The SID does not show the detailed sequence how humans or application components interact to complete a task. It reveals the set of messages that need to be exchanged between them, although the sequence can be concluded in case only a single message is exchanged between 2 subjects. For instance, an NFT-Interested Person (subject) sends an NFT-Request Specification (message) to an NFT-chatbot (subject), in order to receive a list of NFT-candidates (message).

Subject Behavior Diagrams (SBDs) provide a local view of the process from the perspective of individual subjects. They include sequences of states representing local actions (called ‘functions’) and communication actions including sending and receiving messages as specified in the SID. State transitions are represented using arrows, with labels indicating the outcome of the preceding state. The part shown in Figure 6 specifies the communication between NFT-Interested Person and NFT-Chatbot. The subject NFT-Interested Person sends the NFT Request Specification as NFT-request to the subject NFT-Chatbot, which generates an NFT list of candidates. The generated list is sent by the NFT-Chatbot to the NFT-Interested Person.

Since each SID subject is refined by a dedicated SBD, its behavior is only synchronized by the messages exchanged with other subjects. Similar to real-time processes, the DPT is designed through choreographic synchronization of behavior abstractions. The figure reveals the parallel operating nature of the two subjects involved in the interaction. Once the need for an NFT-list of candidates—modeled as a ‘send’ activity—occurs by the subject NFT-Interested Person, a corresponding message is delivered to the subject NFT-Chatbot. When the NFT-Chatbot receives that message, the request can be processed, either recognizing the request cannot be handled (which is not included in the model in Figure 6) or searching for candidates according to the NFT Request Specification (input message to the subject). The result is delivered by ‘send reaction’ to the Medication Scheduler. The subject that has initiated the interaction can now utilize the results, i.e., the Medication Handler processes the reaction of the Personal Scheduler (modeled by the function of the respective SBD).

Internal functions of subjects operate on (the transmitted) data. In our NFT sample scenario, the subject NFT-Chatbot creates a list of NFTs according to the request from the subject NFT-Interested Person. According to the level of detail, additional internal functions can be included in the SBD. This variability is essential for embedding machine intelligence into application designs. It can be activated either by invoking an algorithm as a black box or by refining input data from the received message to select a specific algorithm for best fit. The same holds for output handling of activated machine intelligence components. It can either be delivered directly to the calling subject, or evaluated according to its parameter values. For instance, in case only part of the NFT user requirements given by the NFT Request Specification can be met by a candidate, it is not sent to the subject NFT-Interested Person having sent the request.

An effective design pattern is shown in Figure 7. It is relevant in case several behavior options exist for an actor or application component. Consider the subject NFT-Interested Person (instantiated by Aurora) when having received an NFT-Chatbot answer to the NFT Request Specification. Aurora’s options are:

• Seeking advice, e.g., in contacting social media users, by sending an advice request
• Delegate ordering by having decided which candidate to buy, by sending a delegation request, e.g., to an NFT-Broker subject
• Order when having decided which candidate to buy, by sending an order request to an NFT-Provider subject, and waiting to receive further order information
• Compare own choice to similar business cases, by sending a ‘benchmark’ request to an NFT-consumer data base, and waiting for information based on the triggered evaluation

Overall, a subject-oriented DPT consists of a network of application-relevant actors or components, with each of them refined in terms of abstract behavior specification, so-called Subject Behavior Diagrams. The latter can be validated to check the designed application behavior at runtime. The use of state transitions allows checking whether a procedure is leading to effective behavior. The behavior of each designed subject in the networked context can be experienced by each involved stakeholder before implementing a Metaverse/XR application. Academic and commercial runtime engines include UeberFlow [66], Metasonic Touch (https://www.slideshare.net/MetasonicAG/fact-sheet-metasonic-touch (accessed on 8 November 2023), compunity (www.compunity.eu (accessed on 8 November 2023)), actnconnect (www.actnconnect.com (accessed on 8 November 2023)); for a tool survey, see [67].

Figure 8 shows the MS-Visio-based SiSi tool suite v1 [68] developed for validation and execution with simulation data. It integrates modeling, validation, and simulation features for subject-oriented behavior models. The shapes for subject-oriented DPT design are shown on left side of the screen. The SBD of the application is displayed in the center of the screenshot. The simulation is run based on the parameters and values specified on the right part of the screenshot. They are fed to the model when being executed. All runtime information is also provided on the right side of the screenshot. With respect to sustainability, resource consumption information, such as an ecological footprint, can be assigned to subjects. For instance, in this way, logistic modalities, such as transportation by train or car, could be compared and the consumption evaluated.

Once DPT models have been validated, a prototype of a Metaverse/XR application can be implemented that can be controlled by a DPT. Figure 9 shows the Room for the Interested Person implemented by the spatial.io Creator toolkit v1.0 (https://www.spatial.io/ (accessed on 8 November 2023)). The room consists of a social space where people, e.g., users from social media contacts or expert consultants can be invited participate in advice sessions. The various channels for communication contacts and content are displayed by ovals and open upon selection. The NFT-Chatbot is one of them when the SBD shown in Figure 6 is activated.

Figure 10 shows a screenshot for an NFT setting enriched with yellow notes that are provided by other users to support our user Aurora when selecting NFT candidates and deciding on a certain NFT-provider. The notes include answers from service requests to the provider as well as experience reports from other customers, e.g., from social media contacts. The Room concept allows to structure the information space according to the type of content and communication channel. In addition, the presentation and interaction space of a Metaverse/XR application can be individualized in terms of experiencing immersion and interfaces to other digital components. For instance, social media threads could also be displayed and scrolled through a transparent overlay on the 3D presentations.

4.3. Capturing the Unexpected

In order to capture unexpected behavior sequences, Message Guards can be used at design time. Message guards trigger behavior sequences to handle situations that require particular attention, e.g., due to privacy or security reasons. The principle is shown in Figure 11. At design time, critical cases are described to allow their handling at run time (i.e., once design entities have been instantiated), either by humans or technological systems. Switching from routine behavior (left-hand side of Figure 11) to non-routine behavior is based on flagging states in the regular SBD sequence serving as triggers and point of (re-)entry.

In the addressed NFT example, the Message Guard can be applied when an NFT-Customer decides to change an order request after the delivery process of an NFT has already been triggered and the blockchain mechanism has already been activated. Once the flag is raised, either substitutive procedures that eventually return control to the regular SBD sequence (see left Message Guard in Figure 11 for the non-critical path), or complementary behavior that does not return control to the regular SBD sequence (see right Message Guard) is triggered. In our case, the non-critical path is given when the NFT has not yet been attached to customer data or content. All other cases require a complex event-processing scheme analyzing the roll-back opportunities to implement the delivery process of the NFT.

Message Guards, and thus flagging of actions of a process in various behavior states of subjects can only be used when specific behavior is expected, or for interventions that are activated in case of unexpected events. The receipt of certain messages, e.g., to abort the process, results in a specific sequence of state transitions. Hence, Message Guards should be utilized for functions or states in which recognizing exceptional behavior needs to be considered explicitly. The relevant design decision that has to be taken concerns the way in which the adaptation occurs—either extending an existing behavior, or replacing it from a certain state on.

Dynamic adaptation of design representations or adaptation during runtime is enabled through running subject-oriented DPTs as reference models. Dynamic adaptation of CPS is based on a trigger, such as a subject’s event or signal or the output of a function, which requires special behavior specification. It captures variants of organizational behavior known at design time. The trigger for dynamic adaptation, independently of its implementation, can carry some data as payload. For instance, a value above a certain threshold can trigger behavior changes of a component or physical device. Like an event, a data object representing a trigger can carry three types of information: header, payload, and plain content. The header consists of meta-information about the trigger, like name, arrival time, priorities, etc. The payload contains specific information about the triggering event. Finally, a trigger can also contain free format content.

Additional adaptation features depend on the runtime of subject-oriented DPT engines. Once a model instance is created and messages are sent, these messages become events. They are kept in an input pool attached to an instance of a subject. As a trigger, they get a timestamp documenting the arrival time. Instantaneous events can be handled by Message Guards, whereas others remain queued when such an event occurs. It triggers executing the respective behavior variant depending on the specified conditions (see Figure 11). For instance, the message “call customer service” from the subject NFT-Customer can arrive at any time. This message is handled by a Message Guard specified as the reaction of that instantaneous message.

4.4. Designing the Application Architecture

In order to structure further development steps, in particular the implementation from a system engineering perspective involving particular technology constellations, the architecture of a Metaverse/XR application can specified by layering subject-oriented models [69]. Together with the diagrammatic models of the previous subsection, knowledgeable designs can be established accepting the dynamic nature of design construction which leads to emergent design representations in the sense of ‘organic’ designs [70]. The design activities are considered knowledgeable actions in contrast to mechanistic development [71].

The following knowledgeable actions can be set when designing the addressed sample scenario. The core of the DPT consists of the SID and the corresponding SBDs as detailed in the previous section. Since a Metaverse/XR application needs to be controlled by an end user XR device, like a glass for dual presence in the virtual and physical world, it can be added as an abstract subject, i.e., without being initially detailed by an SBD to the design space (see top of Figure 12). During operation, it controls the CPS parts with physical devices like home healthcare as well as the immersive part comprising the NFT handling. Both can be summarized by their functional concern and positioned as part of the design space, as shown by the abstract subjects below the top layer. In this way, the design space can be restructured through architecture-relevant annotations to provide orientation for including experts for further development or simply to indicate the context of detailed designs.

Abstract subjects do not directly lead to executable behavior sequences, and thus, may not be connected to a dedicated SBD. Their refinement may lead to further abstract subjects until a subject is refined through an adjacent SBD that can be used for validation and/or DPT control by a corresponding subject-oriented modeling and execution tool.

Given the open structure of the design space, placeholders for later design arrangement can be included as abstract subjects if necessary. Lower-level subjects correspond to aggregation of components, bundles of tasks or set of stakeholders, or business sectors. In Figure 12, the elements of the addressed scenario have been arranged to demonstrate the diagrammatic layering that allows assigning technology components, such as the XR glasses without further refinement. Hence, engineers and application architects can develop a complete representation of components required for putting the designed functionalities to operation.

The layer concept allows for structured and contextual refinement of digital DPT designs and is open with respect to the number of layers. In case several experts need to be part of the design team, some SIDs may need to be refined to the stage where the actual behavior can be encapsulated. Although the structure is described in a top-down way, it can be applied through a middle-out or bottom-up approach, e.g., in case a specific CPS component, such as a robotic device for production, is the starting point of CPS development.

Figure 13 sketches the scheme for layering subject-oriented DPTs and how interrelated SIDs need to be refined until SBDs can be defined as part of the DPT design and operation. In order to run a DPT, a set of interconnected SBDs either from refined or originally designed subjects is required (see bottom layer of the architecture).

The subject-oriented DPT layering concept aims to enrich the design space and follows guiding principles for design. Overall, at least one layer needs to be specified, as it requires a SID representing value-generating subjects and their interaction. When a subject on a specific layer of abstraction is refined, the interactions (passing of messages) constrain its behavior on a lower layer. The final refinement needs to be an SBD, as it contains run-time relevant behavior for validation. It also processes the messages exchanged between SBDs within one layer or across layers, depending on the level of refinement.

5. Discussion and Conclusions

The introduced approach targeted stakeholder-relevant requirements on design-relevant support of Metaverse/XR application development [58]. It enables user control of the Metaverse lifecycle by dynamically adaptable behavior models representing Digital Process Twins (DTPs). Layering allows capturing initial designs and subsequent refinements to finally implement Metaverse/XR applications, and thus ensures transparent mapping of design ideas to operation. The dedicated set of modeling elements reduces the cognitive burden of designers and engineers [72]. The DPT models can be executed for validation to ensure the reliability of application development. Adding operational data for simulation allows exploring operational variants that are particularly relevant for cross-cutting concerns, such as sustainability.

Modeling peer-to-peer communication facilitates the design of physical/digital-world-to-user interaction and vice versa. For immersion, access and control devices can be represented as an element of the Metaverse/XR design space. It reflects the engaging character of interaction when users can interact with content from several perspectives and on various levels of detail. In contrast to two-dimensional (social) media, interaction becomes socially more rewarding, in particular when all elements for collaboration are available in the virtual world, and can be accessed instantaneously. The applied Room design principle enables respective arrangements and corresponding user controls.

Conditional user or system behavior can be captured by the DPT modeling approach, e.g., when a Metaverse/XR glass cannot be used for virtual interaction when a user is crossing a street in the physical word. It can either be encoded as part of a regular behavior sequence, or handled as an exceptional case, using a particular DPT design and modeling construct, i.e., Message Guards. The latter are designed to interrupt regular behavior sequences, whenever specific events, such as crossing a street, occur. The design can be experienced in real-time, delivering impactful feedback for design, when the DPT is validated. Hence, Metaverse/XR application behavior can be experienced and observed with respect to its impact on the concerned cyber-physical ecosystem. This aspect is of importance when critical behavior changes need to be developed, e.g., consumptions of system resources in the context of sustainability.

The behavior abstractions of DPT models do not only allow for developing behavior variants for particular situations, but also for embedding AI/ML components and algorithms representing machine intelligence, in order to facilitate decision-making processes [27]. The designer decides whether a black box approach is followed or processing details are captured to better understand and interpret results when AI components are embedded in behavior models [73,74]. In both cases, input parameters capturing the current situation of use need to be provided and derived from the behavioral context of the concerned element of the design space.

The introduced DPT models follow both a functional abstraction and a communication abstraction, as recently proposed [75]. In line with this study, it also aims to simplify the management of communication between application components and actors. Similar to the proposed intent-based messaging system, high-level communication acts are represented in the DPT models. Rather than representing goals in application operation, messages are exchanged that can carry data to meet the request of a sender. The receiving design element is expected to deliver a specific output triggered and/or enabled by the transmitted message. Like for intent-based communication, the request from the sender does not contain information on how the request needs to be processed to deliver an expected output. The task accomplishment is encapsulated and detailed in a behavior model of the receiver. In this way, the presented DPT design approach abstracts from application operation data, including users and Metaverse/XR application components.

Finally, layering of design representation aims at better system understanding for humans [76]. The aligned layered abstraction allows handling complex interactions and architectures of heterogeneous applications [77]. It needs to couple higher-level processes, such as business operations to design and engineering processes [78]. Hence, the proposed DPT design approach follows findings on developing DTs of CPS with respect to the need for high-level abstraction and refinements along design and engineering processes [79,80]. The generic development and adaptation mechanism can work in several ways. In case of middle-out design, the focus is on an intermediate SID-layer; in case of top-down design, the initial SID is the starting point. Bottom-up design starts with a subject’s SBD and needs to be complemented with a corresponding SID.

The proposed high-level language goes beyond existing applications of process modeling and represents design models [81], and also allows for IoT process modeling [82] on a single layer of abstraction. Finally, the presented capabilities for digital validation and simulation meet the transformation requirements triggered by technological advances [83].

Although the research enablers implementing ‘Let everyone know (and design) the design’ ([65], p. 135), the development practice requires recognizing Metaverse/XR applications as part of ecosystems that are characterized by evolving design spaces. It remains to be tested how many design cycles are required to arrive in some ‘state of dynamic stability’, as additional epistemological burdens to bridge the abstraction gap for identifying relevant tasks are likely to be accomplished, e.g., the effect of refinements to handle crosscutting concerns, such as socio-technical autonomy [84]. We are experiencing only the beginning of embedding complex transactions in virtual 3D worlds with users being immersed through dual user interfaces.