High-Level Process Modeling—An Experimental Investigation of the Cognitive Effectiveness of Process Landscape Diagrams

Abstract

Unlike business process diagrams, where ISO/IEC 19510 (BPMN 2.0) prevails, high-level process landscape diagrams are being designed using a variety of standard- or semi-standard-based notations. Consequently, landscape diagrams differ among organizations, domains, and modeling tools. As (process landscape) diagrams need to be understandable in order to communicate effectively and thus form the basis for valid business decisions, this study aims to empirically validate the cognitive effectiveness of common landscape designs, including those BPMN-L-based, which represent a standardized extension of BPMN 2.0 specifically aimed at landscape modeling. Empirical research with 298 participants was conducted in which cognitive effectiveness was investigated by observing the speed, ease, accuracy, and efficiency of answering questions related to semantically equivalent process landscape diagrams modeled in three different notations: value chains, ArchiMate, and BPMN-L. The results demonstrate that BPMN-L-based diagrams performed better than value chain- and ArchiMate-based diagrams concerning speed, accuracy, and efficiency; however, subjects perceived BPMN-L-based diagrams as being less easy to use when compared to their counterparts. The results indicate that differences in cognitive effectiveness measures may result from the design principles of the underlying notations, specifically the complexity of the visual vocabulary and semiotic clarity, which states that modeling concepts should have unique visualizations.

1. Introduction

Organizations strive for operational excellence and competitive advantages in the rapidly evolving space of modern business. The resulting digital transformation and continuous improvement efforts highlight the importance of business process management (BPM) [1], which commonly needs to expand its unit of analysis beyond a single process [2], managing large collections of organizational process models [3]. Therefore, a fundamental aspect of BPM is process architecture, which encompasses designing, implementing, and optimizing various interconnected organizational processes [4]. It also serves as a blueprint that guides the allocation of resources, decision making, and the overall performance of an organization.

A process landscape diagram represents a process architecture’s top-level part [5] (p. 44). It enables a shared understanding of business processes and their relationships across various departments and stakeholder groups. A process landscape model differs from underlying process models since it represents business processes in a ‘black box style’, focusing on relationships between processes and external participants. As such, it enables an organization to acquire and maintain an overview of the processes, simplifying process-related interactions and representing a starting point for more detailed process discovery. Accordingly, a process landscape diagram must be understandable by the focal stakeholders of an organization [5] (p. 44) and [6], which implies using a standard, compact, and intuitive visual language.

In contrast to process modeling, where modeling with BPMN 2.0 is de facto and ISO standardized (ISO/IEC 19510), process landscape diagrams lack formal standardization and consolidation. Consequently, organizations design and apply their own ‘overviews of processes’ such as value chain diagrams, process folder structures, and different BPMN-based semi-formal approaches [7], each supporting a subset of essential landscape concepts. As a result, process landscape diagrams differ conceptually, semantically, and visually. The lack of a shared landscape-modeling language increases threats of inferring a wrong meaning from the depiction of a landscape diagram, which can have negative implications on the decisions made.

A solution to these challenges has been recently introduced as a standardized BPMN 2.0 extension, namely BPMN-Landscapes (BPMN-L) [8]. BPMN-L specifies a set of specific elements for process landscape modeling and reuses existing BPMN elements where applicable. The focal BPMN-L element remains ‘Process’, which represents a ‘meeting point’ between the process ‘interior’ and process ‘exterior’. In this manner, BPMN enables the decomposition and analysis of individual business processes, whereas the extension (BPMN-L) effectively organizes and interrelates processes. A design-science-based validation and analytical evaluation of BPMN-L have already demonstrated the visual compatibility of BPMN-L with commonly applied landscape-modeling approaches, especially value chains. In addition, BPMN-L is also capable of multi-abstraction-layer process modeling [9]. The investigated process landscape diagrams also demonstrated that BPMN-L concepts have practical applicability. It was concluded, at an analytical and theoretical level, that BPMN-L outperforms present process landscape designs; however, this was not tested empirically.

This paper extends previous research by empirically evaluating the effectiveness of typical process landscape designs, including BPMN-L. As already stated, diagrams have to make human communication more straightforward and effective, and this can be realized by reducing the cognitive load [10]. A diagram’s cognitive effectiveness does not naturally occur; instead, it must be intentionally crafted into them through language design, as was systematically considered in the design of BPMN-L. In this light, we defined the following research question, which could be tested empirically—RQ: Are process landscape diagrams modeled in BPMN-L more cognitively effective than their counterparts?

Accordingly, this paper is organized as follows. Section 1 already identifies the problem and motivation for the research. Section 2 contains background on the landscape part of process architectures, typical process landscape designs, and cognitive effectiveness in diagrammatic communication. It ends with a review of related work. Section 3 presents experimental research, which was performed to provide answers to the research question. Section 4 presents and analyzes the results and tests the stated hypotheses. Section 5 discusses the findings, limitations, implications, and future work, and Section 6 concludes the paper.

2. Research Background

This section discusses the foundations of process landscapes, common process landscape designs with practical applications, the concept of cognitive effectiveness in diagrammatic communication, and work related to empirical investigations of the cognitive effectiveness of process landscape designs.

2.1. Process Landscapes

A comprehensive framework depicting the organizational structure and inter-related business processes has emerged as a valuable tool for process-oriented companies in managing extensive collections of business processes. Tracing back to the early 1980s with Porter’s introduction of the value chain model [11], this concept is commonly referred to as a ‘process landscape,’ illustrating a network of interconnected processes within an organizational system. It is also known as a ‘process overview’ or ‘process map,’ although Poels et al. [12] suggest that the term ‘process map’ might encompass either a model of business process architecture or an introductory-level representation (i.e., process landscape) of a business process model architecture [13].

Figure 1 reveals that process landscape diagrams represent the ‘umbrella’ part of a process architecture, which is defined as: “an organized overview of business processes that specifies their relations, which can be accompanied with guidelines that determine how these processes must be organized” [4]. A process landscape model illustrates the arrangement, linkage, categorization, modularization, operational aspects, and technological aspects of business and operational processes. Unlike business process models, processes at the landscape level are depicted as ‘black boxes’, concealing their internal intricacies to prioritize simplicity and clarity.

Process landscape diagrams serve various purposes, catering to business-oriented and technically minded users [14]. At the macro level, they highlight diverse relationship types or dependencies with other processes and artifacts [15]. These diagrams aid process owners, quality managers, and other stakeholders involved in processes by providing a swift overview, thus facilitating process maintenance. Subsequently, in detailed process diagrams, individual business processes can be further broken down into finer levels of detail, such as sub-processes and tasks. To summarize, just as modeling individual processes serves as a starting point for any process-enhancement endeavor, modeling an organization’s collection of its business processes’ architecture is indispensable for any analysis, design, or enhancement endeavor extending beyond the realm of individual processes [12].

2.2. Common Process Landscape Designs

Recent work identified three common approaches to visualizing or managing process landscape diagrams [7]: BPMN-based, value chain-based, and process folder-structure-based, which allow professionals to create business process hierarchies. However, while folder structures do not necessarily apply visual languages, this approach has been excluded from our research.

Evidence from academia and practice shows that business process model and notation (BPMN) is commonly used to model process landscapes. However, while standardized BPMN 2.0 does not formally support process landscapes, non-formal approaches are applied, commonly tending to violate BPMN specification [16,17]. In our related work [17], three different BPMN-based techniques for modeling process landscapes have been analyzed: (1) the use of abstract BPMN Collaboration diagrams, (2) the use of BPMN Conversation diagrams, and (3) the use of ‘enterprise-wide’ BPMN process diagrams. Based on a conceptual analysis, it was found that none of the approaches support the minimum set of concepts necessary for effective process landscape modeling. In addition, these approaches are visually non-compliant with standard landscape designs. For example, in a BPMN-based approach, a process is primarily represented by a BPMN pool, which is depicted as a rectangle. This visually opposes typical landscape designs, in which the dynamic nature of a process is depicted with an arrow symbol or icon. The limitations of standard BPMN 2.0 in landscape modeling have been addressed in BPMN-L, designed by considering the standard BPMN 2.0 extension mechanism, minor interventions into BPMN structure, and commonly applied process landscape designs [8].

In contrast to BPMN, value chain diagrams indicate an organization’s strategy through interconnected models and activities and are often linked to business processes that form the operational backbone [7]. Value chains have roots in the early 1980s, when Porter [11] introduced the value chain model to provide a process view of an organization and represented it as a set of core activities a company must conduct to create customer value. Scheer and Nüttgens [18] adopted the value chain concept by introducing a diagram that represents those processes that create value for the company. In contrast to Porter’s value chain, the processes are arranged in sequence, with each process capable of being broken down hierarchically into subprocesses necessary for the execution of a higher-level process [18]. Value chain diagrams are common in practice; however, the notation’s visual and structural parts (i.e., concrete and abstract syntax) are not formalized and tend to differ among domains and tools. Our research uses an extended version of value chain notations, as investigated in [19] (p. 45) and [20].

In addition, ArchiMate, a technical standard from The Open Group, can be used to model process landscapes. ArchiMate was primarily created to address the shortcomings found in existing IT and business-modeling languages. These deficiencies included ambiguous connections between domains, a lack of model integration and well-defined semantics, a weak formal foundation, and an absence of a comprehensive architectural vision. Within ArchiMate, the Business Layer introduces essential concepts like Role, artifact, Business Actor, Interface, Collaboration, process, service, and event to model the business-architecture domain effectively [21]. Accordingly, the application of ArchiMate can support organizations in understanding, analyzing, and optimizing their process architectures for improved operational efficiency and effectiveness. ArchiMate does not entirely replace the present low-level modeling standards such as BPMN or UML; instead, it can be used in conjunction with them [22]. The latest version of the ArchiMate Specification is 3.2, released in October 2022.

Nevertheless, Malinova also proposed a language dedicated to designing process maps (i.e., process landscapes) [23]—Process Maps Language. The language’s concrete syntax, semantics, and abstract syntax were systematically specified using an exploratory and empirical approach consisting of real-world process map examples.

Table 1. Visual vocabularies of common process landscape designs, including BPMN-L.

Notation	Process Composition (Relationship)	Process	Group (Category)	Trigger Relationship	Specialization Relationship	Management Process	Support Process	Information/Artifact Flow Relationship	Participant	Conditional Trigger Relationship
Value chain (extended)
ArchiMate
BPMN 2.0
BPMN-L
Process Maps Language [23]

shows the visual vocabularies of the above-presented notations for modeling process landscapes.

Table 1 reveals that BPMN-L and the Process Maps Language are the most complete and visually compatible with (extended) value chain notation. In BPMN-L, only the information flow concept has a distinct depiction compared to value chains. Although the Process Maps Language does not directly support participants, it specifies Inputs and Outputs as entities that trigger a process and represent the results of a process. In contrast to the Process Maps Language, BPMN-L is compatible with the visual grammar and concepts of BPMN 2.0, where message flows are specified for “process-to-process” communications. BPMN-L also shares the exact same depictions as BPMN in the cases of the shared concepts. The triggering and conditional triggering relationships also share the visual representation of the BPMN sequence and conditional flows. This could be considered a symbol deficit; however, it is rational since the concepts are applied on different diagrams sharing analogous semantics (they specify a sequence of operations). Regarding semiotic clarity, which stipulates that “there should be a one-to-one correspondence between semantic constructs and graphical symbols” [24], one symbol overload (i.e., two or more elements sharing equal representation) is identified in the case of ArchiMate’s process group and process composition. Moreover, one symbol redundancy (i.e., a single element having two or more representations) is identified in the case of BPMN-L process composition. Compared to value chains, ArchiMate lacks elements for modeling types of processes, artifact flow relationships, and participants with rather unconventional visualizations for the modeling of process-related elements.

2.3. Cognitive Effectiveness in Diagrammatic Communication

While the primary aim of a visual language is to facilitate human communication and problem solving [25], its quality is related to how well visual language-based information is conveyed between involved parties. This aligns with an SLR (systematic literature review) performed by Dikici et al. [26], in which it was found that the modeling notation is the primary process model-understandability factor. Aligned with these premises, Mody [24] conceptualized the cognitive effectiveness of diagrammatic notations on the widely accepted Shannon and Weaver’s communication theory [27], which states that “The effectiveness of communication is measured by the match between the intended message (sent information) and the received message (received information)”. Accordingly, a precondition for effective diagrammatic communication is a clearly defined and understandable visual language.

Cognitive effectiveness is specified as “the speed, ease, and accuracy with which the human mind can process a representation” [28]. Cognitive effectiveness is not an inherent property of diagrams but has to be designed into them. For this purpose, nine principles have been delineated for crafting visually effective cognitive notations [24]: semiotic clarity, perceptual discriminability, semantic transparency, visual expressiveness, complexity management, cognitive integration, dual coding, graphical economy, and cognitive fit. Collectively, these principles constitute a design theory known as the Physics of Notations (PoN), emphasizing the physical (perceptual) attributes of notations rather than their logical (semantic) aspects.

Furthermore, according to Caire et al. [29], semantic transparency is crucial to designing comprehensible visual notations for inexperienced users. This property ensures that the meaning of a symbol is evident (i.e., intuitive and transparent) from its visual representation alone, akin to onomatopoeia in spoken languages. Hence, enhancing semantic transparency is one of the most effective methods for improving comprehensibility, particularly for novices [29]. A recently performed systematic literature review [30] found that cognitive effectiveness is not entirely confirmed have a relationship with semantic transparency, but only with some of its variables. This might have to do with the interrelations of all nine principles for designing effective visual notations and other factors.

2.4. Related Work

The review of related work focused on the empirical investigations of visualizations of business process diagrams and, more specifically, on the cognitive effectiveness of process landscape designs. A systematic literature review (SLR) related to the visualizations of business process models [31] found that in the past decade, out of 46 investigated papers, exactly half of them were based on BPMN, 52.17% presenting evaluations or validations of their proposals, while 35% of these evaluations or validations involved experiments with users. In total, 13% of the articles investigate hypotheses, and 15% present results that are considered statistically significant. Due to the widespread use and standardization of BPMN, the authors conclude that more research should be performed to improve knowledge of this standard, specifically its visual representation.

Besides Moody’s Physics of Notations (PoN) [24], numerous papers have investigated and applied cognitive effectiveness. In [32], the cognitive effectiveness of the “i* visual notation” was improved, and in [33], WebML was analyzed with respect to the same construct. Diagrams in information systems were improved considering their cognitive effectiveness in [34], and a feature for model visual syntax was empirically tested in [35]. The cognitive effectiveness of visual instructional design languages was investigated in [36]. An SLR was also performed in the visual-languages domain to assess the scope and verifiability of PoN applications [37], in which seventy-two notations were identified as being designed or evaluated concerning the concepts impacting the cognitive effectiveness of visual representations. Regarding the adherents of PoN, the systematic literature review (SLR) has revealed that numerous authors implementing PoN principles in their applications assert their usage without providing substantial evidence to support their design rationale. Additionally, a deficiency in user-based research was observed, with only a limited number of studies assessing the influence of PoN application on the cognitive effectiveness of the notation. One of the investigated papers also analytically evaluated the cognitive effectiveness of process maps (i.e., process chains) via PoN principles [37]. The study revealed that while certain process maps employ diverse visual elements to depict their processes in alignment with their objectives, the majority pay scant regard to the design of the process maps. Consequently, due to the multitude of symbols utilized to represent identical process categories, notable symbol redundancy and overload were detected.

Specifically in the domain of process landscapes, several papers on new or improved process landscape designs have been published with the following evaluations. The PICTURE method for documenting process landscapes, which was proposed by Becker et al. [6], included a case study-based comprehensibility assessment of models. The proposed documentation of process landscapes proved viable and efficient; however, the research lacks clear conceptualization and operationalization. In 2016, Dijkman et al. [4] analyzed prevailing approaches to designing business process architectures. The evaluation of approaches was performed analytically and based on a case study. It showed that practitioners prefer using approaches based on reference models and approaches based on identifying business functions or objects. Simultaneously, the assessments indicated that practitioners employ these approaches in conjunction rather than opting for a singular approach. In her thesis, Malinova [23] presented a language for designing process maps (i.e., process landscapes), which were experimentally evaluated for cognitive effectiveness against ‘conventional’ process maps. The performed testing confirmed three of the four hypotheses favoring the proposed language’s score, time, and perceived difficulty. Only the efficiency was not confirmed, although the means indicate a slight difference between the efficiency of both treatments. In 2019, Gonzalez-Lopez and Pufahl [14] proposed a landscape-modeling language for knowledge-intensive processes based on BPMN and CMMN (case management model and notation). The language was empirically evaluated via an online experiment in which interpretation effectiveness, interpretation effort, and interpretation efficiency were measured. The results indicate that interpreting an fCM (fragment-based case-management language) landscape might be more efficient and effective than interpreting an informationally equivalent case model.

In summary, the performed literature review did not reveal papers that would empirically investigate and compare the cognitive effectiveness of common process landscape designs, which have been experimentally investigated in our research, namely BPMN-L, value chains, and ArchiMate.

3. Empirical Research

Empirical research was performed to provide an evidence-based answer to the stated research question. We chose an experimental approach so that we could compare different landscape-diagram designs. The following subsections describe the conceptual research model, followed by a description of the design of the experiment. The latter includes a definition of the subjects and their sampling, experimental process, and the corresponding research instruments.

3.1. Research Model and Process

By considering the research question and research background, we anticipated that the investigated landscape-diagram designs would differently impact the concepts related to cognitive effectiveness as follows (Figure 2).

Figure 2 represents the scope and conceptualization of the experiment. An independent latent variable (construct) was defined, namely the diagram’s notation, with three levels, BPMN-L, ARH (ArchiMate), and VLC (value chain—extended version), representing three popular landscape designs, presented in Section 2.2. To investigate how a diagram’s notations impact cognitive effectiveness, we observed three dependent variables corresponding to the definition of cognitive effectiveness—the speed, accuracy, and ease of processing of the diagram’s representation by the human mind. The accuracy [AC] was primarily measured in the percentage of correct answers to diagram-related statements, with higher values preferred. In addition, we observed the number of undecided answers [AU] chosen by subjects who did not confirm or deny the stated questionnaire statements. The second dependent variable was speed, representing the rate at which someone operates. It was measured by the answering time [TA] of a specific questionnaire part related to a particular diagram, with lower values preferred. Because participants could finish answering the questions faster but with a lower accuracy [38], another dependent variable, namely, efficiency [EF], was introduced, measured as a quotient between correct answers and answering time, with higher values preferred. The ‘ease’ part of cognitive effectiveness was conceptualized with TAM’s perceived ease of use [PEOU] [39]. In line with its definition, it was operationalized by the level of agreement with standardized statements based on a seven-level Likert scale, where an average value of Likert items was used for statistical analysis. Other authors have also applied similar dependent variables and their corresponding measurements [28,40,41,42]. In addition, task effectiveness (i.e., accuracy) and task efficiency (i.e., speed) have been identified as the dominant concepts in measuring the understandability of process models, followed by subjective measures such as PEOU [26].

The alternative and null hypotheses were formulated according to the research model and its operationalization (

Table 2. Formal definition of the experiment hypotheses.

Research Hypothesis (HX)	Null Hypothesis (H0X)
H1: AC (BPMN-L) > AC (ARH, VLC)	H01: AC (BPMN-L) = AC (ARH, VLC)
H1AU: AU (BPMN-L) < AU (ARH, VLC)	H01AU: AU (BPMN-L) = AU (ARH, VLC)
H2: PEOU (BPMN-L) > PEOU (ARH, VLC)	H02: PEOU (BPMN-L) = PEOU (ARH, VLC)
H3: TA (BPMN-L) < TA (ARH, VLC)	H03: TA (BPMN-L) = TA (ARH, VLC)
H4: EF (BPMN-L) > EF (ARH, VLC)	H04: EF (BPMN-L) = EF (ARH, VLC)

Note: AC—accuracy; TA—answering time; AU—undecided answers; EF—efficiency; PEOU—perceived ease of use; VLC—value chain; ARH—ArchiMate; and BPMN-L—BPMN-Landscapes.

). Since BPMN-L was systematically designed for intuitiveness and completeness [8], we presumed that BPMN-L would outperform other landscape designs concerning cognitive effectiveness-related measures.

Because the visual vocabularies of ArchiMate and value chain contain only a subset of the elements used in process landscape modeling (see Table 1), we could not perform three independent treatments as the levels of the independent variable would imply. Instead, we were only able to compare them partially against BPMN-L. Consequently, all stated hypotheses came in two variants: testing BPMN-L against the value chain (variant A) and testing BPMN-L against ArchiMate (variant B). A within-group research design was applied with a randomized assignment of treatments to minimize any possible learning effects because of the sequence of the testing diagrams.

3.2. Research Subjects

Parsons and Cole [43] stated that subject matter experts should not be used to investigate cognitive effectiveness. Although this compromises the experiment’s external validity, the authors contend that participants should extract information solely from the diagram rather than relying on their background knowledge. Consequently, the optimal research participant would understand landscape modeling concepts and semantics but lack experience with the corresponding landscape modeling notations. Accordingly, the first Bologna stage IT students from our faculty, who had no previous formal education in the investigated process landscape designs, were selected as suitable candidates for the research. The sample selection aligns with the findings of Dikici et al. [26], who performed a systematic literature review on the understandability of process models. Their analysis showed that using students in the experiments is the dominant approach in the field, as 88% of the investigated studies used student populations, arguing that they are adequate proxies for novice analysts. In total, 588 subjects were invited to participate in the research. Three hundred ninety-one (391) subjects partially participated in the research, whereas 298 completed it.

3.3. Research Design and Instruments

A typical setup to measure cognitive effectiveness, specifically accuracy, involves a set of questions related to the process models used in the experiments to be answered by the participants while tracking the time that participants spend answering these questions [26]. Accordingly, the focal research instrument was an online questionnaire designed in 1KA, https://www.1ka.si/d/en (accessed on 8 March 2024), an advanced service for online surveys. The questionnaire (Appendix A) was structured into the following parts. In the first part, subjects were asked to provide basic demographic information (age and gender) and their expertise with common visual languages applied in landscapes and related modeling. In the second part of the questionnaire, landscape diagrams in three investigated notations were specified as follows (see Appendix A). Initially modeled in value chain notation, diagram D1, from [44], presented the landscape of product development involving the innovation process, product-planning process, product-development process, and interactions with stakeholders. Initially modeled in ArchiMate, diagram D2, from [19] (p. 45), showed the process landscape of ordering processes, including international and local order-fulfillment processes and check and delivery processes. Diagrams D1 and D2 were remodeled in BPMN-L afterward. Both variants of the D1 and D2 diagrams were information equivalent, meaning that the diagrams used as treatments of the experimental study provided the necessary information to answer the questions correctly [43]. Below a specific diagram, seven statements related to the investigated diagram’s semantics were provided, where the subjects were asked to give answers concerning their validity (‘true’, ‘false,’ or ‘undecided’). In line with the second Parsons and Cole [43] criterion, these statements focused on measuring semantics conveyed by modeling constructs in a diagram without reasoning beyond the information presented. Accuracy was primarily measured by counting the correct answers to these statements. Below them, Likert-based items for measuring the perceived ease of use (PEOU) were specified, and subjects were asked to provide their level of agreement with those items on a seven-level Likert scale. The four investigated diagrams and corresponding statements were presented randomly to subjects to minimize learning effects. Every question was presented on an individual page, whereas the system automatically stored the time stamps when pages had been switched. The duration of a particular diagram and the corresponding statements were obtained by subtracting subsequent time stamps, which were afterward used for measuring speed and efficiency. The instrument was prepared in the Slovenian and English languages and was wholly anonymized.

4. Results

A data analysis was performed using a spreadsheet application and SPSS statistics v29. The data analysis results are graphically represented within the boxplot diagrams. The boxes represent 25 to 75 percent of the responses. The entire range of values the participants chose is indicated by horizontal markers placed outside the boxplot. The whiskers present minimum and maximum values, whereas the midline indicates the median value. To test the differences between the treatments, several paired t-tests on two independent data sets (variant A: testing value chains vs. BPMN-L on diagram D1 and variant B: testing ArchiMate vs. BPMN-L on diagram D2) were performed, which measured whether the means from a within-subjects test group varied over the testing conditions, i.e., diagrams modeled in the investigated notations. The results were considered significant at the α < 0.05 level. Due to the discrete nature of individual Likert items used for the PEOU and PU measures, we could not determine the significance of the differences between their means [45]. However, this precondition was met in the total scores (i.e., Likert scales), which tended to be more normally distributed [46]. The data supporting reported results can be found at https://data.mendeley.com/datasets/zwy5znntjz/1 (accessed on 8 March 2024).

4.1. Data Cleaning

Initially, the cleaning of raw data was performed. First, the raw data were scanned for any missing values, of which none were identified. This was anticipated since all the measurements were obtained via mandatory closed questions or automatically, as in the case of time measurements. Accordingly, no experimental participants were excluded from the data analysis. Second, even though in moderate or larger sample sizes, the t-test may be reasonably accurate even when the normality assumption is violated, we analyzed the skewness and kurtosis of the continuous variables. The highest skewness and kurtosis values were detected with measures of AU, so we excluded them from the performed t-tests. The average skewness of the remaining measures was 0.432 and kurtosis 0.594, which is relatively safe. Additionally, for samples larger than 100, the analysis is usually not degraded even if skewness and kurtosis are higher. Third, the data set was scanned for extreme outliers. SPSS also considers any data value an extreme outlier if it lies outside the following ranges: third quartile + 3 × interquartile range and first quartile – 3 × interquartile range. We performed a winsorizing strategy to identify extreme outliers to the next highest value.

4.2. Descriptive Statistics

As mentioned, 298 subjects completed the questionnaire, with 161 reporting being male (54%) and 137 female (46%). On average, subjects were 21.3 years old and completed the questionnaire in eight minutes and four seconds. The subjects were also asked about their expertise with common modeling languages used in process and landscape modeling by answering predefined statements on an ordinary scale (Figure 3).

As evident from Figure 3, most subjects reported being unfamiliar with any of the stated modeling notations (the answer ‘No’ was chosen in an average of 65% of cases). On average, 7% of subjects reported already testing and using the listed modeling notations in an academic setting. As students may already cooperate and work in companies, only 1% reported using the stated modeling notations in a professional setting. As was also evident from Figure 3, subjects were asked about their expertise with two additional notations, namely UML and ARIS EPC. This was done since UML and ARIS EPC share some commonalities in the investigated process landscape designs, such as the generalization relationship in a UML class diagram. Based on the results, it was concluded that subjects were inexperienced in the investigated and related modeling notations, a precondition for a valid investigation of the cognitive effectiveness of the stated landscape designs.

4.3. Hypotheses Testing

The results of the hypothesis testing were structured following the scales of the dependent latent variables of the experiment into accuracy measures (AC and AU), a speed measure (TA), and a TAM-based measure of ease (PEOU). Due to the discrete nature of the Likert items used for measuring PEOU, we were unable to determine the significance of the differences between the means of individual items [45]. However, this precondition was met by the total scores (an average of individual items’ scores), which generally tend to be more normally distributed [46]. Because the visual vocabularies of ArchiMate and value chains contain only a subset of elements used in process landscape modeling (see Table 1), we could not simultaneously test all the investigated modeling notations. Instead, as already stated, we compared them independently to BPMN-L via two independent landscape diagrams (D1 and D2, see Appendix A).

The boxplots in Figure 4 show the accuracy measure ‘AC’ results on a scale [0, 1], indicating the percentage of correct answers related to the stated diagrams with higher preferred values. When responding to diagram D1, the mean was higher when using BPMN-L notation, AC(D1-VLC) = 0.58 and AC(D1-BPMN-L) = 0.63. Less difference was identified when comparing the means of applied modeling notations to diagram D2; however, this was still in favor of BPMN-L: AC(D2-ARH) = 0.43 and AC(D2-BPMN-L) = 0.46.

Table 3. Accuracy-related statements for diagram D1.

ID	Accuracy-Related Statements (Diagram D1)	Valid?	VLC	BPMN-L
1	The »Market challenge« link represents the sequence of execution.	FALSE	26%	43%
2	The »product-planning process« can be conducted first.	TRUE	41%	57%
3	The »Prototype description« link represents the flow of information.	TRUE	75%	66%
4	The »product-development process« is always performed last.	TRUE	69%	76%
5	The »product-planning process« and the »product-development process« may be carried out simultaneously.	FALSE	68%	67%
6	»Stakeholders« are related to processes only through information flows.	TRUE	49%	57%
7	The »product-planning process« provides information to the »product-development process«.	TRUE	77%	74%

and

Table 4. Accuracy-related statements for diagram D2.

ID	Accuracy-Related Statements (Diagram D2)	Valid?	ARC	BPMN-L
1	»International order fulfillment« is completed first.	FALSE	39%	42%
2	»Order fulfillment« is executed before »checking«.	FALSE	48%	49%
3	»Order fulfillment« can be conducted in international and local variants.	TRUE	63%	70%
4	The »primary processes« are executed last.	FALSE	47%	43%
5	»Checking« and »delivery« can be performed simultaneously.	FALSE	78%	78%
6	The processes of »international order fulfillment« and »national order fulfillment« can be performed simultaneously.	FALSE	17%	26%
7	»Order fulfillment« is a »process«.	TRUE	11%	12%

represent the details of the measuring accuracy by representing the percentage of correct answers to the individual accuracy-related statements of the investigated diagrams D1 and D2 (see Appendix A). The validity of the statements is presented in a separate column (“Valid?”).

As is evident from the above table, the percentage of correct answers was in favor of the investigated value chain diagram D1 in three cases and the BPMN-L-based diagram D1 in four cases.

As is evident from the above table, the percentage of correct answers was in favor of the investigated ArchiMate diagram D2 in one case and the BPMN-L-based diagram D2 in five cases. In the case of statement no. 5, the results were equal.

Besides the percentage of correct answers, we also observed accuracy with the number of ‘undecided’ answers (AU), a response that subjects could select if they were unable or unsure of providing a precise answer to a specific statement (Figure 5). Lower values on a scale [0, 7], where 7 indicates the highest possible number of undecided answers related to the semantics of a specific diagram, were preferred.

The boxplots in Figure 5 show that ‘undecided’ answer (AU) values are non-normally distributed, with median values of 0, mean values between 0.81 and 1.00, and several mild (represented by circles) and extreme (represented by asterisks) outliers detected. The frequency analysis of AU responses indicates that in 51.8% of cases, subjects did not select this option. Concerning diagram D1, mean values were lower for value chain notation, AU(D1-VLC) = 0.81 and AU(D1-BPMN-L) = 0.95, whereas concerning diagram D2, mean values were lower for BPMN-L, AU(D2-ARH) = 1.00 and AU(D1-BPMN-L) = 0.96. Due to the specific distribution of the AU measure, Kolmogorov–Smirnov and the Shapiro–Wilk tests were performed, indicating highly non-normal distributions. Accordingly, the accuracy measure ‘AU’, indicating the number of ‘undecided’ answers, was not subject to further statistical tests.

The following graph (Figure 6) presents the ‘answering time’ measure (TA) boxplots that indicate speed construct, where lower values measured in seconds were preferred.

The results indicate that in both investigated diagrams (D1 and D2), lower mean values of the times of providing the answers to the statements were measured in the case of BPMN-L-based diagrams: TA(D1-VLC) = 74.89 s; TA(D1-BPMN-L) = 66.01 s; TA(D2-ARH) = 57.19 s; and TA(D2-BPMN-L) = 37.85 s. The boxplots in Figure 6 show that upper outliers had been detected, which is explainable since the time frame of providing answers was theoretically not limited.

While measuring speed with time may be invalid, since the measure also considers wrong answers, we additionally specified a measure that considered this limitation. Efficiency (EF) was defined as a quotient of correct answers divided by time, with the results presented in the following set of boxplots (Figure 7). Higher values were preferred in this case, with upper values theoretically limited to 100 [1/s], representing 100% of accurate answers responded to in one second.

The descriptive analysis shows that efficiency (EF) measures were higher in the case of BPMN-L-based diagrams, meaning that subjects, on average, needed less time to provide correct answers in the case of BPMN-L diagrams: EF(D1-VLC) = 3.81 [1/s]; EF(D1-BPMN-L) = 4.29 [1/s]; EF(D2-ARH) = 3.81 [1/s]; and EF(D2-BPMN-L) = 8.58 [1/s].

The last set of boxplots depicts the ‘ease’ concept, which was operationalized with an average value of TAM-based ‘perceived ease of use’ (PEOU) Likert items.

As is evident from Figure 8, eight outliers were detected in the lower bound of two treatments. The descriptive analysis shows that PEOU measures were lower in the case of BPMN-L-based diagrams, meaning that subjects perceived BPMN-L notation-based diagrams as less easy to use: PEOU(D1-VLC) = 4.69; PEOU(D1-BPMN-L) = 4.61; PEOU(D2-ARH) = 4.44; and PEOU(D2-BPMN-L) = 4.27.

Finally, several paired t-tests were performed, which measured whether the means from a within-subjects test group varied over the testing conditions (BPMN-L, ARC, and VLC). As already stated, we could test them only pairwise due to the differences in the sets of supported concepts in the investigated notations. As evident from

Table 5. Results of paired-samples t-test.

Paired-Samples Test	Paired Differences					t	df	Significance
	Mean	Std. Dev.	Std. Error Mean	95% Conf. Interv. of the Diff.				One- Sided p	Two- Sided p
				Lower	Upper
AC(D1-VLC)– AC(D1-BPMN-L)	−0.05	0.25	0.01	−0.08	−0.02	−3.45	297.00	<0.001	<0.001
AC(D2-ARH)– AC(D2-BPMN-L)	−0.03	0.18	0.01	−0.05	0.00	−2.40	297.00	0.01	0.02
TA(D1-VLC)– TA(D1-BPMN-L)	8.88	41.98	2.43	4.10	13.67	3.65	297.00	<0.001	<0.001
TA(D2-ARH)– TA(D2-BPMN-L)	19.34	37.58	2.18	15.05	23.62	8.88	297.00	<0.001	<0.001
EF(D1-VLC)– EF(D1-BPMN-L)	−0.17	1.08	0.06	−0.29	−0.04	−2.63	297.00	0.01	0.01
EF(D2-ARH)– EF(D2-BPMN-L)	−1.52	2.68	0.16	−1.83	−1.22	−9.80	297.00	<0.001	<0.001
PEOU(D1-VLC)– PEOU(D1-BPMN-L)	0.27	0.98	0.06	0.16	0.38	4.80	297.00	<0.001	<0.001
PEOU(D2-ARH)– PEOU(D2-BPMN-L)	0.17	0.98	0.06	0.05	0.28	2.93	297.00	0.00	0.00

Note: AC—accuracy; TA—answering time; EF—efficiency; PEOU—perceived ease of use; VLC—value chain; ARH—ArchiMate; BPMN-L—BPMN-Landscapes; D1—diagram 1; and D2—diagram 2.

, all t-tests were significant, either observed as one-sided (as the stated directional hypotheses) or two-sided.

After considering the above results, it was possible to test the stated hypotheses (Table 2). We either failed to reject or rejected the null hypotheses in favor of alternative ones (i.e., research hypotheses). All stated hypotheses came in two variants: testing BPMN-L against the value chain (variant A) and testing BPMN-L against ArchiMate (variant B).

As is evident from

Table 6. Hypothesis testing.

Null Hyp.	Measure	Var.	Findings
H01	Accuracy (AC)	A	H01A was rejected in favor of H1A. When using a BPMN-L-based diagram, the reported mean of the percentages of correct answers was significantly higher (p < 0.001) when compared to the same diagram modeled in value chain-based notation.
		B	H01B was rejected in favor of H1B. When using a BPMN-L-based diagram, the reported mean of the percentages of correct answers was significantly higher (p < 0.05) when compared to the ArchiMate-based diagram.
H02	Perceived ease of use (PEOU)	A	H02A was rejected. When using a BPMN-L-based diagram, the reported mean of perceived ease of use was significantly lower (p < 0.001) than the value chain-based diagram (higher values were hypothesized).
		B	H02B was rejected. When using a BPMN-L-based diagram, the reported mean of PEOU was significantly lower (p < 0.01) when compared to the ArchiMate-based diagram (higher values were hypothesized).
H03	Answering time (TA)	A	H03A was rejected in favor of H3A. When using a BPMN-L-based diagram, the reported mean of the time of providing answers was significantly lower (p < 0.001) when compared to the value chain-based diagram.
		B	H03B was rejected in favor of H3B. When using a BPMN-L-based diagram, the reported mean of the time of providing answers was significantly lower (p < 0.001) when compared to the ArchiMate-based diagram.
H04	Efficiency (EF)	A	H04A was rejected in favor of H4A. When using a BPMN-L-based diagram, the reported mean of the efficiency (measured by the percentage of correct answers divided by answering time) was significantly higher (p = 0.01) when compared to the value chain-based diagram.
		B	H04B was rejected in favor of H4B. When using a BPMN-L-based diagram, the reported mean of the efficiency was significantly higher (p < 0.001) when compared to the ArchiMate-based diagram.

Note: variant A: BPMN-L vs. value chain; variant B: BPMN-L vs. ArchiMate.

, we rejected all the stated null hypotheses. Except for PEOU-related hypotheses, all were rejected in favor of the stated alternative hypotheses, favoring BPMN-L-based diagrams D1 and D2.

5. Discussion

The performed research resulted in the following new insights. According to the stated hypotheses, we found that BPMN-L diagrams performed significantly better than value chain-based diagrams and ArchiMate-based diagrams concerning (1) accuracy, which was measured by the percentage of correct answers, (2) speed, which was measured using the times of answers being provided, and (3) efficiency, which was measured by the quotient of first two measures. These findings align with the work of Malinova [23], who demonstrated that saturated process maps (i.e., one with visualizations of additional concepts) outperformed the control group, represented by a conventional process map with fewer visual elements. The additional measure applied to the accuracy construct, namely, the number of undecided answers, indicates that subjects were, on average, 13.3% insecure in agreeing or disagreeing with statements related to the semantics of investigated diagrams.

With respect to diagram D1 (see Appendix A), which was modeled in value chain and BPMN-L notations, the detailed analysis of the accuracy (Table 3) shows that in the case of interpreting the value chain-based diagram, lower mean values were reported in statements related to the interactions of stakeholders (i.e., participants) with processes (Table 3 statements 1, 2, 4, and 6). This may be theoretically explained with semiotic clarity, the central principle for designing cognitively effective visual notations [24], stating that modeling concepts should have unique visualizations. As value chain specifies equal visualization (solid arrow) for information flows and triggering relationships (Table 1), this could negatively impact subjects’ interpretations. In the case of the BPMN-L-based diagram D1, lower mean values have been reported, with statements 3 and 7 both being associated with information flows. Despite BPMN-L having a dedicated visualization for information flows, subjects could not infer the meaning from the depiction (element) and the context (diagram), as they reported being unfamiliar with the investigated notations. Figure 9 highlights the above-stated accuracy challenges when interpreting diagram D1.

With respect to diagram D2 (see Appendix A), which was modeled in ArchiMate and BPMN-L notations, the detailed analysis of the accuracy construct (Table 4) shows fewer differences in mean values when compared to D1. The highest differences between the mean values can be spotted in statements 3 and 6, which are associated with interpreting the relationships between “International order fulfillment” and “National order fulfillment” processes (Figure 10).

The visual comparison of fragments of processes in Figure 10 reveals that in the case of ArchiMate, similar arrow icons are used for the specialization relationship and for specifying processes (dotted circles). This may again imply challenges with the semiotic clarity of a notation, as two similar icons are associated with two different concepts. In contrast, BPMN-L visualizes process elements with a distinct shape rather than an icon. Moreover, ArchiMate has visually similar elements for representing different process landscape concepts (e.g., process composition, processes, and process groups), primarily consisting of rectangle shapes and arrow icons (Table 1). According to Winn [47], a greater visual distance between symbols increases the speed and accuracy of the recognition of elements, which may also explain the significantly faster provision of the answers in the case of the BPMN-L-based diagram.

Concerning the ease construct, which was measured with standard TAM’s PEOU Likert scale items, the results were (significantly) in favor of the value chain-based diagram (D1) and ArchiMate-based diagram (D2), meaning that BPMN-L-based diagrams D1 and D2 were perceived as more challenging to use. The PEOU findings oppose the stated hypotheses and may have roots in a more complex visual vocabulary of BPMN-L (Table 1). The results of the hypothesis testing may be summarized to provide an answer to the stated research question as follows. The design-science-based development of BPMN-L, which considered and systematically addressed conceptual and visual limitations of standard landscape designs, resulted in a landscape-modeling notation, of which the diagrams demonstrated being cognitively more effective than the investigated value chain and ArchiMate diagrams. This means that users may intuitively understand and interpret BPMN-L-based diagrams more precisely, faster, and efficiently, leading to better and quicker business (process-management) decisions. Probably due to the higher complexity of the visual vocabulary, BPMN-L-based diagrams outperformed the value chain and ArchiMate-based diagrams in terms of perceived ease of use. This perception of inexperienced users may not be as critical as in a real organizational setting, and a modeler may quickly catch up with this deficit through formal or informal training and experiences gained.

Based on these results, BPMN-L should be considered for end-to-end process modeling as it is visually aligned with typical process landscape designs, especially value chains. In simple terms, this means that modelers can shift from value chain to BPMN-L without perceiving any (visual) difference while benefiting from an extended and consolidated notation, as well as the ability to create a consistent business process architecture based on the standardized multi-abstraction-based modeling of business processes.

5.1. Research Limitations

The results of this research should be considered with the following internal and external validity limitations in mind. First, the experiment’s internal validity could be negatively impacted despite performing random assignments, as a within-group research design was applied. In such an experimental design, a carryover effect (i.e., learning effect) may occur when the impact of receiving one treatment affects participants in subsequent conditions. Concerning external validity, there is a certain risk of generalizing the results above the research sample. While students reported being not skilled in BPMN and landscape-modeling languages, it could be challenging for them to understand the research instrument’s context and semantics. Another group of subjects (e.g., experiences of process analysis) could interact differently with the research instrument, which could impact the observed variables. In addition, the investigated diagrams were relatively simple, whereas we have no insights into how the results would be impacted when using more complex or multiple diagrams. However, due to the rich vocabulary and more complexity-management mechanisms, we presume that BPMN-L would perform better than languages with less complex vocabularies and fewer complexity-management mechanisms. There is also a certain degree of risk related to measuring the time. Although it was recorded automatically via the questionnaire’s meta-data, the subject could unintentionally stay on a specific questionnaire page longer than necessary, impacting the speed-related results.

5.2. Implications and Future Work

A comprehensive review of the biggest BPM challenges [2] states that BPM will need to expand its unit of analysis and move from the study of single processes to the study of ‘big processes’. In this manner, ‘Expansive BPM’ was introduced, which extends current BPM practices by enacting three ‘critical lenses’ of an extended process architecture: (1) end-to-end process modeling, (2) multiple processes view, and (3) multi-disciplinary views. Process landscape designs are directly associated with the first two ‘critical lenses’ of Expansive BPM, whereas BPMN-L provides a standardized landscape extension of the BPMN 2.0 process modeling standard. So, at some point, it would be rational to consider integrating it in a potential forthcoming revision of BPMN as a part or an extension of process modeling conformance.

Stein Dani et al. [31] spotted the necessity and simultaneous lack of empirical research in visual notations, as presented in this work. Our work could also motivate other researchers to perform similar investigations in the domain, potentially enrolling subjects with different background knowledge and expertise and investigating different notations, scenarios, and levels of complexity. We also plan research in this direction. To improve external validity, we plan to repeat our research with more complex diagrams and subjects with expertise in process modeling. In this manner, we aim to estimate how these moderating factors impact the cognitive effectiveness of investigated notations, including the investigation of the cultural impacts on the processes’ visualizations [48].

BPMN-L is currently being applied to the BPM course at our university. Within the practical part of the course, students aim to digitally transform selected public-administration processes, which also considers the establishment of a three-level process architecture. On the level of landscape modeling, the students apply different modeling languages, including BPMN-L. Qualitative and in-depth insights will be acquired based on students’ interactions with applied languages, such as identifying the strengths, weaknesses, opportunities, and threats of applying BPMN-L. Additionally, the usability of the applied modeling languages will be assessed using a standardized questionnaire known as the System Usability Scale (SUS). In addition, while BPMN-L is currently implemented in a diagramming tool as a stencil (Microsoft Vision 2013), we plan to implement the proposed extension in an open-source modeling tool (e.g., bpmn.io) on the visual and meta-model levels. Such an implementation will enable new use cases for the process modeler, such as BPMN-L syntax verification, the serialization of BPMN-L models, and multi-level process modeling.

Nevertheless, since this research demonstrated that BPMN-L may be more challenging to use compared to its counterparts, we will investigate individual elements of the notation for their usefulness and intuitiveness, leading to potential improvements in the visual vocabulary. This may be based upon Moody’s principles for designing cognitively effective visual notations [24], which have been presented in Section 2.3. For example, as process landscape diagrams are commonly colored in practice, the visual expressiveness of BPMN-L could be improved by facilitating the use of retinal visual variables such as color, brightness, or texture. One of the proposed research directions could also be exploring and specifying the best practices for highlighting parts of process landscape diagrams, as the technique significantly impacts the cognitive effectiveness of process diagrams, including perceived ease of use [49].

Due to the emerging expansive BPM and the importance of process landscape modeling, we also hope other researchers will perform research in the process landscape domain. As Reijers [50] stated, this type of work (i.e., multi-layer process modeling and multi-view process modeling) may well open an entirely new angle for process modeling research.

6. Conclusions

Business process landscapes are essential for providing an overview of organizational processes, yet no unified modeling language exists. While these high-level process maps must be intuitively and effectively communicated between process stakeholders, we performed empirical research in which process landscape diagrams modeled in three different process landscape designs have been investigated, all being a de facto standard or representing a formal extension of a standard in the domain. The primary research construct was cognitive effectiveness, conceptualized with the speed, ease, accuracy, and efficiency of processing the investigated diagrams. The empirical research involving 298 subjects demonstrated that BPMN-L notation, which was systematically designed for its completeness and effectiveness, performed significantly better than the investigated counterparts in speed, accuracy, and efficiency. A detailed analysis of the accuracy construct indicates that differences in the associated cognitive-effectiveness measures may result from the design principles of the underlying notations, specifically semiotic clarity, which states that modeling concepts should have unique visualizations. On the contrary, value chain and ArchiMate-based diagrams performed significantly better than BPMN-L concerning the investigated diagrams’ perceived ease of use, possibly due to BPMN-L’s more complex visual vocabulary. Our work fits a research gap as a literature review identified a lack of empirical research in visual notation design. Specifically, no empirical comparisons of typical landscape designs concerning speed, ease, accuracy, and efficiency have been identified. With empirical insights in the still-not-standardized field of process landscape modeling, our research may contribute to the knowledge and consolidation of emerging Expansive BPM, which aims to capture the ‘true’ beginnings and ends of process models and focuses on multiple interrelated processes when solving a problem or capitalizing on an opportunity.