Investigating the Effect of Organization Structure and Cognitive Profiles on Engineering Team Performance Using Agent-Based Models and Graph Theory

Abstract

In large engineering firms, most design projects are undertaken by teams of individuals. From the perspective of senior management, the overall project team must maintain scheduling, investment and return on the investment discipline while solving technical problems. Various tools exist in systems engineering (SE) that can reflect the value provided by the resources invested; however, the involvement of human decision makers complicates most types of analyses. A critical ingredient in this challenge is the interplay of the cognitive attributes of team members and the relationships that exist between them. This aspect has not been fully addressed in the literature, rendering many studies relatively oblivious to team dynamics and organization structures. To this end, we propose a framework to incorporate organization structure using a graph representation. This is then used to inform an agent-based model where team dynamics are simulated to understand the effects of cognitive attributes and team member relationships. In this work, we aim to understand team dynamics in the context of product development. The organization is modeled using the Barabasi–Albert scale-free network. The information regarding member relationships can be acquired through graph metrics such as the various centrality measures associated with the members and the distance between them. This is then used to model the dynamics of the members when they work on a technical problem, in conjunction with their other cognitive attributes. We present some results and discuss avenues for future work.

1. Introduction

Engineering organizations are constantly looking to maximize return on investment and achieve clarity on the expected cost, time taken and the expected performance of the product or process they will realize at the end (see Figure 1). There are many impediments to designing and executing such projects. Systems engineering (SE) tools such as the VEE model [1], which is a graphical representation of system development lifecycle, provide a structure to the product and process development (PPD) process, but do not help with predicting the time and cost since they do not have the element of project management. Prevalent project management tools focus on time and resources but there is little focus on the technical methods. Involvement of human decision makers further complicates the process. In this work, we aim to understand team dynamics in the context of product development, or, more precisely, how a product or a system can be designed when different “human” aspects of collaboration are at play. Figure 1 gives a few examples of how different cognitive skills in team members can facilitate different project steps. Understanding these allows us to obtain an improved estimate of the number of resources that need to be invested, in terms of money, time and individuals. It may also help us to inform ideal organizational structure and team composition for projects.

While engineering design has been explored from various different perspectives, including optimization [2,3,4], reliability [5,6,7,8], manufacturability [9] and decision making [10], among others, there are limited studies of the human dynamics that are inherent to engineering organizations. To gain a better understanding of engineering project management, we clearly need to model the processes and people involved in executing them. We must also model the organizational structure so that the relationships between individuals are better understood [11,12]. Additionally, we also need to be able to propagate these inputs through a typical process and predict outcomes. To this end, we propose a framework to incorporate organization structure using a graph representation. This is then used to inform an agent-based model where team dynamics are simulated, to understand the effects of cognitive attributes and team member relationships. This is implemented on an engineering design problem as the groups progress through the design steps governed by engineering constraints. We then investigate the effect of the make-up of the design teams (for example, the relative distance between individuals within the organization) on the quality of the designs produced and present results. Through the framework proposed, in future work we aim to investigate scenarios such as team composition, team members’ cognitive attributes and graph realizations, among others, with hopes to make recommendations on organization structures and team formation in a general setting.

2. Background and Literature Survey

In organizations, at various levels, collective decision making is prevalent. Many organizations are increasingly hiring teams as opposed to individuals. Several studies have considered team member cognitive skills and their impact on the team’s performance [13,14,15,16]. It is argued that team members can have shared mental models that help form accurate explanations and expectations for a task. Another study, consisting of employees from different disciplines of an organization, posited that focusing on merely team cognition may not be sufficient [17]. One must also investigate the motivational aspects. Team dynamics have also been investigated in this context. Buffinton et al. [18] used the Kirton Adaption–Innovation Inventory (KAI) to determine cognitive styles. They concluded that cognitive style theory may be applied to characterize and better understand the personal dynamics of individuals working in teams. Jablokow et al. [19] used cognitive characteristics along with interaction behaviors to model the engineering problem-solving process. Detailed surveys of the literature can be found in [13] and [19]. In this work, we include the three personality traits from [14] as the factors that influence team performance. We enhance the application of team dynamics as discussed below.

A limitation in much of the literature in this area is that team member cognitive profiles are assumed in a simplistic fashion. Additionally, organization structure is not explicitly considered, which directly affects how team members interact with each other. The organization’s structure deeply affects whether certain team members have a working relationship (which could be negative) or not, or how new ideas propagate through teams and eventually organizations. There is a large body of literature on the study of business organizations where graph and network theory has been applied [20]. However, it is disconnected from engineering projects and product development. In this work, we address these limitations together. In this work, the organization graph directly affects the team simulation model. This also provides us with a way to feed back to the organization what is learned as part of the product development process. The overarching goal is to make recommendations regarding team formation and organization structures that best facilitate engineering projects.

Agent-Based Models

Although experimentation with human subjects would be more realistic [13], it poses huge survey and data collection challenges. Additionally, the results are almost never generalizable because they are deeply affected by the traits of the participants involved, and logistical challenges render it difficult to investigate a large number of scenarios. For this reason, many researchers have resorted to mathematical or computer modeling and simulation of realistic experiments [21]. An agent-based model (ABM) creates a representation of sentient decision makers on the computer that follows some given rules for their behavior [22]. In such models, one does not explicitly assume the dynamics of the system; rather, one simulates multiple such individuals or entities and measures the emergent behavior of the entire system. An example of this is the work by Lapp et al. [23]. Clearly, the issue of validation remains in such cases, so a benchmarking process is generally undertaken. For example, one examines a few general trends and observes if the output of the model matches what one would expect in real life. Once confident that the ABM can actually model the dynamics of the problem at hand, we can extrapolate it to other more complex scenarios.

3. Proposed Approach

The proposed approach can be divided into six steps as outlined below:

Step 1.

Model a representative organization as a graph with given properties.

Step 2.

Acquire graph invariants for the nodes to help select a subgraph (team).

Step 3.

Select a suitable subgraph to form a team for further analysis.

Step 4.

Select a suitable engineering design problem that the team will work on.

Step 5.

Using the design team thus generated, simulate the design process.

Step 6.

Acquire and interpret results.

3.1. Graphs

A graph is defined as an ordered pair $G=(V,E)$, where $V$ is a set of vertices (also referred to as nodes or points and $E$ is the set of edges. More specifically, $E\subseteq \left\{\right(x,y\left)\right|(x,y\in V \mathrm{a}\mathrm{n}\mathrm{d} x\ne y\}$, which states that an edge may exist between two distinct vertices and all such edges are encoded in the set $E$. In undirected graphs, the direction of the edge does not matter, while in directed graphs it matters where the edge originates and where it terminates. There are many application areas for graph models, including biology, chemistry, engineering, social networks and mathematics. In engineering in particular, they are used to study computer networks, product architectures, as well as many other applications.

3.2. Adjacency Matrices

Adjacency matrices are an alternate representation of graphs [24]. They are square matrices used to represent a finite graph. The entries of the matrix denote whether the node associated with the column is adjacent or connected to the node associated with the row or vice versa. In the simplest case, it can be a (0,1) matrix. denoting the existence or lack of connection between node pairs. Such graphs have zeros on their diagonal, indicating that a node is not connected to itself. Undirected graphs have symmetric adjacency matrices. The study of the adjacency matrix of graphs is performed under spectral graph theory. Other matrices associated with graphs are incidence matrices and degree matrices, with their own applications. The Figure 2 below shows a six-node graph and the corresponding adjacency matrix.

3.3. Random Graphs

To make general inferences about graphs and the real-life networks represented by them, we often need to consider random graphs. A random graph represents a probability distribution over graphs. In a mathematical context, random graphs refer to Erdős–Rényi random graphs, named after the researchers who originally proposed and studied them [25]. When we generate random graphs, we are often interested in the properties of the graphs that help them represent a network under question. For example, when analyzing a social network of a certain type, we may be interested in properly modeling the number of hubs, the distance to nodes, etc. For this reason, models such as the Barabasi–Albert and Watts–Strogratz models have been proposed in the literature. The Barabasi–Albert model is used in this work because it follows the power law distribution [26] and is frequently observed in social networks [27].

3.4. Scale-Free Networks

Scale-free networks or graphs are graphs that (asymptotically) follow a power law degree distribution. This implies that the fraction $p\left(k\right)$ of total nodes in the network with k connections to other nodes can be represented, for large values of $k$, as follows:

$$P\left(k\right)~k^{-\gamma }$$

where $\gamma$ is a parameter that defines a particular type of scale-free graph. This is a reasonable expectation in modeling engineering organizations using a graph since we expect certain properties of the organization to stay the same as it grows—particularly what fraction of individuals has a given number of connections. The value of $\gamma$ is 3 for the Barabasi–Albert model. It should be noted that simply preferential attachment is not enough to obtain the scale-free property, as growth is also required [28]. Both of these can be expected in real-life small- to mid-sized organizations that are increasing in business size and sales, and, therefore, number of employees. The BA model is suited to represent social networks and has been used for this purpose in the literature, for example by de Weck et al. [27].

3.5. Centrality Measures

Various centrality measures tend to be used to measure the relative “importance” of different nodes of a graph. Some commonly used ones are degree, closeness, betweenness and Eigenvalue centralities. They are briefly defined here:

• Degree: Counts the number of nodes incident on a node. This denotes how many direct connections a node has with other nodes.
• Closeness centrality: Measures the average distance of other nodes in the network to the given node.
• Betweenness centrality: Measures the number of pairs of other nodes that must go through a given node to reach each other.
• Eigenvalue centrality: Measures the relative importance or influence of a node by looking at the corresponding component of the eigenvector of the adjacency matrix.

These centrality measures can be used to select a project team. For example, including a member with high betweenness centrality can help with team cohesiveness since the member will contribute to bringing the team members together. Alternatively, it may also be possible that a member with high closeness centrality may negatively affect team progress because they may favor one individual over others and promote their own ideas expecting, often correctly, support from familiar individuals. The idea can also be extended to the idea of sub-teams, especially cross-functional teams, which are short-term teams tasked with a specific purpose. It is expected that expertise and familiarity within the sub-team will need to be balanced for the best results [23,29].

3.6. Engineering Design Problem

The representative product design problem that is solved in individual and team settings is similar to that in [14]. In this problem, the process of design is represented by designers solving the following optimization problem. A product with $n_c$ components is to be optimized by selecting the best variant for each component. Each component is assumed to have $n_v$ variants. Therefore, the decision vector $\mathbf{x}$ is an $n_c\times 1$ vector, with each element representing the corresponding variant and $\mathbf{x}\in {\left\{1,\dots ,n_v\right\}}^{n_c}$.

$$\begin{array}{l}\mathit{Minimize} f\left(\mathbf{x}\right)=\left\{\begin{array}{cc}0.2+0.1\mathrm{s}\mathrm{i}\mathrm{n}\mathsf{\omega }{f}_0\left(\mathbf{x}\right)+f_0\left(\mathbf{x}\right) & h\left(\mathbf{x}\right)\ge 2.5\\ \left(0.2+0.1\mathrm{s}\mathrm{i}\mathrm{n}{\omega f}_0\left(\mathbf{x}\right)+f_0\left(\mathbf{x}\right)\right)/2& 1\le h\left(\mathbf{x}\right)<2.5\\ \begin{array}{c} \\ 0\end{array} & h\left(\mathbf{x}\right)<1\end{array}\right.\\ \mathit{where} f_0\left(\mathbf{x}\right)=\left({\sum }_{i=1}^{n_c}\left|x-x_l^{*}\right|\right)/n_c; h\left(\mathbf{x}\right)=\left({\sum }_{i=1}^{n_c}\left|x-x_g^{*}\right|\right)/n_c \\ \mathit{Subject} \mathit{to}: x_i\in \left\{1,\dots ,n_v\right\}\forall i\in \{1,\dots ,n_c\}\end{array}$$

This non-convex optimization problem aims at minimizing the function $f\left(\mathbf{x}\right)$ as defined. The objective function is chosen to notionally have characteristics of a representative optimization problem. This is accomplished as follows. The function is lowest near the global minimum $x_g^{*}$, but also takes small values near the local minimum,$x_l^{*}$. First, we define a normalized deviation from the global minimum using $h\left(\mathbf{x}\right)$ and a normalized deviation from local minimum as $f_0\left(\mathbf{x}\right).$ Then, we define regions in the feasible set based on the value of $h\left(\mathbf{x}\right)$. Within each region, the objective function value is a function of $f_0\left(\mathbf{x}\right)$, the distance from the local minimum. The ability of a decision maker to find the local or the global minimum is a function of their cognitive attributes, which are denoted by scalars $l, t$ and $c$, lying between 0 and 1, measuring leadership, technical ability and creativity, respectively. A decision maker with a high technical aptitude will approach the local minimum systematically, while a creative decision maker is more likely to find the global minimum; this is explained in the next subsection. The sinusoidal noise, with frequency controlled by the parameter $\omega$, shown in the objective function, ensures that the designers cannot trivially find a solution—this is typical of real-life non-convex optimization problems, where each step may marginally improve or worsen a solution even if the general trend is good.

3.7. Implementing Individual Attributes in Individual and Team Settings

The designers first solve the problem separately (individually), with their performance being a function of their intrinsic attributes of technical ability, $t$, and creativity, $c$. They then bring their best solutions to a team meeting, where they again solve the problem; however, their leadership level and ability to collaborate determine the performance of the team. The cognitive attribute of leadership is also a scalar between 0 and 1 and the ability to collaborate is a function of the organizational architecture. We describe the details below.

Individual setting: In this stage, individuals independently solve the optimization problem. For each individual, we first randomly select the modality the decision maker uses to solve the problem: technical ability or creativity. The probability that the creativity modality is selected is equal to their creativity score, $c$, i.e., a more creative decision maker will prefer a creative solution over a technically derived solution. If a technical solution pathway is selected, then we add noise to the objective function equal to $0.1R_1\left(1-t\right)$ in the case that $h\left(\mathbf{x}\right)\ge 2.5$, where $R_1$ is a standard normal random variable and $t$ is the individual’s technical proficiency; see above. Therefore, a technically adept but creatively limited individual will likely prefer a technical solution and is likely to end up with the locally optimal solution. When the decision maker selects a creative pathway, the objective function is kept the same as before in the case that $h\left(\mathbf{x}\right)$ is between 1 and 2.5; while it is 0, i.e., the global minimum, when $h\left(\mathbf{x}\right)<1.$ This ensures that when a creative decision maker is close to the globally optimal solution, they quickly find it.

Team setting: The extrinsic attribute of familiarity, or closeness, is derived from the graph and is a matrix of distances between pairs of individuals. This allows for the optimization problem to be solved in a team setting, where the relative familiarity between individuals plays a part. In terms of implementation, in the team setting, an individual (individual i) begins the discussion with a probability proportional to their relative leadership skill, $l_i$. This implies that the objective function we use in the design optimization problem corresponds to the individual. Subsequently, after one GA generation, they hand over the discussion to another individual (individual j) for an $n$-member team with a probability $\frac{l_j{e}^{-\lambda d_{ij}}}{\sum _{k=1}^i{l}_k{e}^{-\lambda d_{ik}}}$, where $\lambda$ is a scaling parameter. This functional form is selected because it implements the following logic. An individual has a chance to talk and influence the solution to the design problem with a probability proportional to their intrinsic leadership level and inversely proportional to the distance from the previous speaker. In other words, individuals not familiar with each other are not as likely to hand over the discussion to each other unless the other individual has a high leadership level. The value of the parameter $\lambda$ controls how quickly the probability drops as a function of distance between individuals.

4. Results

In this section, we present the preliminary results acquired using the model presented so far. The initial network was generated in Matlab using the graph theory toolbox. For this work, we selected a 100-member organization. To model the organization architecture, we selected an initial Barabasi–Albert scale-free network as shown in Figure 3a below. Figure 3b confirms the scale-free property of the network, which points to the generalizability of the results on larger size organizations. A 100-member organization was selected because this is classified as a small- or midsize business, depending on the classification system. Organizations of this size start developing characteristics of a large business such as different internal divisions and large inter-member distance, without becoming as intractable to analyze as a much larger organization with thousands of employees would be. Additionally, in accordance with the growth and preferential attachment requirement for the scale-free property, a small- to midsize organization is generally expected to increase in size in the future, making the analysis scalable.

To make the organization structure more representative of real-life organizations, disconnected subgraphs and self-loops were then removed from the initial graph and the largest connected subgraph was derived, as shown in the following Figure 4. This is because in organizations, while there may be divisions that are separate with a certain degree of autonomy, it is highly unlikely that they have no connection with the rest of the organization. Also, self-loops do not make sense from an individual-to-individual connection standpoint. Looking at the structure generated, we see that it also appears representative of a typical organization with three “main” divisions denoted by the locations of hubs. This largest connected subgraph of the original graph generated was then used for further analysis, as we discuss below.

• Case 1: Team with smaller node-to-node distance

Using the pairwise distance between nodes, we selected a six-member team with a small node-to-node distance on average. The team thus selected comprised members with indices {19, 29, 3, 60, 37, 47}; they are also labeled A–F for ease of reference later. It can be verified from the graph above and the distance matrix shown in

Table 1. Distance matrix for the members chosen for team 1.

		Team Members
		19(A)	29(B)	3(C)	60(D)	37(E)	47(F)
Team Members	19(A)	0	3	2	3	3	3
	29(B)	3	0	1	2	4	2
	3(C)	2	1	0	1	3	1
	60(D)	3	2	1	0	4	2
	37(E)	3	4	3	4	0	4
	47(F)	3	2	1	2	4	0

below that the members are close to each other. It is important to note that this is not a collection of the closest group possible. This effect of inter-member distance is implemented in the simulation as follows. In the team optimization setting, an individual is initially selected with a probability proportional to their leadership value. In the subsequent GA generations, this probability was updated to include the distance from the individual selected in the first generation, as presented in Section 3. This process repeats and enforces the notion that individuals are more likely to hand over discussion to people they are more familiar with. When an individual is selected, the entire team performs one generation of evolutionary optimization using the individual’s objective function.

The intrinsic cognitive attributes of the team members are given by the following matrix in

Table 2. Intrinsic cognitive profiles of the team members selected.

Team Member	Graph Vertex	Leadership	Technical Ability	Creativity
A	19	0.1	0.8	0.1
B	29	0.75	0.1	0.1
C	3	0.1	0.75	0.75
D	60	0.6	0.6	0.6
E	37	0.4	0.4	0.4
F	47	0.1	0.1	0.1

. We notice that member 19 has high technical skills but lower leadership ability and creativity. Member 29 has high leadership but low technical and creativity skills. Member 3 has high technical and creativity skills but lower leadership ability. The rest of the members have equal but progressively lower skills in each attribute.

Figure 5 shows the result of the simulation of the above team. Since we used an evolutionary algorithm for optimization, we report both the best and average objective values in the solution population. The simulation completed 100 cycles, each comprising team members working individually and in a team. To reduce the noise from individual simulations, we averaged the values over 30 runs. We notice that there is a significant improvement in the beginning few cycles followed by a slower improvement, as expected. We also notice that despite the fact that the individuals with strong technical ability, A and C, did not have enough leadership skills, they were still able to improve the solutions, possibly because of their closeness to the leader (member B).

• Case 2: Team with smaller node-to-node distance

We repeated the same study with member locations given by vertices {28, 56, 50, 63, 52, 58} that are farther from each other in node-to-node distance but possess the same intrinsic cognitive profiles (same individuals) as before and are again labeled A-F for comparison purposes. The distance matrix is presented below in

Table 3. Distance matrix for the members chosen for team 2.

		Team Members
		28(A)	56(B)	50(C)	63(D)	52(E)	58(F)
Team Members	28(A)	0	7	2	10	3	10
	56(B)	7	0	7	7	8	7
	50(C)	2	7	0	10	3	10
	63(D)	10	7	10	0	11	6
	52(E)	3	8	3	11	0	11
	58(F)	10	7	10	6	11	0

.

We notice in Figure 6 that the progress in this case is slower and the final solution is also inferior. This can be verified by comparing the objective values at 20, 40, 60, 80 and 100 cycles. This hints towards the conjecture that since the graph distance between the technically adept members and the leader is larger, it is difficult for the technically adept members (A and C) in this case to influence the team, especially because their leadership levels are lower.

• Case 3: Team with smaller node-to-node distance and lower values of intrinsic attributes

In the next simulation, we selected the distances corresponding to team 1; however, the intrinsic profiles of the team members were all changed to 0.1 across all attributes. We ran the simulation again and found that the solution this time took much longer to converge and the objective function value achieved was much inferior (Figure 7).

• Case 4: Investigating changeovers in team discussions

Another aspect of team dynamics we investigated is how many times decision makers changed over, i.e., handed the conversation to each other in case 1 versus case 2. We see, as expected, that while the close team in case 1 saw changeovers 270 times, this number was only 234 for the team where team members were at greater distances on the graph. Given that the member cognitive profiles were the same in each case, the closeness within an organization helped better communication in case 1. Delving deeper into the amount of time each individual led the conversation, from Figure 8 we see that, in general, leaders led more. However, in case 1, we see that technically adept members led more and were able to contribute to the design more than they did in case 2. This explains the inferior solution found in case 2.

With these results, we see that the graph model for organizational relationships, coupled with the agent-based model that includes personality profiles in three dimensions (leadership, creativity and technical skill), is able to model expected team dynamics and predict trends. We believe that further exploration into the impact of different cognitive profiles as well as different organization structures should provide more insights.

5. Conclusions and Future Work

In this work, we presented a graph-theoretic approach to model an organization and combined it with an agent-based model to understand the effect of organizational structure and cognitive attributes on engineering project progress. The work used a simulation where individuals worked on an optimization problem (design) and then shared their ideas within a team setting. This was simulated using an evolutionary algorithm. The structure of the organization was modeled using a Barabasi–Albert graph, which is a scale-free network suitable for our analysis. Our results demonstrated the feasibility of using this approach to investigate team dynamics, where the organization structure plays a role. The simulation model showed the following:

• There was an initial steep improvement in the solutions followed by a slower improvement. This is typical, as one generally is able to find acceptable solutions faster, with better solutions coming later in the design process.
• Teams with closer distances between the individuals within the organizations converged to solutions faster than when individuals were far apart in terms of network distance. This was because, in the team setting, individuals were able to collaborate better. This hints that for skilled individuals with limited leadership skills, a cohesive team may be a better choice. It is important to note that closer distances may also prove to be detrimental when the skill level of the individuals is limited.
• As individual cognitive attributes were lowered, we saw a worsening of the solutions despite the individuals being close in terms of network distance. This dynamic confirms that cohesiveness cannot compensate for lower skill in workers.
• Looking at changeovers, we see that in the team where individuals are closer in distance, there was more changeover in who led the conversation. This was able to compensate for lower leadership skills in the technically adept members chosen.

While the above results are interesting, the following limitations are identified along with the steps envisioned to fully explore the idea in future work. It is clear that this type of an approach can generally be difficult to validate in a traditional sense. However, more sensitivity analyses can help identify strategies that organizations can use. For example, we aim to investigate various scenarios such as team composition, their cognitive attributes and different graph realizations, among others. We also will explore the effect of different graph invariants as well as other types of networks such as the Watts–Strogatz network. Another limitation is that we did not consider the organization culture or subject-matter expertise of individuals. While the underlying rationale is that a sub-team is usually selected based on subject-matter expertise, in the future we can use a more holistic model that uses detailed design problems involving many disciplines. In such cases, the ability of individuals to contribute to a particular sub-area of a particular design can come to the fore. Finally, on the basis of the results, we hope to make recommendations on organization structures and team formation in a general setting.