*Article* **Catastrophe Theory Applied to Neuropsychological Data: Nonlinear Effects of Depression on Financial Capacity in Amnestic Mild Cognitive Impairment and Dementia**

**Dimitrios Stamovlasis 1,\*, Vaitsa Giannouli <sup>2</sup> , Julie Vaiopoulou 3,4 and Magda Tsolaki 2,5,6**


**Abstract:** Financial incapacity is one of the cognitive deficits observed in amnestic mild cognitive impairment and dementia, while the combined interference of depression remains unexplored. The objective of this research is to investigate and propose a nonlinear model that explains empirical data better than ordinary linear ones and elucidates the role of depression. Four hundred eighteen (418) participants with a diagnosis of amnestic MCI with varying levels of depression were examined with the *Geriatric Depression Scale* (GDS-15), the *Functional Rating Scale for Symptoms of Dementia* (FRSSD), and the *Legal Capacity for Property Law Transactions Assessment Scale* (LCPLTAS). Cusp catastrophe analysis was applied to the data, which suggested that the nonlinear model was superior to the linear and logistic alternatives, demonstrating depression contributes to a bifurcation effect. Depressive symptomatology induces nonlinear effects, that is, beyond a threshold value sudden decline in financial capacity is observed. Implications for theory and practice are discussed.

**Keywords:** cusp catastrophe; complexity; nonlinear dynamics; financial capacity; amnestic mild cognitive impairment; depressive symptoms

## **1. Introduction**

## *1.1. The Psychocognitive Framework*

This section focuses on explaining the psychocognitive framework that hosts the present investigation, which has a predominately methodological orientation. Thus, from the existing extensive literature on dementia, amnestic mild cognitive impairment, and the factors involved in defining, measuring, and treating them, only the most relevant pieces will be cited, which will adequately familiarize the reader with the phenomenon under study.

An issue that has become crucial in modern societies, since it relates to legal implications, is the assessment of the financial capacity of older adults with psychocognitive problems [1–3]. The matter concerns a number of specialists, including not only clinical neuropsychologists and forensic psychiatrists, but also judges and lawyers, while there is an increasing theoretical interest in proposing models for describing and predicting empirical results. Although there is not a consensus among researchers about defining and measuring financial capacity [4–6], a predominant model (Marson's model) [7] proposes an effective way to deal with the multidimensionality of the latent variable in question,

**Citation:** Stamovlasis, D.; Giannouli, V.; Vaiopoulou, J.; Tsolaki, M. Catastrophe Theory Applied to Neuropsychological Data: Nonlinear Effects of Depression on Financial Capacity in Amnestic Mild Cognitive Impairment and Dementia. *Entropy* **2022**, *24*, 1089. https://doi.org/ 10.3390/e24081089

Academic Editors: Franco Orsucci and Wolfgang Tschacher

Received: 14 July 2022 Accepted: 5 August 2022 Published: 7 August 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

and it is acceptable for the legal systems in most countries. Conceptually, the model includes two components: the first encompasses the financial activity of a general domain of functioning, and the second takes into consideration specific financial abilities tasks. This general model has inspired potential endeavors for developing assessment instruments in different countries, given that the underlying process is culture-specific [8]. The interest in developing such tools aims at their implementation in clinical assessments, as direct measurements of relevant neuropsychological deficits. Analogous endeavors have been realized in other domains by examining different mental resources, such as memory skills or verbal fluency [9,10]. Given the abovementioned legal implications, the central idea continues to inspire a growing concern, specifically for the financial capacity, since it has been proven highly susceptible to Alzheimer's disease (AD) and related disorders [11]. In this direction, lately, a financial capacity test for the Greek population, namely the *Legal Capacity for Property Law Transactions Assessment Scale* (LCPLTAS), was developed and validated [12], with psychometric properties that allow the performance of both healthy older adults and those suffering from different types and stages of dementia to be investigated. It should be noted that individuals with mild cognitive impairment (MCI) are also distinguished via this test. Relevant diagnostic cognitive tests, such the Mini-Mental State Examination (MMSE), were used to predict financial capacity performance, enhancing the validly of the LCPLTAS. In addition, two more instruments, the *Geriatric Depression Scale* (GDS-15) and the *Functional Rating Scale for Symptoms of Dementia* (FRSSD) were used as predictor variables in order to associate LCPLTAS scores with the other scales.

#### *1.2. The Effects of Psychocognitive Resources on Financial Capacity*

Research on psychocognitive performance, based on empirical evidence, has established a number of relationships among latent factors related to some mental deficits.

Patients with amnestic mild cognitive impairment (aMCI) are found to be inferior performers in financial capacity tasks compared to healthy individuals [13,14], and the anticipated decline over time in MCI converters is significantly greater than that of the MCI non-converters or healthy control cases [13]. However, there are circumstances where additional factors can concomitantly affect financial capacity, such as comorbid depression. Research has shown that decline in financial capacity in Alzheimer's Disease, Parkinson's Disease, and vascular dementia is observed, specifically when depression is identified during neuropsychological assessment [15,16]. Moreover, studies have supported declining and impaired financial capacity in aMCI individuals [13,14], while some empirical evidence for financial capacity in aMCI with concurrent depressive symptomatology (aMCI-D) has been provided [17].

Regarding methodological issues, all relevant research has been promoted via traditional approaches with linear statistical modeling, the limitations of which are already well known [18]. The present endeavor, fostering the meta-theoretical framework of complexity theory and nonlinear dynamics, aimed to test the nonlinear hypothesis in psychocognitive performance by applying catastrophe theory and implementing financial capacity, GDS, and FRSSD.

Any neuropsychological process is characterized by an inherent complexity. The involved latent constructs, such as financial capacity, are also complex, involving a variety of mental functions, which are operationalized by specific ability tests (e.g., arithmetic, counting coins/currency, paying bills) and judgment decision-making skills [14,15]. All involved mental resources (such as working memory and logical thinking) interact with each other in time via a dynamical process where, in addition to the positively contributing components, counteracting variables and inhibitory factors operate as moderators leading to deteriorated outcomes. Based on the evidence, depression is a moderator factor, which, when combined with additional deficits, leads to a worse performance. It is reasonable to consider that depression not only in AD, but also in other neurocognitive disorders, is a moderator factor for financial capacity [19,20]. This is a hypothesis though that hasn't re-

ceived systematic investigation providing a coherent and interpretable model that describes the phenomenon. that describes the phenomenon. To this end, the present article proposes a novel approach in exploring medical data

hasn't received systematic investigation providing a coherent and interpretable model

*Entropy* **2022**, *24*, 1089 3 of 14

To this end, the present article proposes a novel approach in exploring medical data in this area by fostering complexity theory and nonlinear dynamics. It is a fundamental theoretical consideration that the latent constructs involved in neuropsychological processes are dynamically interacting and the emergent behavior is described by the notion of complex adaptive systems (CASs) [21]. In a CAS, the behavior is deemed as inherently nonlinear, and changes can often be discontinuous and unpredictable. Complexity science has already gained considerable attention in social sciences [22,23], behavioral sciences [24–28], and life and medical sciences [29–32]. The present endeavor employs catastrophe theory for modeling financial capacity as the state variable dependent measure, while the Geriatric Depression Scale (GDS-15) and the Functional Rating Scale for Symptoms of Dementia (FRSSD) are the predictor variables. Elements of catastrophe theory are presented in the following section. in this area by fostering complexity theory and nonlinear dynamics. It is a fundamental theoretical consideration that the latent constructs involved in neuropsychological processes are dynamically interacting and the emergent behavior is described by the notion of complex adaptive systems (CASs) [21]. In a CAS, the behavior is deemed as inherently nonlinear, and changes can often be discontinuous and unpredictable. Complexity science has already gained considerable attention in social sciences [22,23], behavioral sciences [24–28], and life and medical sciences [29–32]. The present endeavor employs catastrophe theory for modeling financial capacity as the state variable dependent measure, while the Geriatric Depression Scale (GDS-15) and the Functional Rating Scale for Symptoms of Dementia (FRSSD) are the predictor variables. Elements of catastrophe theory are presented in the following section.

#### *1.3. Catastrophe Theory 1.3. Catastrophe Theory*

Catastrophe theory as a mathematical theory was founded on the works of Thom [33] and Arnold [34] and concerns the classification of the equilibrium behavior of dynamical systems in the neighborhood of singularities. It proves that, at these critical points, the system can be locally modeled by seven elementary catastrophes, from which cusp catastrophe is the most known and applicable [35]. Catastrophe theory presupposes a dissipating or potential-minimizing system, and the cusp model is expressed by the first derivative of a potential function, *U*, with respect to the outcome, *y*, by Equation (1): Catastrophe theory as a mathematical theory was founded on the works of Thom [33] and Arnold [34] and concerns the classification of the equilibrium behavior of dynamical systems in the neighborhood of singularities. It proves that, at these critical points, the system can be locally modeled by seven elementary catastrophes, from which cusp catastrophe is the most known and applicable [35]. Catastrophe theory presupposes a dissipating or potential-minimizing system, and the cusp model is expressed by the first derivative of a potential function, *U*, with respect to the outcome, *y*, by Equation (1): ሺ, , ሻ

$$\frac{\partial \mathcal{U}(y, a, b)}{\partial y} = y^3 - by - ay \tag{1}$$

By setting *∂U*(*y*, *a*, *b*)/*∂y* = 0, the resulting equilibrium function is represented by the three-dimensional surface as a function of the two control parameters (α and *b*). three-dimensional surface as a function of the two control parameters (α and *b*). The development of stochastic catastrophe theory, which is based on the initial work

By setting ሺ, , ሻ/ ൌ 0, the resulting equilibrium function is represented by the

The development of stochastic catastrophe theory, which is based on the initial work of Cobb [36], allows for testing the relevant models with empirical data. Catastrophe theory is an area of complex dynamical systems and has shown high applicability in behavioral science. The notion of a potential-optimization process is compatible with a neuropsychological system, since it could be considered as pursuing the optimization of some function, e.g., related to adaptation or cognitive dissonance. The description of the cusp model is made via the response surface (Figure 1), where its fundamental features can be observed, such as bimodality, hysteresis, inaccessibility area, divergence, bifurcation, and sudden jumps [37]. The above phenomenology is interpreted via the underlying self-organization processes [38] and is theoretically connected to other areas of nonlinear sciences, such as Prigogine's non-equilibrium [39,40] and Haken's synergetics [41,42]. of Cobb [36], allows for testing the relevant models with empirical data. Catastrophe theory is an area of complex dynamical systems and has shown high applicability in behavioral science. The notion of a potential-optimization process is compatible with a neuropsychological system, since it could be considered as pursuing the optimization of some function, e.g., related to adaptation or cognitive dissonance. The description of the cusp model is made via the response surface (Figure 1), where its fundamental features can be observed, such as bimodality, hysteresis, inaccessibility area, divergence, bifurcation, and sudden jumps [37]. The above phenomenology is interpreted via the underlying self-organization processes [38] and is theoretically connected to other areas of nonlinear sciences, such as Prigogine's non-equilibrium [39,40] and Haken's synergetics [41,42].

**Figure 1.** Cusp catastrophe surface. **Figure 1.** Cusp catastrophe surface.

The interpretation of the cusp model is facilitated by the three-dimensional response surface (see Figure 1), which demonstrates the geometry of behavior. At the back region

of the surface, where the bifurcation, *b*, has low values, the surface is smooth and a linear relationship between the state variable (dependent measure) and the asymmetry*, a*, holds. In the forward-facing part, the surface folds and the two regions, i.e., the upper and the lower parts, appear, representing the two behavioral modes that in the language of CDS are called *attractors*. At this region, the probability density function of the empirical data becomes bimodal, whereas, in the area between the two modes, the behavior is unlikely to occur, and it is called the inaccessibility area. Thus, in this region, changes can only occur as jumps or transitions between the two behavioral attractors. Mathematically, these changes are called *discontinuities* and the splitting of the system into two states or different modes of behaviors consists of a bifurcation [43]. Looking from the front of the surface, a sigma-like feature, the *hysteresis effect*, can be observed. These are dynamic effects occurring when the bifurcation variable, *b*, goes beyond a critical value. It is pertinent to emphasize that bifurcation characterizes only nonlinear systems and is considered as a fingerprint of complexity [38].

#### **2. Materials and Methods**

#### *2.1. Rationale and Research Hypotheses*

In this research and for the relevant diagnoses, well-known neuropsychological assessment tools, such as the GDS, FRSSD, MMSE, and LCPLTAS, were implemented. The instruments operationalize specific psychological resources and are used concomitantly to consolidate conclusions and to help make decisions. The validation of the LCPLTAS [12] was supported with its functional relationships with the rest of the instruments, while the statistical methodology was based on linear modeling. Considering the epistemological and methodological limitations of the traditional linear approaches [18], this research was initiated to examine the applicability of catastrophe theory in this area and to provide insights about the theoretical and practical implications. This neuropsychological endeavor is apparently inductive in nature, since it is a new application in the field, encompassing a supplementary analysis of available data. However, it is also theory-driven, because it is based on the theory of complex adaptive systems (CASs), which is used here to reexamine the outcomes of dynamical processes, such as the neuropsychological processes taking place in assessment procedures. The cognitive factors involved in financial problem solving do not act as parts of a mechanical system, where the outcome can be expressed as a linear function of the contributing mechanisms [18,44,45]. As parts of a CAS system, these components act with no predetermined scenario, but execute their tasks via an iterative dynamical process. The potential nonlinearity can lead to changes encompassing sudden shifts, discontinuities, or transitions, which can be captured by catastrophe theory models. In the present research framework, among the psychological resources involved, *depression* is known as a moderator factor of financial capacity, competing against the positively acting resources. As an inhibitory agent, depression is a potential factor for inducing nonlinear effects.

To this end, the research hypotheses posited in this endeavor concern the potential role of depression in financial capacity, along with testing the applicability of catastrophe theory in neuropsychology, and are stated as follows:


#### *2.2. Participants and Measures*

The participants were 418 Greek adults (68.2% women), whose age ranged from 45 to 98 years (*mean* = 72.55, *SD* = 8.08, *median* = 72.0). The mean years of education was 8.61 years (*SD* = 4.41, *median* = 6.0). A total of 34.4% were healthy control individuals, while the rest were diagnosed with varying degrees of AD and cognitive impairment. This sample composition ensures large variances in the measured construct and facilitates

the variable-centered analyses. The neuropsychological assessments were carried out at the Memory Clinic of Papanikolaou General Hospital and elderly daycare centers during 2012–2016. Written informed consent from each participant was obtained and the study was approved by the Ethics Committee of the Aristotle University of Thessaloniki (protocol code 2.27/3/2013) [12], while the research was performed according to the Declaration of Helsinki.

Financial capacity was assessed with the *Legal Capacity for Property Law Transactions Assessment Scale* (LCPLTAS) short form [12]. The LCPLTAS consists of seven main domains: (1) basic monetary skills, (2) cash transactions, (3) bank statement management, (4) bill payment, (5) financial conceptual knowledge, (6) financial decision making, and (7) knowledge of personal assets [12]. The depressive symptomatology was measured by the *Geriatric Depression Scale* (GDS-15) [46], and the functionality evaluation was provided by the *Functional Rating Scale for Symptoms of Dementia* (FRSSD), which measures activities of daily living (ADLs) [47]. The above three instruments, along with the MMSE scale, are commonly used by psychiatrists, neuropsychologists, and neurologists in Greece [12,48–51]. The reported scores were available for all participants in this sample, since they were important part of assessment protocols in medical settings. Note that in the data set there were no missing values.

#### *2.3. Method*

Cusp analysis was carried out via a modeling procedure based on the probability function, *pdf*, of the dependent measure (Equation (2)):

$$pdf(y) = \xi \exp\left[-\frac{1}{4}y^4 + \frac{1}{2}by^2 + ay\right] \tag{2}$$

As the optimization method, the maximum likelihood [52] was used, while the *pdf* was obtained from empirical data. The analysis was performed in R via the cusp package [53]. The cuspfit algorithm utilizes numerical procedures for parameter estimates by minimizing a negative loglikelihood function, on which the model-fit evaluation is based, along with the indices: AIC (Akaike's information criteria), corrected AIC, and BIC (Bayesian information criteria) and the statistically significant coefficients of the model. Moreover, a comparison of the cusp with the linear and logistic alternative model is provided [53]. The literature offers other modeling procedures as well, such as the GEMCAT methodology [54] and a method implementing Equation (2) and least squares as the optimization method [55]. The details of these methods could be found in a lucid review elsewhere [56].

In the cusp analysis, the financial capacity was the dependent measure (LCPLTAS), known as the state variable, while the depressive symptomatology measured by the *Geriatric Depression Scale* (GDS-15) and the *Functional Rating Scale for Symptoms of Dementia* (FRSSD) were the two control variables. Results from a power analysis [57] (power levels of 80%, a medium effect size, two-tailed test with alpha = 0.05, required sample size of 75) showed that the available sample (*N* = 418) is adequate for testing the multivariate effects under study.

Initially, the model was conceived with the FRSSD as the asymmetry factor and the *Geriatric Depression Scale* (GDS-15) as the bifurcation. The conceptual and mathematical model, however, considers that asymmetry and bifurcation factors that represent antagonistic processes can be operationalized by a combination of the proposed controls, and, consequently, linear functions of the FRSSD and GDS scales were tested as contributing factors to both the asymmetry and bifurcation. The alternative cusp catastrophe models utilize rotated axes [58] and are analogous to the conflict cusp model that has been proposed for Piaget's conservation task [59]. This cusp model implements (FRSSD − GDS) and (FRSSD + GDS) as asymmetry and bifurcation, respectively.

#### **3. Results**

Table 1 presents the descriptive statistics (means, standard deviations, minimum and maximum values) for the variables under study. For financial capacity, the LCPLTAS and its short version sLCPLTAS were used.



Table 2 depicts the correlation matrix for the above variables. Both tables include age and measures of the MMSE, which, however, were not used in the present analysis. Note that both the GDS (*r =* −0.220, *p* < 0.001) and FRSSD (*r =* −0.792, *p* < 0.001) are negatively correlated with financial capacity.

**Table 2.** Pearson's correlations.


\* *p* < 0.05, \*\* *p* < 0.01, \*\*\* *p* < 0.001.

Subsequently, cusp catastrophe analysis was carried out, testing a model with the FRSSD and GDS as control variables (*Cusp 1*) and a model with the linear combination of them (*Cusp 2*). Tables 3 and 4 show the slopes, standard errors, Z-tests, and model fit statistics for the cusp and the alternative models.

**Table 3.** The cusp model estimated by maximum likelihood method: slopes, standard errors, Z-tests, and model fit statistics for cusp and the alternative models. Financial capacity as a function of FRSSD (asymmetry) and Geriatric Depression Scale (bifurcation variable).



**Table 4.** The cusp model estimated by maximum likelihood method: slopes, standard errors, Z-tests, and model fit statistics for cusp and the alternative models. Financial capacity as a function of (FRSSD − GDS) as asymmetry and (FRSSD + GDS) as bifurcation variable.

#### *3.1. Cusp 1*

In Cusp 1 (Table 3), the FRSSD acts as the asymmetry factor (*b* = −1.4557, *p <* 0.001) and the GDS acts as the bifurcation factor (*b =* −0.3490, *p <* 0.001). The chi-square test of the linear vs. cusp model gives *χ* <sup>2</sup> = 247.0, *df* = 2, and *p* < 0.001, and the model fit statistics in terms of AIC, AICc, and BIC favor the cusp catastrophe model. The values for the cusp model (AIC = 538.190, AICc = 538.392, and BIC = 562.404) are minimum compared to the linear (AIC = 781.203, AICc = 781.300, and BIC = 797.345) and logistic (AIC = 744.210, AICc = 744.351, and BIC = 764.388) models, respectively. The values of pseudo *R* <sup>2</sup> are close, but this index is not reliable, and it is not interpreted as the usual percentage of variance explained.

Figure 2 is a visual display of the lower part of the cusp surface, where the shaded region is the bifurcation area. If at least 10% of the observations fall within this area, it is considered as evidence supporting the cusp model [60]. The size of the dots in Figure 2 is a function of the observed bivariate density of the bifurcation factor's values at that point's location, and the color is evocative of their position relative to the distance between the two parts of the surface (two attractors), i.e., the observations that are darker in color indicate that they are on or closer to the upper attractor and the observations that are lighter in color are on or closer to the lower attractor. Finally, Figure 3, which depicts the three-dimensional cusp surface as a function of the two control variables, provides an additional visual support for the cusp model, showing that the observations are located at the upper and the lower surface, but not within the area of inaccessibility.

#### *3.2. Cusp 2*

This cusp catastrophe model utilizes rotated axes [58], analogous to the conflict cusp model [59] using the axes *m* and *n* depicted in Figure 1. It implements a combination of the initially proposed controls, specifically their difference (FRSSD − GDS) and their sum (FRSSD + GDS), as asymmetry and bifurcation factors, respectively.

In Cusp 2 (Table 4), (FRSSD − GDS) acts as the asymmetry factor (*b* = 1.3450, *p* < 0.001) and (FRSSD + GDS) as the bifurcation factor (*b =* 1.2804, *p* < 0.001). The chi-square test of the linear vs. cusp model gives *χ* <sup>2</sup> = 147.6, *df* = 2, and *p <* 0.001, and the model fit statistics in terms of AIC, AICc, and BIC favor the cusp catastrophe model. The values for the cusp model (AIC = 271.213, AICc = 271.801, and BIC = 289.277) are minimum compared to the linear (AIC = 414.767, AICc = 415.043, and BIC = 426.809) and logistic (AIC = 395.039, AICc = 395.456, and BIC = 410.093) alternatives. The values of pseudo *R* <sup>2</sup> are 0.63, 0.39, and

*Entropy* **2022**, *24*, 1089 7 of 14

(asymmetry) and Geriatric Depression Scale (bifurcation variable).

a[FRSSD] Functional Rating Scale for

**Cusp 1** 

**Table 3.** The cusp model estimated by maximum likelihood method: slopes, standard errors, Z-tests, and model fit statistics for cusp and the alternative models. Financial capacity as a function of FRSSD

**Model b seb Z-Value** 

Symptoms of Dementia −1.4557 0.1468 −9.91 \*\*\*

a(Intercept) 1.0628 0.1248 8.52 \*\*\*

b(Intercept) −1.5417 0.2165 −7.12 \*\* b[GDS] Depression Scale −0.3493 0.0912 −3.83 \*\*\* w(Intercept) 0.8830 0.0355 24.87 \*\*\* w(FC) Financial Capacity 1.2059 0.02921 41.28 \*\*\* Models' fit statistics (chi-square test of linear vs. cusp model: *χ*<sup>2</sup> = 247.0, *df =* 2, *p* < 0.001) **Model Pseudo***-R***<sup>2</sup> Npar AIC AICc BIC**  Linear model 0.61 4 781.203 781.300 797.345 Logistic model 0.61 5 744.210 744.351 764.388 Cusp model 0.63 6 538.190 538.392 562.403 Note: \*\*\* *p* < 0.001, \*\* *p*< 0.01, \* *p*< 0.05, **†** *p* < 0.05 (one-tailed); ns = non-significant.

0.47 for the cusp, linear, and logistic models, respectively; however, they are not counted in the assessment criteria. dimensional cusp surface as a function of the two control variables, provides an additional visual support for the cusp model, showing that the observations are located at the upper and the lower surface, but not within the area of inaccessibility.

Figure 2 is a visual display of the lower part of the cusp surface, where the shaded region is the bifurcation area. If at least 10% of the observations fall within this area, it is considered as evidence supporting the cusp model [60]. The size of the dots in Figure 2 is a function of the observed bivariate density of the bifurcation factor's values at that point's location, and the color is evocative of their position relative to the distance between the two parts of the surface (two attractors), i.e., the observations that are darker in color indicate that they are on or closer to the upper attractor and the observations that are lighter in color are on or closer to the lower attractor. Finally, Figure 3, which depicts the three-

**Figure 3.** Three-dimensional cusp response surface financial capacity using maximum likelihood estimation. FRSSD is the asymmetry factor and depressive symptomatology (GDS-15) is the bifurcation factor. The gray dots represent observed values from empirical data. **Figure 3.** Three-dimensional cusp response surface financial capacity using maximum likelihood estimation. FRSSD is the asymmetry factor and depressive symptomatology (GDS-15) is the bifurcation factor. The gray dots represent observed values from empirical data.

*3.2. Cusp 2*  This cusp catastrophe model utilizes rotated axes [58], analogous to the conflict cusp model [59] using the axes *m* and *n* depicted in Figure 1. It implements a combination of the initially proposed controls, specifically their difference (FRSSD − GDS) and their sum (FRSSD + GDS), as asymmetry and bifurcation factors, respectively. In Cusp 2 (Table 4), (FRSSD − GDS) acts as the asymmetry factor (*b* = 1.3450, *p* < 0.001) Figure 4, as the visual display of the lower part of the cusp surface, shows that most points are located within the shaded region, the bifurcation area, and in both attractors, the upper and the lower. Figure 5, which depicts the three-dimensional cusp surface as a function of the two control variables, clearly reveals the bifurcation structure with the two diverging slops that are joined at the cusp point and are spreading in each attractor area, while no observations are located in the area of inaccessibility.

#### and (FRSSD + GDS) as the bifurcation factor (*b =* 1.2804, *p* < 0.001). The chi-square test of the linear vs. cusp model gives *χ*2 = 147.6, *df* = 2, and *p <* 0.001, and the model fit statistics *3.3. Model Interpretation*

the assessment criteria.

**Cusp 2** 

in terms of AIC, AICc, and BIC favor the cusp catastrophe model. The values for the cusp model (AIC = 271.213, AICc = 271.801, and BIC = 289.277) are minimum compared to the linear (AIC = 414.767, AICc = 415.043, and BIC = 426.809) and logistic (AIC = 395.039, AICc For *Cusp 1*, which implements the FRSSD and GDS as control variables (Figure 3), the interpretation of the model suggests that, at low values of depression, changes in the state variable (the financial capacity) occur in a smooth and linear manner. In this region,

= 395.456, and BIC = 410.093) alternatives. The values of pseudo *R*2 are 0.63, 0.39, and 0.47 for the cusp, linear, and logistic models, respectively; however, they are not counted in

**Table 4.** The cusp model estimated by maximum likelihood method: slopes, standard errors, Z-tests, and model fit statistics for cusp and the alternative models. Financial capacity as a function of

**Model b seb Z-Value** 

Symptoms of Dementia 1.3450 0.2181 6.19 \*\*\*

a(Intercept) −0.1606 0.1096 −1.46 ns

b(Intercept) 0.98163 0.3388 2.90 \*\* b[GDS + FRSSD] Depression Scale 1.2804 0.1886 6.79 \*\*\* w(Intercept) 0.02605 0.0528 0.50 ns w(FC) Financial Capacity 1.02722 0.0472 21.75 \*\*\* Models' fit statistics (chi-square test of linear vs. cusp model: *χ*<sup>2</sup> = 147.6, *df* = 2, *p* < 0.001) **Model Pseudo***-R***<sup>2</sup> Npar AIC AICc BIC**  Linear model 0.39 4 414.767 415.043 426.809 Logistic model 0.47 5 395.039 395.456 410.093 Cusp model 0.63 6 271.213 271.801 289.277 Note: \*\*\* *p* < 0.001, \*\* *p*< 0.01, \* *p*< 0.05, **†** *p*< 0.05 (one-tailed); ns = non-significant.

a[FRSSD − GDS] Functional Rating Scale for

(FRSSD − GDS) as asymmetry and (FRSSD + GDS) as bifurcation variable.

the linear relationship between the state variable and the asymmetry factor, the FRSSD, holds. At higher values of depression, that is, as approaching the forward-facing part of the surface, where surface folds and two behavioral attractors appear, the changes occur merely as transitions between the two attractors. In this region, people with the same control-factor values can be found at the lower and/or at higher attractor regions. This introduces unpredictability in the system and implies that changes in behavior occur as sudden jumps between two modes. Figure 4, as the visual display of the lower part of the cusp surface, shows that most points are located within the shaded region, the bifurcation area, and in both attractors, the upper and the lower. Figure 5, which depicts the three-dimensional cusp surface as a function of the two control variables, clearly reveals the bifurcation structure with the two diverging slops that are joined at the cusp point and are spreading in each attractor area, while no observations are located in the area of inaccessibility. Figure 4, as the visual display of the lower part of the cusp surface, shows that most points are located within the shaded region, the bifurcation area, and in both attractors, the upper and the lower. Figure 5, which depicts the three-dimensional cusp surface as a function of the two control variables, clearly reveals the bifurcation structure with the two diverging slops that are joined at the cusp point and are spreading in each attractor area, while no observations are located in the area of inaccessibility.

*Entropy* **2022**, *24*, 1089 9 of 14

*Entropy* **2022**, *24*, 1089 9 of 14

**Figure 4.** A visual display of the lower part of the cusp response surface of financial capacity using maximum likelihood estimation. (FRSSD − GDS) and (FRSSD + GDS) act as asymmetry and bifurcation factors, respectively. **Figure 4.** A visual display of the lower part of the cusp response surface of financial capacity using maximum likelihood estimation. (FRSSD − GDS) and (FRSSD + GDS) act as asymmetry and bifurcation factors, respectively. **Figure 4.** A visual display of the lower part of the cusp response surface of financial capacity using maximum likelihood estimation. (FRSSD − GDS) and (FRSSD + GDS) act as asymmetry and bifurcation factors, respectively.

**Figure 5.** Three-dimensional cusp response surface financial capacity using maximum likelihood estimation. (FRSSD − GDS) and (FRSSD + GDS) act as asymmetry and bifurcation factors, respectively. The gray dots represent observed values from empirical data. **Figure 5.** Three-dimensional cusp response surface financial capacity using maximum likelihood estimation. (FRSSD − GDS) and (FRSSD + GDS) act as asymmetry and bifurcation factors, respectively. The gray dots represent observed values from empirical data. **Figure 5.** Three-dimensional cusp response surface financial capacity using maximum likelihood estimation. (FRSSD − GDS) and (FRSSD + GDS) act as asymmetry and bifurcation factors, respectively. The gray dots represent observed values from empirical data.

*3.3. Model Interpretation*  For *Cusp 1*, which implements the FRSSD and GDS as control variables (Figure 3), the interpretation of the model suggests that, at low values of depression, changes in the state variable (the financial capacity) occur in a smooth and linear manner. In this region, the linear relationship between the state variable and the asymmetry factor, the FRSSD, holds. At higher values of depression, that is, as approaching the forward-facing part of the surface, where surface folds and two behavioral attractors appear, the changes occur *3.3. Model Interpretation*  For *Cusp 1*, which implements the FRSSD and GDS as control variables (Figure 3), the interpretation of the model suggests that, at low values of depression, changes in the state variable (the financial capacity) occur in a smooth and linear manner. In this region, the linear relationship between the state variable and the asymmetry factor, the FRSSD, holds. At higher values of depression, that is, as approaching the forward-facing part of *Cusp 2* has an analogous interpretation. Note that both the FRSSD and GDS are negatively associated with the financial capacity. When their difference is large, the negative effect is smaller, and as it increases the outcome increases as well. These changes are expected to be smooth and linear compared to the effect of their sum (FRSSD + GDS). When the net moderating effect of the combined high FRSSD and high GDS becomes unexpectedly increased, in that scale, a threshold value is likely to exist, beyond which abrupt changes

the surface, where surface folds and two behavioral attractors appear, the changes occur

occur, inducing bifurcation effects. The present analysis supports the above-described roles by providing empirical evidence and establishes catastrophe phenomena in this type of neuropsychological data. It is imperative to repeat that bifurcations and hysteresis effects are complex phenomena, due to the dynamics of the systems and due to self-organization mechanisms.

#### **4. Discussion**

The present catastrophe theory model, to our knowledge, is the first application reported in the domain of neuropsychology, and it has twofold implications. The first is epistemological and regards the underlying theory, while the second concerns measurement issues, classifications, and decision-making. The identification of bifurcation effects challenges the epistemological assumptions that adhere to the linear and mechanistic views of a neuropsychology system. Given that bifurcations only characterize CASs [38], their detection indicates that the underlying system is ontologically a complex adaptive system, and it should be investigated as such, i.e., the linear modeling is inadequate and epistemologically incompatible to describe and interpret the system's behavior [18]. The cusp catastrophe model designated discontinuous changes in a neurocognitive system under gradual changes in the two independent variables, the control factors, namely the asymmetry and the bifurcation. The discontinuous changes occur as transitions between two attractors, which for a neurocognitive system might represent qualitatively distinct modes of behavior, such as a high or low/suboptimal level of performance.

Note that the present cusp analysis was applied to cross-sectional data, but the interpretation of the model also needs to be extended for the dynamical path of the single case. The individual's mind involved in a cognitive task, ontologically acting as a CAS, follows a trajectory driven by self-organization mechanisms and the outcome emerges via a dynamical iterative process [21,61,62]. Bifurcations potently occur in those systems and the interpretation of the present model suggests that, in the course of such a dynamical process, even small random fluctuations in the parameters can induce sudden, unexpected transitions from a state of high performance to a state of failure.

It is pertinent to mention here that bifurcations can be observed and captured analogously when a single case (*N* = 1) is analyzed. Catastrophe phenomena might be relevant and worth examining when dynamical processes are investigated via time series, where nonlinear methods and tools should be employed. Complexity theory offers a theoretical framework and a rich array of methodological tools to support research designs and data analysis. Even though the present investigation used a large sample and cross-sectional data to infer nonlinearity, the effective methodological approach to study CASs is time series analysis [63,64]. This framework has been fruitfully applied in many process approaches [65–68], where bifurcation phenomena are theoretically anticipated and are worth examining. In those cases, catastrophes of this kind represent changes: cognitive, attitudinal, shifts to coherence, or therapeutic changes. Relevant also is the notion of *ergodicity* in a time series of the analysis [69]. Sudden shifts, transitions, and discontinuities denote a *non-ergodic process,* and the present cusp catastrophe structure supports this idea in neuropsychological data.

What has been learned for the AD and aMCI research, is that depression is not merely a linear moderator of mental operators, but it also reacts with other neurocognitive resources and prompts nonlinear effects. To further stimulate a discussion that will bridge the mathematical/methodological domain with the theoretical premises of neuropsychology, it would be pertinent to think and reflect on the role of other coexisting conditions (e.g., diabetes, heart disease) or other factors of biological and/or psychological origin. The common methodological thought suggests that, in addition to the present choices, additional variables could be included in the cusp model specification and tested with empirical data. The effect of additional candidates is an open issue for further research. However, there are some more interesting aspects to reflect on.

A pertinent epistemological remark is that a bifurcation effect should be perceived as a process, where the relevant variables being tested, as factors contributing to the underlying mechanism. Under the CAS perspective, this self-organization mechanism concerns the evolution of an interaction system that possibly includes both biological and psychological factors within a *mutual causality* connective state. The representation of such a system is explicitly the ontology of networks, which is in line with complexity theory assumptions. The proper methodology for this is the *network analysis*, where the latest advances offer a better way to approach and understand those systems [70–72] compared to the traditional methods. In addition, the network ontology explains the possibility of linear and nonlinear changes, and thus catastrophe theory is on the scene.

Another interesting remark that is explicated by catastrophe theory is that in the vicinity singularities, e.g., the bifurcations and discontinuities, the behavior could be described merely by a small number of variables. In the present context, depression is one of them. The levels of depression (GDS), even though might be affected by the dynamic interplay of other factors (biological, medical, and/or psychological), contribute to operationalization of the ensuing bifurcation mechanism, in conjunction with the functional symptoms of dementia (FRSSD).

The existing cusp structure in the data and the operating critical points beyond which nonlinear effects occur, directly concern the measurement issues and the relevant theory. The determination of such thresholds is an open issue and of paramount importance in the actual utilization of the *Legal Capacity for Property Law Transactions Assessment Scale* (LCPLTAS) and financial decision-making. In addition, given that the actual bifurcation process is induced by a composite variable, the determination of the critical point is a challenge. The issue is important because it concerns the measurement processes, diagnosis, and further prevention and treatment.

There are of course limitations in this study, originating from its exploratory character, and since it is the first report with neuropsychological data, the findings should be replicated and extended with other data sets. Cusp analysis could also be tried in other neurocognitive assessments, such as for Parkinson's disease, and in other neurocognitive assessment tools, such as the MMSE or HoNOS and GAF, to extend the model to different socio-medical inquiries. The present report sets a framework for the application of catastrophe theory with neurocognitive resources in AD research and opens new avenues for investigations.

Last, but not least, the message that the present findings convey is mainly epistemological and concerns the adoption of the meta-theoretical framework of CASs, the paradigm shift that is gaining ground in interdisciplinary research.

**Author Contributions:** Conceptualization, D.S.; methodology, D.S. and J.V.; software, D.S. and J.V.; validation, D.S., J.V., M.T. and V.G.; formal analysis, D.S.; investigation, D.S. and J.V.; resources, D.S., V.G., J.V. and M.T.; data curation, V.G. and M.T.; writing—original draft preparation, D.S. and J.V.; writing—review and editing, D.S., V.G., J.V. and M.T.; visualization, D.S.; supervision, D.S. and M.T.; project administration, V.G. and M.T. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was approved by the Ethics Committee of the Aristotle University of Thessaloniki (protocol code 2.27/3/2013), while the research was performed according to the declaration of Helsinki.

**Informed Consent Statement:** Written informed consent from each participant was obtained.

**Data Availability Statement:** The data presented in this study are available upon request from the corresponding author.

**Conflicts of Interest:** The authors declare no conflict of interest.

## **References**


## *Article* **Dynamics of Remote Communication: Movement Coordination in Video-Mediated and Face-to-Face Conversations**

**Julian Zubek 1,\*, Ewa Nagórska <sup>1</sup> , Joanna Komorowska-Mach 1,2, Katarzyna Skowro ´nska <sup>1</sup> , Konrad Zieli ´nski <sup>1</sup> and Joanna R ˛aczaszek-Leonardi <sup>1</sup>**


**Abstract:** The present pandemic forced our daily interactions to move into the virtual world. People had to adapt to new communication media that afford different ways of interaction. Remote communication decreases the availability and salience of some cues but also may enable and highlight others. Importantly, basic movement dynamics, which are crucial for any interaction as they are responsible for the informational and affective coupling, are affected. It is therefore essential to discover exactly how these dynamics change. In this exploratory study of six interacting dyads we use traditional variability measures and cross recurrence quantification analysis to compare the movement coordination dynamics in quasi-natural dialogues in four situations: (1) remote video-mediated conversations with a self-view mirror image present, (2) remote video-mediated conversations without a self-view, (3) face-to-face conversations with a self-view, and (4) face-to-face conversations without a self-view. We discovered that in remote interactions movements pertaining to communicative gestures were exaggerated, while the stability of interpersonal coordination was greatly decreased. The presence of the self-view image made the gestures less exaggerated, but did not affect the coordination. The dynamical analyses are helpful in understanding the interaction processes and may be useful in explaining phenomena connected with video-mediated communication, such as "Zoom fatigue".

**Keywords:** remote communication; movement coordination; recurrence quantification analysis

#### **1. Introduction**

When two people engage in a dialogue, they do much more than just exchanging strings of words. According to Fusaroli et al. [1], dialogue participants coordinate on multiple levels, establishing a functional organization fit to a particular situation. Essentially, they form a coupled system within which meanings are co-created, and interaction dynamics are essential to this process [2]. The ability to coordinate movements during interaction is already present in infancy [3] and constitutes the most basic form of bonding with others [4]. Movement coordination allows the establishment of informational and affective coupling [5,6]. This has consequences for various processes of social cognition. As demonstrated by numerous empirical studies, spontaneous movement coordination of people engaged in natural conversations can predict rapport [7], affiliation [8], empathic accuracy [9], joint-action task performance [10,11] or psychotherapy outcomes [12]. The connections between movement coordination and social interaction may go in both directions: particular patterns of movement coordination may be constitutive factors for the interaction or they can be merely indicators of a successful interaction taking place [13]. In any case, by analyzing interpersonal movement coordination, we can infer much regarding the quality of an interaction.

**Citation:** Zubek, J.; Nagórska, E.; Komorowska-Mach, J.; Skowro ´nska, K.; Zieli ´nski, K.; R ˛aczaszek-Leonardi, J. Dynamics of Remote Communication: Movement Coordination in Video-Mediated and Face-to-Face Conversations. *Entropy* **2022**, *24*, 559. https://doi.org/10.3390/e24040559

Academic Editors: Franco Orsucci and Wolfgang Tschacher

Received: 1 February 2022 Accepted: 11 April 2022 Published: 15 April 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

In the present pandemic, many social interactions have moved online. Remote video calls are used as an alternative to face-to-face conversations, both in professional and casual contexts. Video-mediated interactions indeed allow the use of visual cues (gestures, face expressions, body posture) and the establishment of some form of functional movement coordination between participants, which is not possible in audio-only interactions. Studies comparing video-mediated communication to audio-only communication report benefits such as increased effectiveness of group problem-solving, shorter discussion time, and increased emotional bonding [14,15]. However, the experience with video-mediated interactions is not always smooth. In some cases, people were more satisfied with audio-only interactions than with video-mediated interactions, and audio-only interactions seemed more efficient [16–18]. Recently, there have been discussions regarding "Zoom fatigue", a form of exhaustion experienced by participants of video conference meetings [19–22]. The possible causes of this phenomenon include both a lack of proper social cues (i.e., eye contact, body language), leading to increased cognitive effort, and information overload (i.e., self-image visible, multiple faces visible on the screen), leading to additional stress [23].

A deeper understanding of video-mediated communication can be gained by studying the process of interaction itself [16,24]. Different media provide characteristic constraints and afford specific communicative actions with different degrees of synchronicity. This shapes the ongoing interaction process and, consequently, interaction outcomes. In the case of video-mediated interactions, disrupted social cues and visual information overload may affect the capabilities of nonverbal communication, leading to different coordination dynamics than in face-to-face interactions. We suspect that altered coordination capabilities in online communication may influence informational and affective couplings between participants, may be a possible cause of decreased satisfaction with an interaction, as stated in the recent literature, and may also cause decreased effectiveness of communication as compared to face-to-face interactions.

#### *1.1. Dynamics of Video-Mediated Interactions*

Patterns of social interaction dynamics are emergent properties shaped by multiple interrelated factors [13,25]. In the case of natural conversation, any change in a participant's impression of their interlocutor influences the way the participant responds, which in turn influences the interlocutor. This ongoing feedback loop, constituting patterns of interaction dynamics, may work differently in mediated interactions. A communication medium—such as a video-conferencing setup—is one of the factors that may significantly constrain interaction dynamics. In the language of dynamical systems, if a medium offers fewer possibilities for interaction than the number of available options in unmediated communication, the number of degrees of freedom of the system is reduced. On the one hand, when the preferred interaction means are taken away, it may disrupt the interaction. On the other hand, when the redundant modes of communication are reduced, it may present a case of functional reduction in degrees of freedom facilitating the interaction. Either way, the patterns of interaction dynamics are changed.

Constraints imposed by the communication medium can be traced through the analysis of interactions between a person and the medium. In this case, the ecological psychology notion of affordance is helpful [26]. Affordances are opportunities for action and perception offered by the environment to an active subject. They are not simply objective properties of the external objects (shape, size), but meaningful relations in which complementarity between the subject and its environment manifests (graspability, possibility to sit upon). In the social realm, affordances are created and used dynamically by each interactant "on the fly" [27]. Introducing a video-based communication medium creates new possibilities for actions and forms of interaction, while precluding others. The landscape of affordances available for the individuals and the dyad changes, which changes their behavior and cocreated meanings [28].

Affordances of video-mediated interaction are significantly changed by the presence of video latency—a mean delay between the moment the movement is made, and the moment it is visible on another user's screen. Another aspect is jitter—variability of the delay, caused by the different length of time each data packet takes to arrive. If the jitter is large, movement in the video is not smooth. Video latency during a high quality video call may be 150 ms with 40 ms jitter [29], but these values may vary depending on the network traffic, connection bandwidth and hardware configuration/quality. Since latency works in both directions, the effective time between a communicative action and the perceived response may double. Additionally, glitches in the form of video freezing or distorted images are common during video calls. These factors modify the affordances of interaction participants, for instance, by limiting the possibility of reacting quickly to each other thus constraining their patterns of coordination. It is known that people are able to perceive delays of 200 ms [30,31], which suggests that even relatively small video latency may affect coordination in a video-mediated interaction. Boland et al. [32] studied turn-taking during face-to-face and Zoom conversations and discovered that delays introduced by the latter significantly disrupted the rhythm of conversation, increasing the average turn transition time from 135 ms to 487 ms. Such altered coordination patterns may have further consequences for communication. The length of the gap between turns may provide information on the valence of the upcoming response, with preferred responses coming quicker and taking simpler forms [33]. A gap as short as 300 ms may be sufficient to project that a straightforward acceptance is less probable [34]. Because of the prolonged gaps due to the video latency, speakers may erroneously expect more dispreferred reactions than in face to face communication. Additionally, according to the studies on telephone communication, the longer the delays are, the more interlocutors are perceived as less attentive, less friendly, less extraverted and less conscientious [35].

Another aspect that differentiates video-mediated and face-to-face interactions is the way the image of the conversation partner is presented to the interaction participant. In natural face-to-face conversations, people typically face each other, moving their glances between the face, body and hands of the interaction partner [36], which provides them with specific means to fluently structure the interaction (see, e.g., R ˛aczaszek-Leonardi and Nomikou [37]). In contrast, in a typical video-mediated interaction (for instance, using a laptop computer with a built-in camera), the captured field of vision is much narrower, limiting visual cues concerning whole body movement and hand gestures. This may severely limit nonverbal communication, as hand gestures play an important role in supplementing speech with additional content, disambiguating expressions or organizing turn-taking [38–40]. It is possible to compensate for this through the use of other modalities such as head gestures, which are captured well in video-conferencing settings. Head gestures are considered to be important for coordinating interaction, providing confirmatory feedback for the speaker [41] and signaling turn claims [42]. In many cultures head nodding and head shaking are associated with affirmative and negative responses, respectively (Refs. [43–45], but with exceptions [46]). Being able to convey approval through head gestures during conversation would be an important factor contributing to the perceived naturalness of an interaction. Additionally, the need to fit within the field of view of the camera may limit the overall movement and induce a feeling of being physically trapped [21]. In face-to-face meetings, people can shift their position and stretch, but during video communication their mobility is limited to a narrow space. This reduced mobility may undermine cognitive performance [47], further disrupting communicative abilities.

Moreover, in many video conferencing programs, there is a setting in which a selfimage of the participant is displayed along with the image of their interaction partner. This may be potentially disturbing in several ways. It may change the basic gaze dynamics, which was claimed to serve as a "glue" for interaction [48], and introduce effects on individuals' behavior similar to the presence of a mirror. Research in social psychology shows that seeing the self-image in a mirror can heighten self-focused attention, which in the case of longer exposition can have negative psychological consequences, including decreased mood or even depression ([19,49–51], but see [52,53]). In the interactive context, self-focused attention was reported to decrease prosocial behavior in some contexts [54], although it is possible to find conditions in which it enhances prosocial behavior [55]. Finally, seeing oneself in a mirror provides visual feedback—an additional affordance that might be used for more precise control of one's appearance and expression. Little is known about the consequences of the visible self-image for coordination with a conversation partner.

Our investigation complements existing studies on the naturalness of online interactions through the introduction of the movement coordination perspective and the dynamical systems methodology, which goes beyond individual cognitive processes by focusing on coupling. We show how the movements of individuals are constrained in video-mediated interactions, and what patterns of interpersonal coordination emerge.

#### *1.2. Current Study and Hypotheses*

The goal of our study was to explore movement coordination dynamics shaped by the affordances altered by video-mediated means of communication. We identified factors such as: restricted mobility in front of the camera, video latency and jitter, and the optional presence of one's own mirror image. All these components potentially constrain movement of the individual, modify adopted nonverbal communication strategies, and, finally, reshape interactive patterns of interpersonal coordination. To disentangle the influences of the video medium and the mirror image, we adopted an experimental design in which casual, friendly conversations of the same dyads were recorded in four conditions: (I) videomediated remote conversation with the mirror image displayed, (II) video-mediated remote conversation without the mirror image, (III) face-to-face conversation with the mirror image, and (IV) face-to-face conversation without the mirror image. We expected the differences to be manifested at the individual level and at the dyadic coordination level. At the individual level:

**Hypothesis 1.** *Overall movement will be more restricted in remote interactions, because of the need to stay visible (in the field of view of the camera) and to see the interlocutor.*

**Hypothesis 2.** *Intentional communicative gestures will be exaggerated (in comparison to the overall movement) in remote interactions to compensate for potential disruptions.*

**Hypothesis 3.** *The availability of the self mirror image in remote interactions will allow participants to calibrate their expressions, making the movement more natural and less exaggerated. No such effect is expected for face-to-face interactions, where natural instantaneous feedback is available through the partner's reactions.*

#### Regarding interpersonal movement coordination, we expected that:

**Hypothesis 4.** *Coordination will be more stable in face-to-face interactions, and episodes of coordination will be longer.*

**Hypothesis 5.** *Coordination will be less stable with the mirror image present, as it presents an additional distraction (participants captivated by their own movement may be less attentive to their partners).*

To operationalize our hypotheses, we tracked participants' head movement during conversations using OpenPose software [56]. We focused on head movements, as they were important and visible both in face-to-face and remote conversations. According to the existing literature, the dominant head gesture during conversations is nodding, which is associated with vertical motion [40,41]. Head nodding (vertical motion) and head shaking (horizontal motion) are typically distinguished as they are associated with positive and negative responses, respectively [44]. Head nodding was reported to increase the perceived likability and approachability of a person [43]. Following this logic, we decided to differentiate between vertical and horizontal motion in our analyses. After watching the collected video material, we discovered that there were multiple episodes of head nodding

in response to the partner, but hardly any head shaking. This was consistent with the friendly character of the conversations, where head nodding is expected to be much more prominent than head shaking [57]. Horizontal head movements in our recordings seemed to result not from head shaking, but mostly from body sways and position adjustments less connected with the conversation dynamics. Thus, at the risk of oversimplification and with the limits of cross-cultural generalization in mind, we interpreted vertical head movement as an indicator of intentional communicative gestures expressing positive reaction to the interlocutor, and horizontal head movement was treated as a control—an indicator of general body movement.

When operationalizing interpersonal coordination, we decided to focus on the congruence of head movement direction within the dyad. Two people moving their heads in the same direction (nodding, tilting, turning, etc.) simultaneously or with a constant delay examplify coordinated behavior. We quantified the coordination using cross-recurrence quantification analysis (cRQA) [58], a nonlinear technique providing measures of coordination stability.

#### **2. Materials and Methods**

#### *2.1. Participants and Setup*

The examined material consisted of 24 recordings (137 min in total), collected from interactions of two groups of three people: Group A consisted of three men, and Group B consisted of three women (age 22–35). All participants were university students. The study was approved by the ethics committee of the Faculty of Psychology, University of Warsaw. Participants gave their consent to record their conversations and use them for research purposes.

Participants were students in the same program. Their level of acquaintance was assessed through a short interview. Participants from Group A were attending online courses together and had a chance to get to know each other while doing a group project together. Participants from Group B were engaged in research within the same research group and spent some time socializing before participating in the study. They can be described as colleagues, but there were no close friends within either group. All conversations were held in English, which was the second language for all participants. All participants had previous experience using videoconferencing software and were used to this form of communication.

Within each group, everyone was paired up, therefore creating six dyads (three per group) in total. Each dyad engaged in two conversations: one conducted remotely and one face-to-face, and each of these conversations was divided into two parts: with the mirror image and without. Each part lasted approximately five minutes. We briefed the participants regarding the purpose of the study, length of the conversations and the differences between experimental conditions. Participants knew that their movement will be tracked and their coordination will be analyzed. They were not informed on the detailed study hypotheses. Participants were instructed to keep the conversations casual and choose the topic freely. Most of the conversations started with a general opening question ("What's up?") and then developed spontaneously. Topics such as university studies, work, vacations, hobbies, etc., emerged. All conversations were friendly in tone, and no controversial topics or heated debates occurred.

Figure 1 presents the general schema of the four experimental conditions.

**Figure 1.** General schema of experimental conditions. In Condition (1), "remote no-mirror", the participant sees their partner on the screen; in Condition (2), "remote mirror", the participant sees their partner and their own mirror image side by side. In Condition (3), the "live no-mirror" participant sits in front of their partner with a dimmed smartphone screen placed in between, and in Condition (4), the "live mirror" participant sits in front of their partner with a smartphone displaying mirror image placed in between.

The remote conversations took place on the Google Meet platform. Two participants engaged in the conversation, and the researcher joined the meeting and recorded the interaction using OBS Studio software for screen recording. The researcher recorded the meeting in a "gallery view" mode, where images of the two interlocutors were placed side by side. Both participants were recorded with lag characteristic for the videoconferencing platform. In the mirror condition (with self-view), the participants saw both the other person and their own face, while in the no-mirror condition (without self-view) they could only see their interlocutor. They conducted a single 10-min conversation starting without self-view and switching self-view after 5 min. Participants used their own laptops with built-in video cameras.

Before the actual recordings of remote conversations, trial recording sessions took place during which participants were able to familiarize themselves with the setup. After the trial sessions, participants were instructed to adjust their setup (position of the camera, lighting) to improve the quality of the recordings.

Face-to-face conversations were recorded via a smartphone camera connected to a laptop (using Droidcam OBS and OBS studio software). We connected two smartphones to the same laptop via a local WiFi network and used OBS studio to combine the two image streams into a single output video file in which images of two interlocutors were placed side-by-side (as in the typical videoconference setup). We placed each smartphone in front of one of the interlocutors, with the front camera filming one's face and upper body. The participants were given a few minutes to sit down and adjust their positions to make them feel comfortable and ensure they fit into the video frame. The mirror condition was reproduced by showing the person's face and upper body position and movements in real time on the smartphone screen. The participants had a single 10-min conversation, in which smartphone screens were dimmed for the first half and were switched on for the second half.

#### *2.2. Movement Tracking*

We converted the video recordings to a common video format with 20 FPS. Each video frame contained the images of two participants side by side. We cropped the videos to obtain a separate video file for each participant during each conversation. The minimal resolution of the cropped video was 530 × 304 pixels. All videos were downscaled to this resolution. We processed the videos with OpenPose software [56] to obtain the x-y coordinates of key body parts (see Figure 2). For our analyses we extracted coordinates of points P0 (tip of the nose) and P1 (point in the middle of the torso on the shoulder level). There were missing values due to the algorithm not identifying a keypoint on a particular frame. In the recordings of one male dyad in the remote condition, the numbers of missing values were particularly large (16–55%). We removed these two recordings from the analysis. A small number (<5%) of missing values in other recordings were imputed using linear interpolation. Afterward, we applied a running median filter with a window size of five for each coordinate separately to remove possible outliers.

**Figure 2.** Output from OpenPose program: a video frame with detected key points marked. The two key points used in our analysis are P0 (tip of the nose) and P1 (point in the middle of the torso on the shoulder level).

#### *2.3. Measures and Data Analysis Techniques*

In our analyses, we focused on the movement of two points: the tip of the nose (point P0), as an indicator of head movement, and the middle of the torso (point P1), as the reference (see Figure 2). To normalize the data, we used the average P0-P1 distance for each person as a natural scale of movement. To operationalize our hypotheses regarding individual movement, we introduced the following measures:


The described measures were calculated separately for each of the two members of the six dyads in each of the four conditions, which should result in 48 data points. Since we excluded two recordings of the particular dyad in the remote condition (see Section 2.2), the final number of analyzed data points was 44.

To analyze the properties of interpersonal coordination, we focused on the direction of frame-to-frame movement of P0 point. For each frame we calculated a 2D vector, representing the shift in position from the previous frame. All vectors were normalized to have unit length. A low-pass Butterworth filter was used to smoothen the data. Then we calculated the interpersonal coordination statistics using the methodology inspired by multidimensional cross-recurrence quantification analysis [59]. We constructed separately for x and y coordinates time-delayed embeddings using a delay of 7 frames and embedding dimension 4 (values chosen using minimal mutual information heuristic for delay and false nearest neighbors for dimension [60]). Embeddings for the two coordinates were concatenated, resulting in a final dataset with eight columns. We constructed a recurrence matrix by calculating distances between all pairs of 8-dimensional vectors and thresholding them using a fixed value. All distances below the threshold formed recurrent points. We chose the threshold value for each matrix separately to ensure that the fraction of recurrence points was always 10%. In this way, RQA statistics were normalized across dyads and experimental conditions (This methodology is different from some other studies using RQA (e.g., R ˛aczaszek-Leonardi et al. [11]), where threshold value is fixed across all samples and the fraction of recurrent points (RR) was compared across conditions. In the case of our data, differences in optimal threshold level were too large for this kind of comparison.).

In layman's terms, a cross-recurrence matrix represents the temporal structure of "meetings" of two evolving systems. A recurrent point with coordinates (*i, j*) means that system *A* at time point *i* was in the same state as system *B* at time point *j*. In the context of participants of our study, recurrence means that two participants moved in the same direction relative to their cameras. Recurrent points on the main matrix diagonal indicate that participants' movements were synchronized, while recurrent points outside the main diagonal indicate more complex kinds of coordination. We controlled for the fraction of recurrent points—denoting the overall strength of coordination— and quantified characteristic patterns of coordination through the analysis of diagonal and vertical lines formed by recurrent points. We will use the following notation: *l*—length of diagonal line, *P*(*l*)—probability of a diagonal line of length *l* occurring, *v*—length of vertical line, *P*(*v*) probability of a vertical line of length *v* occurring. Then, popular recurrence quantification measures can be defined as follows:

• Determinism, fraction of recurrent points forming diagonal lines.

$$DET = \frac{\sum\_{l=l\_{\min}}^{N} lP(l)}{\sum\_{l=1}^{N} lP(l)}$$

A large DET means that there are stable episodes of coordination and that coordination is more predictable. In interaction it suggests that partners may anticipate each other's actions and successfully maintain coordination.

• Entropy of the distribution of diagonal line lengths.

$$ENTR = -\sum\_{l=l\_{\min}}^{N} P(l) \ln P(l)$$

A large ENTR means that the coordination is more complex with more characteristic patterns of coordination. This suggests that the interaction process is more varied.

• Average length of a diagonal line.

$$L = \frac{\sum\_{l=l\_{\min}}^{N} IP(l)}{\sum\_{l=l\_{\min}}^{N} P(l)}$$

A large L means that the episodes of coordination are longer on average.


$$LAM = \frac{\sum\_{\upsilon=\upsilon\_{\min}}^{N} \upsilon P(\upsilon)}{\sum\_{\upsilon=1}^{N} \upsilon P(\upsilon)}$$

Vertical lines form when one participant remains in the same state (moving uniformly or being still) for some time. A large LAM indicates that participants' movement is steadier.

• Trapping time, average length of a vertical line.

$$TT = \frac{\sum\_{\upsilon=\upsilon\_{\min}}^{N} \upsilon P(\upsilon)}{\sum\_{\upsilon=\upsilon\_{\min}}^{N} P(\upsilon)}$$

A large TT means that the episodes of steady movement are longer on average.

We counted only diagonal and vertical lines of length 10 or more (*lmin* = *vmin* = 10 corresponds to episodes of coordination or steady movement lasting 0.5 s or more; this value was chosen empirically to ensure sufficient variability of DET and LAM statistics). RQA measures were calculated for each of the 6 dyads across 4 conditions, except for the one dyad for which recordings of remote interactions were excluded from the analysis (see Section 2.2). The final sample consisted of 22 observations.

We performed statistical analysis using mixed-effects linear models adequate for the repeated measures experimental design. All analyses were performed in Julia programming language using the packages DynamicalSystems.jl [61] and MixedModels.jl [62].

#### **3. Results**

#### *3.1. Horizontal and Vertical Mobility*

We started by comparing participants' mobility along horizontal and vertical dimensions across the experimental conditions (see Figure 3). The differences were quantified using mixed-effects linear models, with model coefficients presented in Table 1.

**Figure 3.** Average participants' mobility in horizontal (**a**) and vertical (**b**) dimensions, and their ratio (**c**) across experimental conditions. Mobility is defined as the standard deviation of the participant position on the video frame. For each dyad, two lines are drawn: one for Participant A, and one for Participant B (same color lines for participants in each dyad).

Horizontal mobility was similar in all conditions, vertical mobility was larger in remote conditions (*p* = 0.001), and the ratio was significantly larger in face-to-face conditions (*p* < 0.001). Additionally, studying the plot (Figure 3b) suggested that vertical mobility might be slightly larger in the "remote no-mirror" condition in comparison to the "remote mirror" condition. To verify this, we applied an additional paired samples Student's t-test which compared the two conditions. We obtained *t* = −4.8922 (*DF* = 10) and *p* < 0.001, which gives support to the hypothesis that the conditions differ.

Interpreting the results in the light of research hypotheses, we had to reject Hypothesis 1, as neither horizontal nor vertical mobility was visibly restricted in remote interactions. Hypothesis 2—stating that in remote interaction, participants exaggerate communicative gestures—was confirmed by the differences in vertical mobility and horizontal-vertical mobility ratio. Larger vertical mobility and a smaller ratio in remote conditions suggest that participants increased their range of nodding movements while restricting other movements. Finally, comparison of vertical mobility between the "remote mirror" and "remote no-mirror" conditions supports Hypothesis 3: the presence of self-image in the mirror condition reduced exaggerated nodding gestures.

**Table 1.** Coefficients of mixed-effects linear models comparing horizontal and vertical mobility across experimental conditions.


#### *3.2. Interpersonal Movement Coordination*

Figure 4 presents cRQA statistics for interactions in all four conditions, while Table 2 contains coefficients of mixed-effects linear regression models verifying the strengths of effects for each statistic. As we can see, differences between remote and face-to-face interactions are evident on all measures except TT, which is congruent with Hypothesis 4. We found no visible effect of mirror image presence on movement coordination; there is no support for Hypothesis 5.

**Figure 4.** cRQA statistics describing properties of participants' movement coordination across experimental conditions. For each dyad a single line is drawn.

**Table 2.** Coefficients of mixed-effects linear models comparing various RQA measures across experimental conditions.



**Table 2.** *Cont.*

#### **4. Discussion**

Our results show that shifting to remote communication changes the dynamics of movement manifested by individuals and on the dyadic level. Our intuitions that participants move differently during remote and live interactions were confirmed by quantitative analyses. During remote interactions, they exaggerated their nodding gestures, which may stem both from the awareness of their lesser visibility by the partner and compensation for the unnaturalness of the situation. This effect was reduced when the self-image was present. One of the possible explanations is that in remote conversations participants lacked some immediate feedback from their interlocutors and were unsure whether their gestures were visible. The self-image might have provided compensatory feedback allowing them to calibrate their expression.

On the dyadic level, we demonstrated that in video-mediated remote conversations interpersonal coordination was less stable (smaller DET), less complex (smaller ENTR) and occurred in shorter episodes (smaller L and Lmax). Our findings suggest that partners interacting remotely do not form a coupled system with the same properties as in natural face-to-face interactions. According to De Jaegher et al. [13], particular dynamics of social interaction enable processes of social cognition. With the altered interaction dynamics, these processes might be disrupted, diminishing mutual understanding between interaction partners.

Contrary to our expectations, the presence of the self-image in the mirror condition had no visible effect on movement coordination. It is possible that the movements we captured do not reflect the changes that might be induced by this presence—such as changes in gaze behavior. In any case, these changes did not result in altered coordination. It is also possible that the effects were too subtle to be detected in the current experimental design, e.g., due to the brevity of five-minute conversations.

The study of movement coordination not only provided objectively measurable determinants of the quality of communication but also allowed us to transfer the analysis from the level of the individual to the level of dyad dynamics. This is in line with the embodied and enacted perspectives on social interactions [2,13] and compatible with Burgoon's "principle of interactivity" [16], suggesting that the process of interaction afforded by a communication medium should be characterized first before investigating interaction outcomes. Our investigation of movement coordination complements individualistic studies pertaining to individual satisfaction and cognitive load during online conversations [19,20].

The interactive perspective might potentially provide an alternative explanation of the "Zoom fatigue" phenomenon. Our results demonstrate that interaction properties deemed to enable social cognition [13] are altered, and the coordination is overall less complex (smaller DET and ENTR) in remote interactions. In that case, what is missing are not so much individual social cues (such as gestures or facial expressions) but rather "interactive cues"—specific properties of the interaction dynamics that allow us to tell an affiliative conversation from a quarrel, the continuation of an ongoing conversation topic from the beginning of a new topic, etc. Lack of this interactional scaffolding might lead to confusion and frustration. Further research could test this hypothesis by combining the two perspectives and checking how the satisfaction reported by the respondents participating in video-mediated interactions is reflected in their coordination. This would confirm whether coordination properties are actually connected with the experienced fatigue. The results could also be compared with previous studies associating movement synchrony with positive outcomes in face-to-face interactions [7,8,63].

Another intriguing perspective that may provide a framework for reflection on the sources of perturbations in video communication is the comparison with audio-only communication. From an information theory perspective, a video call offers a channel of greater capacity—it allows transmitting more information than a phone call. However, despite audio communication being more limited, we observe no "phone fatigue" phenomenon. This may suggest that perhaps extra information provided by video communication is actually more cognitively demanding than helpful. For the "receiver", the nonverbal message might be more difficult to interpret because some contextual social cues facilitating the interpretation are altered in remote interactions (for instance, response times allowing the prediction of positive or negative reactions [34]). Increased channel capacity in the case of video-mediated interactions may also be more demanding from the sender's perspective. For example, being aware that at least some parts of their body are visible to the partners and therefore gestures are an important source of information on interaction, senders feel obliged to use their body language in the same manner as in a normal face-to-face conversation. This makes a difference with audio-only interactions, since the same body language that is appropriate during a phone call is no longer appropriate within video conversation. At the same time, remote communication limits the possibility of the natural use of body language, as demonstrated by our result of more exaggerated nodding gestures, which may lead to an experience of frustration or fatigue. Examination of the impact of these factors in comparison between video and audio-only conversations is another interesting line of further research, especially with an attempt to untangle the experience related to sender and receiver perspectives.

Continuing the information-theoretic considerations, we should also discuss the role of noise in the communication channel or the reliability of a medium. From the user point of view, a tool that offers less functionality but is more predictable is still more effective than a more powerful but unreliable tool [64]. A video call is a channel of greater capacity than an audio-only call, but at the same time, it is more affected by noise due to latency and jitter. Video calls are prone to image and audio lags and disturbances, which even if they are minor and seemingly insignificant, may keep both sender and receiver in a state of constant uncertainty about how much information is lost during the transmission. Shorter episodes of stable coordination in video-mediated interaction discovered in the study may be a sign of low reliability of this medium: whenever participants began to coordinate on a nonverbal level, an unpredictable signal distortion might have destroyed the coordination.

Our small exploratory study does not allow us to formulate any strong recommendations concerning preferred forms of remote communication. Nevertheless, some cautious observations can be formulated. Despite worries that the presence of the self-image makes the conversation less natural, it may have its use as a source of compensatory feedback during interaction. Using this option can thus be recommended. As coordination in remote interaction is overall less stable, some conscious effort could be made to stabilize it. The simplest idea would be to deliberately slow down and avoid fast gestures, which could be misinterpreted due to video lag. Assessing such a strategy would require additional studies.

#### *Limitations*

Although our results confirmed that movement coordination is impaired in remote communication, they do not allow us to draw conclusions as to the main factors that contribute to this result. We note that our data are not conclusive on the effect of the presence of the self-image in the mirror condition, which might be one possible source of distraction. We observed no significant differences in movement coordination comparing these two conditions of both live and remote conversations; the ineffectiveness of the variable manipulation may be the underlying reason. The participants, being aware that they were being recorded, might have a lower tendency to focus attention on their image than in a natural environment. Additionally, looking in a mirror while talking to someone across the table is much less natural than seeing one's own image during a video call, which could have had an impact on our results in face-to-face conversations. Furthermore, as our sample was very small, it did not allow us to study interindividual differences in responses to online interactions.

The study can be extended through tracking whole body position during conversations and including hand gestures, body positions, etc., in the analysis. It would be possible to supplement coordination measures with the measure of behavior matching, that is body position mirroring [9]. Specific gestures or expressions could be identified automatically using machine learning techniques [65]. To better render the differences in coordination in remote and live interactions it would be crucial to obtain measures on other "coupling means" in dyadic conversations than the movement itself, such as gaze coordination and vocal dynamics. Related to body movement coordination they would inform about the use of the relevant cues as affordances for interaction and allow for forming a fuller picture of the relevant differences.

#### **5. Conclusions**

The differences between video-mediated and face-to-face interactions cannot be explained by either the technical properties of the medium or individual cognitive processes alone. In this study, we tried to apply an interactive perspective to identify key factors shaping our experience of online interactions. In line with this perspective, our study revealed significant differences in patterns of interlocutors' coordination between video-mediated remote and live interactions. We demonstrated that in video communication, the stability of movement coordination is lower, which may have a negative impact on the overall quality of interaction. The presence of the mirror image did not have a detectable effect on coordination; however, it seems that the mirror image helped to control one's expression during remote interactions, making the communicative gestures less exaggerated. Vast differences in coordination patterns indicate that the remote medium radically alters the landscape of affordances for communicative actions. It remains to be seen which affordances result in those differences when they are altered.

**Author Contributions:** Conceptualization, J.Z., E.N., J.K.-M., K.S., K.Z. and J.R.-L.; data collection J.Z., E.N. and K.S.; methodology, J.Z.; investigation, J.Z.; writing—original draft preparation, J.Z., E.N., J.K.-M., K.S., K.Z., writing—review and editing, J.R.-L. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the Faculty of Psychology, University of Warsaw, from the funds awarded by Ministry of Science and Higher Education in the form of subsidy for the maintenance and development of research potential in 2021 (501-D125-01-1250000, zlec. 5011000615).

**Institutional Review Board Statement:** The study was approved by the Ethical Committe of the Faculty of Psychology, University of Warsaw.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are openly available in OSF at DOI 10.17605/OSF.IO/8YA47.

**Acknowledgments:** We are grateful to Urszula Kuczma, Fernando Garcia Hipola, Rati Tsiklauri and Yevhenii Rachynskyi for help in data collection.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**

