*1.8. Why Crowd Sensing?*

In order to investigate complex and dynamical societies, appropriate data are required. Unfortunately, acquiring such data is a challenge. The traditional methods of analysis in sociology gather qualitative data from interviews, observation, or from documents and records, and carry out surveys of samples of people [5].

The crowd comes in for three related reasons. First, the nearly ubiquitous use of mobile devices makes it simply possible to sense the required information on a large scale, at least in principle. Secondly, a crowd is necessary: without the information obtained from the set of persons that constitutes a crowd, no simulation would be possible at all. Thirdly, the focus on crowds goes along with two ongoing paradigm shifts in social research. The first of these paradigm shifts concerns the changing survey landscape (trend towards the use of non-probability samples) and the second one concerns the tendency from survey research towards social media research. Moreover, just recently, social research encountered the suggestion towards the creation of mass collaboration for data collecting purposes [17]—a trend in line with similar suggestions towards citizen sciences.

The analysis of crowd sensing data and crowd behaviour can be performed with different methods: 1. analytical methods [4]; 2. machine learning (ML); 3. simulation. Simulation and ML can be seen as counterparts to analytical methods or can be used in conjunction.

One major drawback of classical surveys is that they come from measurements made at one moment in time [5]. Crowd sensing combined with real-time simulation can extend the data time scale significantly.

### *1.9. The Role of Incentive Mechanisms*

Crowd sensing can be participatory or opportunistic. Furthermore, crowd sensing can be classified in platform- or source- and user-centric architectures [18]. In all cases incentive mechanisms have a high impact on crowd user selection and participation, directly affecting the quality of sensed data [19]. Although the usage of mobile phones introduces new opportunities in social data sensing, adapted incentive mechanisms are required. Distributed and self-organising crowd sensing commonly requires the installation of software, a high barrier for most users, with an impact on participation and data bias (young users vs. older users). Mobile chat-bot agents, e.g., embedded as (*JavaScript*) code in a WEB page (but executed on the user device) as considered in this work can overcome this limitation.

Among participation, the quality of the sensed data contributed by individual users varies significantly by crowd sensing [19]. Sensor fusion and localised pre-processing (filtering) are required. Correlating the quality of information provided by user to the incentive rewarded and as part of the auction model can improve overall data quality of mobile crowd sensing significantly.

### *1.10. Smart City Management*

Today administration and management of public services and infrastructure relies more and more on the user and ubiquitous data collected by many domestic and private devices including smartphones and Internet services. People use social digital media extensively and provide private data, enabling profiling and tracing. User data and user decision making have a large impact on public decision-making processes, for example, plan-based traffic flow control. Furthermore, intelligent behaviour, i.e., cognitive, knowledge-based, adaptive, and self-organising behaviour based on learning, emerges rapidly in today's machines and environments. Social science itself exerts influence on public opinion and decision formation.

Smart city infrastructures and management are characterised by adaptive and self-organising control algorithms using real-world sensor data. In [20], crowds are considered a collective intelligence that can be employed in smart cities, although such self-organising and self-adapting intelligence driven by a broad range of individual goals is inaccurate and can compromise society's goals.

Traffic is a classic example of group decision making with (social) neighbouring interaction, but commonly, traffic control relies on physical sensor processing only [21]. In [21], adaptive and partly self-organising traffic management was achieved by using agents with multi-levels of decision making and a hierarchical organisational structure. Car traffic control in smart cities can be achieved by global traffic sign synchronisation. But pedestrian, bicycle, and domestic traffic usage cannot be controlled this way. A future vision to control domestic traffic flows is the deployment of digital and social media with chat bots to influence people's decision making regarding goal-driven mobility, sketched with the methods introduced in this paper.

Traffic control can be performed by perception and analysis of vehicle and/or crowd flows. Furthermore, vehicle-flows can be classified, e.g., introducing weights for individual and public vehicles.

Surveys play another important role in the modelling and understanding of interaction patterns. Artificial Intelligence (AI) and chat bots shift classical survey methods towards computational methods introducing possible implications, i.e., usage of different data, different methods, exposition to different threats to data quality (concerning sampling, selection effects, measurement effects, and data analysis), and different rules of inference are likely to result in comparably different conclusions and public reports, and hence in different input to public opinion formation. Among field studies, simulations can contribute to the investigation and understanding of such interactions.

### *1.11. The Underlying Computer-Science Perspective: A New Augmented Simulation Paradigm*

In real worlds, hardware and software robots can be considered close together in a generalised way by an association with the agent model. One prominent example of a software robot is a chat or social bot. Additionally, real and artificial humans can be represented by the agent model, too. Multi-agent systems enable modelling of agent interaction and emergence behaviour. Agent-based modelling and simulation is a suitable methodology to study interaction and mobility behaviour of large groups of relevant human actors, hardware, and software robots, using methods with inherent elements of AI (e.g., learning algorithms) or which use data which may be influenced partly by bots.

Agent-based methods are established for the modelling and studying of complex dynamic systems and for implementing distributed intelligent systems, e.g., in traffic and transportation control (see [21,22]). Therefore, agent-based methods can be described by the following taxonomy [15]:


The fifth paradigm is the novelty introduced in this work with the application to social mobility and crowd interaction simulation.

Two promising fields of current studies in computer science are data mining (DM) and Agent-based modelling and simulation (ABMS) [15]. An agent-based simulation is suitable for modelling complex social systems with respect to interaction between individual entities, manipulation of the world, spatial movement, and emergence effects of groups of entities. The main advantage is the bottom-up modelling approach composing large-scale complex systems by simple entity models. The main disadvantage of ABM is the (over-) simplified entity behaviour and simplification of the world the entities are acting in. Commonly, simulations are based on synthetic data or data retrieved by field studies. Many simulations and models are lacking in diversity that exists in the real world. Commonly, sensor and model data (parameters) used in simulations (virtual world) are retrieved from experiments or field studies (real world), shown in Figure 1.

But there is neither feedback from the virtual to the real world nor an interaction of the real world with the virtual world. For example, in [21] self-organised traffic control was simulated with agents by using the JACK software for intelligent agents, which is an agent-oriented development environment built on top of and integrated with the Java programming language and Matlab software. The simulation was based on a parameterised traffic model. Sensor data were created synthetically within the simulation without real-world coupling.

Parameterised models (modelling behaviour and interaction), discussed in Section 3, play an important role in ABM and ABS. The achieved results and conclusions rely on the parameter settings, and the selection of appropriate and representative parameter sets is crucial for modelling real-world scenarios accurately. Furthermore, parameter sets derived from real-world sensing enables variations and evaluation for different real-world situations.

*To overcome the limitations of closed and synthetic simulation worlds, we investigated a new augmented agent-based simulation paradigm coupling real and virtual worlds in real-time, although this is an optional feature.*

Controlling ABS by human participation appears in early work in stock market analysis and prediction [23].

ABS and ABMS are commonly constructed using collections of condition–action rules to be able to percept and react to their situation, to pursue the goals they are given, and to interact with other agents, for example, by sending them messages [5]. The *NetLogo* simulator is an example of an established ABS tool used in social and natural sciences [14], but is limited to behavioural simulation only (ABM domain). In this work, a different simulation approach combining ABM, ABC, and ABS methodologies, is used by deploying the widely used programming language JavaScript. JavaScript is increasingly used as a generic programming language for ABC and ABS [24,25].

**Figure 1.** (**Left**) Traditional field studies and agent-based simulation of social systems. (**Right**) New combined agent-based simulation and agent-based crowd sensing enabling bidirectional data exchange.

### *1.12. Sensing the World*

In crowd sensing, users of mobile devices are sensors and actuators [20]. Mobile devices like smartphones are valuable sources for social data [26], either by participatory crowd sensing with explicit participation of users providing first-class data (e.g., performing surveys or polls) or implicitly by opportunistic crowd sensing collecting secondary class data, i.e., traces of device sensor data delivering, e.g., actual position, ambient conditions, network connectivity, and digital media interaction. Crowd sensing and social data mining as a data source contribute more and more to investigations of digital traces in large-scale machine–human environments characterised by complex interactions and causalities between perception and action (decision making). This is relevant for the algorithmic and architectural design of smart cities [20]. Agent-based software is already used in crowd sensing applications [27], which has advantages in adaptivity over traditional static client–server architectures.

But mobile devices are not limited to being sensors. They can be used as actuators, too, by interacting with the user via the chat dialogue or social media that can affect the user behaviour (e.g., mobility decisions).

It is difficult to study such large-scale data collection, data mining, and their effect on societies, domestic services, and social interaction in field studies due to a lack of reliable data and complexity. Controlling all aspects of the virtual world allows explaining the source of structural variation in human

interactions, i.e., tackle the hard-to-observe mechanisms of micro–macro interactions. For instance, it is known that the creation processes of social networks rely heavily on the (organisational) environment [28]. Utilising real-world interactions together with ABMs in carefully selected, systematically diverging environments creates a framework in which we can experimentally distinguish between ideal typical situations and understand how different structures result in similar/diverging behaviour. First applications might involve hierarchical/unstratified environments represented by different types of organisations. In so doing, we would add to the question of how the same micro-mechanisms (e.g., homophily, balance) are created by and reproduce different network structures and, hence, extend emerging approaches on network ecological theory.

Agent-based modelling of socio-technical systems is well established [29] and can be applied for smart city management, however commonly applied in an artificial world, i.e., a simulation is performed in virtual reality worlds only to derive and proof models under hard limitations. In this work, a new concept and framework for augmented virtual reality simulation are introduced, suitably, but not limited to, investigate large-scale socio-technical systems. Mobile agents are used already successfully in field crowd sensing [30]. In this work, mobile agents are used to combine in-field ubiquitous crowd sensing, e.g., performed by mobile devices with simulation.

A chat bot agent is capable of performing dynamic dialogues with humans to get empirical data into the simulation and to propagate synthetic data from the simulation world into the real world. The chat bot can act as an avatar providing information for users, e.g., for optimised and dynamic traffic control based on real (covering actual and history data) and simulated data (addressing future predictions).

Chat bot agents (as computational agents) can operate both in real and virtual worlds including games and providing a fusion of both worlds by seamless migration. Mobile computational agents, i.e., mobile software, have advantages in strong heterogeneous systems, existing in opportunistic and ad-hoc crowd sensing environments. Examples are traffic control by smartphone data. Agents are loosely coupled to their environment and platform and interact with each other, e.g., via tuple spaces (generative data-driven communication) and via unicast or broadcast signals (addressed messages) [31].

The novel MAS crowd sensing and simulation framework, introduced in the next sections, is suitable to combine social and computational simulations with real-world interaction at run-time and in real-time, e.g., by integrating crowd sensing, using mobile agents. There are two classes of mobility that can be modelled in the simulation: (1) Social mobility on short- and long-term time scales, e.g., creating group formation and segregation, and (2) goal-driven mobility, e.g., creating traffic.

Agent-based traffic management simulation still neglects social interactions and the influence of social media, e.g., MATISSE, the Multi-Agent based Traffic Safety Simulation System [32], which is a large-scale multi-agent-based simulation platform designed to specify and execute simulation models for agent-based intelligent transportation systems.

Among the capability of real-time simulation, the simulation framework is capable of creating simulation snapshot copies that can be simulated in parallel in non-real-time to get future predictions (simulation branches) from actual simulation states that can be backpropagated into the current real-time simulation world.

### **2. Intelligent Machines and Socio-Technical Systems**

There are basically three classes of entities existing in real and virtual worlds: 1. Humans, 2. machines, and 3. robots. Automatic and autonomous vehicles belong to class 3; semi-automatic driving is, e.g., relevant for smart traffic flow management.

Human–machine interaction is an emerging field. The interaction with intelligent machines is much more sophisticated and created two classes of interaction: 1. human–machine (HM), and 2. machine–machine (MM). Today, robots can be considered as intelligent machines (i.e., AI bots) posing some kind of autonomous behaviour, learning, and interacting with environments and humans. The main advantage of ABMS and the proposed agent-based framework is the unified handling of

totally different entities using the same agent model. Autonomous vehicles are examples of both HM and MM interaction.

There is a micro-context of everyday life and the macro-context of public decision and opinion formation related to human–robot interaction, affecting, e.g., future city management like adaptive and dynamic traffic control considering a broad range of participants and vehicles (cars, buses, trains, pedestrians, and bicycles). Automatic and probably autonomous driving systems can be considered as intelligent machines (i.e., robots) interacting with humans and environments, too. Social behaviour of humans and interaction with intelligent machines will depend on the way intelligent machines will be controlled and the way they interact with humans and other intelligent machines. For example, chat bots influence human opinions, but humans and other chat bots can influence the behaviour and knowledge base of chat bots, too. There are different involved ways of mutual affection that are difficult to investigate in classical field studies. The proposed simulation method integrating real-world environments by using mobile agents can boost investigations of such complex interacting environments. For example, decisions of traffic participants (e.g., car drivers) cause wide-ranging effects on global behaviour (emergence), and centralised or decentralised traffic control will influence individual behaviour.

Since both sides (humans and bots) may learn from each other in this interaction, the studies shall explore possibilities of targeted responses of the human side. That way we expect to reveal options of understanding the influence on the possible ways that AI will shape future society. In addition, we expect to find out the factors in robot–human interaction that let AI gain acceptance in society. This list of factors shall, amongst others, include the visibility of AI during a communication (e.g., can a human discern whether a comment comes from a bot or a human and what difference this makes to his/her response?) and different programmable behavioural and ethical principles in robot–human interaction (by systematically varying the reward function). Furthermore, since we expect in the medium- and long-term, respectively, a trend towards the use of robots that not only assists but replace humans in daily communication, we project the development and exploration of the first occurrence of this in a home environment, and the evaluation of its applicability in practice.

At society's macro-level, robot–human interaction takes place too, though less visible and less direct. The investigation of public decision and opinion formation is of high relevance for the design of public management services. We project an exploration of how this process is shaped in the interaction of groups of relevant human actors who interact with intelligent machines through its reports of findings of projects which either uses methods with inherent elements of AI (e.g., learning algorithms) or which use data which may be influenced partly by bots. In this regard, the study is targeted on possible emergent effects of AI at the system level of society.

To summarise, the agent model used in this work captures different entity classes, mapping activities (performing actions like environmental interaction) *A*, transitions *T* between activities (that can be conditional based on *V* and *S*), data variables *V*. and sensors *S*, on actions *X*, discussed in Section 4:

$$\text{Ag}(A, T, V, S) : \mathbb{S} \times A \times T \times V \to X. \tag{1}$$

### **3. Parameterised Modelling and Emergence**

Modelling of social and socio-technical systems can be classified in macro-scale and micro-scale models addressing a macro- and micro-context discussed in the previous section.

Commonly, social behaviour is studied on a macro-scale level with a unified model assuming a common mean of individual behaviour or on a micro-scale level with a unified behaviour model. Social behaviour and interaction models with parameters are used to study more heterogeneous and individual behaviour beyond a unified mean.

On a micro-scale level, it is assumed that there is an individual parameterised behaviour model *M* for individual entities or groups of entities, i.e., a model of social interaction, which is used to model physical entities and digital twins representing individual real humans (with *P*: Parameter set, *S*: Sensor set, *X*: Action set):

$$M(S, P): S \times P \to X. \tag{2}$$

A set of individual entities (agents) modelled by *M*(*P*) interacting with each other within a bound space will create a specific observation on system level, i.e., emergence effects. One example is a traffic jam as a result of individual behaviour and constraints.
