**The Impact of Technology on People with Autism Spectrum Disorder: A Systematic Literature Review**

#### **Katherine Valencia \*, Cristian Rusu \*, Daniela Quiñones and Erick Jamet**

School of computer Engineering, Pontificia Universidad Católica de Valparaíso, Valparaíso 2340000, Chile; daniela.quinones@pucv.cl (D.Q.); erick.jamet@gmail.com (E.J.)

**\*** Correspondence: katherinevalencia25@gmail.com (K.V.); cristian.rusu@pucv.cl (C.R.); Tel.: +56-942-954-648 (K.V.); +56-988-244-506 (C.R.)

Received: 4 September 2019; Accepted: 11 October 2019; Published: 16 October 2019

**Abstract:** People with autism spectrum disorder (ASD) tend to enjoy themselves and be engaged when interacting with computers, as these interactions occur in a safe and trustworthy environment. In this paper, we present a systematic literature review on the state of the research on the use of technology to teach people with ASD. We reviewed 94 studies that show how the use of technology in educational contexts helps people with ASD develop several skills, how these approaches consider aspects of user experience, usability and accessibility, and how game elements are used to enrich learning environments. This systematic literature review shows that the development and evaluation of systems and applications for users with ASD is very promising. The use of technological advancements such as virtual agents, artificial intelligence, virtual reality, and augmented reality undoubtedly provides a comfortable environment that promotes constant learning for people with ASD.

**Keywords:** user experience; accessibility; autism spectrum disorder; game-based learning; systematic literature review

#### **1. Introduction**

Currently, autism spectrum disorder (ASD) affects a significant number of people who have difficulties with communication and socialization, which results in complexities for their learning. Studies have examined the use of technology and computer-based interventions to teach people with ASD language and social skills [1]. Specifically, students on the autism spectrum enjoy playing games, which provides a safe environment [2]. Thus, we reviewed the existing literature about the relationship between technology, games, user experience, accessibility, and the education and skill development of people with ASD. This article is organized as follows: Section 2 presents the theoretical background, Section 3 describes the research methodology, Section 4 analyzes the results obtained, and finally, Section 5 highlights the conclusions and recommendations for future work.

#### **2. Theoretical Background**

#### *2.1. Autism Spectrum Disorder*

Asperger's syndrome was defined in 1944 by Hans Asperger [3]. The fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) [4] defines autism spectrum disorder (ASD) as a condition characterized by deficits in two core domains: (1) social communication and social interaction and (2) restricted repetitive patterns of behavior, interests, and activities. Since 2013, the DSM-5 has recognized Asperger's disorder, childhood disintegrative disorder, Rett's disorder, and several other related disorders, as part of ASD. However, many studies still use the Asperger's syndrome and ASD almost interchangeably.

In a study carried out by the National Institute of Health (NIH) of the USA [5] published in June 2018, it was estimated that 2.41% of children in the United States of America have an autism spectrum disorder. This shows an increase of 0.94% compared to 2010.

#### *2.2. User Experience*

The international standard on ergonomics of human system interaction, ISO 9241-210 [6], defines user experience as "user's perceptions and responses that result from the use and/or anticipated use of a system, product or service". In other words, the user experience is the degree of "satisfaction" that the end user has with the system or service after using it, that is based on each of the interactions that he or she has.

According to Peter Morville [7], user experience is meaningful and valuable when a product, service or system is useful (that is, its content is original and satisfies a need), usable (the product is easy to use), desirable (the image, identity, brand, and other design elements produce positive emotions towards the product), locatable (the content is accessible to people with disabilities), credible (users have confidence in the product), and valuable (an added value is generated from the product).

#### *2.3. Accessibility*

The international standard on ergonomics of human system interaction, ISO 9241-171 [8] defines accessibility as the "extent to which products, systems, services, environments, and facilities can be used by people from a population with the widest range of user needs, characteristics and capabilities to achieve identified goals in identified contexts of use". In other words, accessibility is the condition that environments, services, processes, and objects (everything that involves an interaction) must meet, which must be understandable and usable by the broadest range of people, regardless of their capabilities.

#### *2.4. Game-Based Learning*

Games that use technology are widely used to teach people conceptual knowledge and skills. There are different implementations of such games, such as serious games, gamification, and e-learning.

#### 2.4.1. Serious Games

Serious games are games whose main objective is not fun or entertainment but the learning or practice of skills. In 1970, Clark Abt [9] defined this concept as follows in his book called "Serious Games"—"games that have an explicit and carefully thought-out educational purpose and are not intended to be played primarily for amusement. This does not mean that serious games are not, or should not be, entertaining".

#### 2.4.2. Gamification

The concept of gamification was developed in 2003, and its use became widespread in 2010 through the work of multiple professionals. Gamification is formally defined as "the use of game elements and game design techniques in nongame contexts" [10]. When we talk about gamification, we tend to interpret it as a methodology where the purpose is to provide rewards to users to inspire personal and collective commitment, but this interpretation is very far from reality. Many authors maintain that the success of a gamified system or process lies in good design and adequate feedback, among many other factors. Other authors have supported this argument: for example, Kapp [2] stated, "Do not think of gamification as only the use of badges, rewards, and points. Instead, think of the engaging elements of why people play games—it is not just for the points—its [sic] for the sense of engagement, immediate feedback, and the success of striving against a challenge and overcoming it".

#### 2.4.3. E-Learning

The term "e-Learning" comes from the abbreviation of "electronic learning". Khan [11] defined e-Learning as "a hypermedia instructional program that uses the attributes and resources of the Internet to create meaningful learning environments." That is, e-Learning refers to online teaching and learning through the Internet and technology.

#### *2.5. Game Elements*

Game elements are the components that make up a game to create an attractive experience for players. Werbach [10] described 25 such game elements. For the purpose of our study, we identify the relevant game elements are as follows:


#### **3. Research Methodology**

This systematic literature review was carried out following the process proposed by Kitchenham [12]. Kitchenham outlined three fundamental phases for conducting a review of the literature: (1) planning the review, which includes creating the research questions and reviewing the protocol; (2) conducting the review, which includes the review, the selection and quality of studies, data extraction and data synthesis; and (3) publicizing the results after the review. Next, we detail the process followed for this document.

#### *3.1. Research Questions*

To cover every topic of interest in this systematic literature review, we formulated three research questions. These questions consider relevant and general aspects important for comprehending the concepts that we think are important for this study. These questions can be seen in Table 1.


**Table 1.** Research questions for the systematic literature review.

#### *3.2. Data Sources and Search Strategies*

To conduct this systematic literature review, we searched for scientific papers on five databases: IEEE Xplore Digital Library, ACM Digital Library, Science Direct, Scopus, and Web of Science. For

these sources, we considered only documents that were relevant in computer-related categories, such as technology, engineering and computer science, excluding categories related to medicine or chemistry. Additionally, we selected articles published during the last 10 years, between January 2009 and June 2019.

#### *3.3. Article Selection*

Once we chose the databases to search, we determined the specific search strings to find articles to answer the research questions and defined the exclusion and inclusion criteria to refine and filter the articles found.

#### 3.3.1. Search Strings

We formulated the search strings based on the relevant topics to our systematic literature review. We determined a set of specific keywords to use in our queries, i.e., "Autism Spectrum Disorder", "Accessibility", "User Experience", "Gamification", "Serious Games", and "Game Elements" that would be useful to answer our research questions.

These strings were focused on finding studies that analyzed or experimented with the use of games with people with ASD, considering aspects such as the user experience, accessibility, and game elements. In Table 2, we present the specific search strings that were used in the selected databases.



#### 3.3.2. Study Selection Criteria

To answer the research questions based on the selected articles and develop a general knowledge of the concepts that we were working with, we included the conditions listed in Table 3.


**Table 3.** Inclusion criteria.

The types of papers presented in Table 4 were excluded.

#### **Table 4.** Exclusion criteria.


#### *3.4. Document Selection*

Applying the selection criteria, we gathered a total of 94 articles. Figure 1 shows the general process flow of the search and study selection for this review, detailing the inclusion and exclusion criteria applied in each step.

**Figure 1.** Flow chart with the results of the article selection process.

#### *3.5. Data Synthesis*

After the search, we extracted the information from each of the 94 studies, summarizing and tabulating the information based on different metrics, such as the year published, document type and paper category. In the following steps, we detail each of the metrics.

#### 3.5.1. Year of Publication

As detailed above in the inclusion criteria section, we considered studies published during the last 10 years, between 2009 and 2019. As shown in Figure 2, we plotted the number of studies that were found that were published between 2009 and 2018, and we observed an increase in publication on this topic over this period. The studies found in 2019 are not presented in this plot because it would have been misleading to show incomplete data, as this review was finished in June 2019. Seventeen studies published in 2019 were found (almost equal to the number of publications in 2018), which led us to believe that this number will undoubtedly increase significantly during the remaining months of 2019.

**Figure 2.** Year of publication.

#### 3.5.2. Document Type

We analyzed the origin of the studies reviewed and determined whether they were conference proceedings or had been submitted to a scientific journal. Figure 3 shows a relative balance between the number of papers that were published as conference proceedings and in journals.

**Figure 3.** Document Type.

3.5.3. Document Categories

The studies were categorized as follows:


Figure 4 shows that 74.5% of the studies analyzed were case studies. It is believed that this is because the researchers were focused mainly on conducting investigations and accomplishing their study objectives, such as teaching conceptual skills.

#### **4. Results and Discussion**

After applying each of the filters described in the "Study Selection Criteria" section, as shown in Figure 1, a total of 94 studies were obtained. These studies were analyzed under different metrics, as seen in the "Data Synthesis" section. Based on our review of these studies, we now answer our research questions, considering those studies that are relevant to the specific context of each question.

**RQ1. In what way does the use of technology contribute to the education of people with autism spectrum disorder?**

As mentioned in the previous sections, ASD is a condition that is categorized as a disability due to the cognitive disorders that people with ASD face [13]. Several studies showed that most people with autism show a natural affinity for technology and a good disposition for using technology and learning through the use of computers [14]. This is because the environment and context that these experiences provide are predictable and structured, which helps people with ASD to maintain their routines and repetitive behaviors without affecting their comfort [15].

Several studies proposed the use of modern technologies to help teach skills to people with ASD. Some interesting examples of new technological approaches are the use of sensors, virtual reality, virtual agents, augmented reality, geolocation, and Kinect, as presented in the following studies. Wojciechowski et al. [15] developed a mobile application that, in conjunction with the use of Estimote Beacon sensors to identify objects, supports children with ASD in pronouncing new words and identifying their meanings. Lorenzo et al. [16] proposed an application that uses virtual reality and robots with cameras to detect children's emotions, adapt system interactions and thus develop social skills in students with autism spectrum disorder. Bernardini et al. [17] presented ECHOES, which is a serious game that focuses on the development of activities to promote social communication in children with ASD using an autonomous virtual agent that acts as a companion for children during their interactions with the system. Sorce et al. [18] developed an exploratory study to evaluate the effectiveness of the use of Kinect as a tool to allow people with ASD to explore works of art in a touchless virtual environment and assess whether this generates greater interest in them. Escobedo et al. [19] presented the Mobile Social Compass (MOSOCO) application, which makes use of augmented reality through a mobile device camera to include game elements in real social situations with the aim of developing social skills in children with ASD. Silva et al. [20] presented a serious game that, through geolocation, virtual reality and augmented reality, creates a virtual environment with 3D virtual monsters positioned all over the world that aim to teach children with ASD relevant educational content, such as vocabulary.

In addition to examining the studies from a technological perspective, we categorized the 94 studies based on the following learning topics with the goal of understanding the contribution of technology to education for people with ASD in terms of the specific skills that they focus on teaching: Conceptual Skills (subtopics: Language, Money, Colors, Mathematics, Programming, and Science), Practical Skills (subtopics: Health, Daily Life, and Transportation), Social Skills (subtopics: Communication, Emotions, and Interpersonal Relationships) and General Skills (subtopic: General). Table 5 shows the percentage of studies for each of the topics and subtopics, and in the same way, Table A1 (available in Appendix A) details each of the topics and subtopics according to which the articles were categorized. The results obtained after categorizing the studies are presented in the following sections.


**Table 5.** Learning Topic.

#### *4.1. Conceptual Skills*

First, 25.53% of the studies focused on analyzing and fostering skills within the range of Conceptual Skills. Studies in the Language subcategory focused on promoting the learning of expressions, thoughts and feelings through words. Examples of this include studies [13,21]. Arciuli and Bailey [13] analyzed a small group of children with ASD that were literate using the ABRACADABRA application and observed significant improvements in reading accuracy in participants who interacted with the system but not in children who did not use the application. For the children who did not use the application, their lack of improvement was believed to be due to their lack of socialization aspects that children must exhibit when interacting with a teacher to develop reading ability. Khowaja et al. [21] developed a prototype of a serious game for children with ASD to learn vocabulary. The effectiveness of the game was assessed through the comparison of children's performance at the beginning of the intervention, after the use of the prototype and 1–2 weeks after the use of the prototype, which enabled the researchers to track the improvement in the children's vocabulary.

Another subcategory of Conceptual Skills is the Money subcategory and only one study [22] was assigned to this subcategory. Caria et al. presented the design of a game that helps people with autism spectrum disorder acquire skills to help them understand the concept of money and its applications in real life, which was tested by obtaining positive and promising results.

In addition, like the Money subcategory, the Colors subcategory also included only one study, [23]. In this study, based on cognitive theories, Tu ˘gbagül et al. developed a computer interface for students with ASD and mild mental disability that used their preferred colors and helped them maintain their concentration.

Additionally, the studies in the Math subcategory aimed to develop skills related to numbers. Examples of studies in this subcategory are [24,25]. Tashnim et al. [24] developed the Play and Learn Number (PLaN) application, which teaches arithmetic and calculus to children who have ASD and helps children memorize and recognize numbers (in or not in sequences) through animated images. Muñoz-Soto [25] developed an application to support professionals in teaching functional mathematics and calculus to children with ASD. Through tests, it was possible to demonstrate that this application promotes the development of mathematical skills. However, it was suggested that the application should be tested by more users and in different institutions.

The Programming subcategory included studies that aimed to develop skills related to computational programming, for example, to design and order actions and commands. Only two studies were assigned to this subcategory, i.e., [26,27]. Eiselt and Carter [26] planned and conducted programming classes through Scratch for children with ASD with the aim of developing their technical and social skills. Despite their efforts, no real evidence of an increase in students' social learning or behavior was found. However, while the students did not develop social skills as expected, the authors suggested that the students knew more about programming after the experiment since at the beginning, they did not have any notion of programming, but after the experiment, they could read and write processing programs. Schmidt and Beck [27] proposed a learning intervention based on digital games for young people with ASD to develop their social skills as they worked on teams to solve introductory computer programming problems with virtual and programmable robots. According to the authors, this intervention has the potential to help participants develop social skills, however, because this study was only concerned with the initial stages of development, there was no analysis of the data, so conclusions regarding cognitive skills could not be made with certainty.

Finally, the studies in the Science subcategory investigated and interpreted natural, social, and artificial phenomena. For this subcategory, we found only one study [28], in which Eder et al. developed a mobile game application as a complementary learning material to teach children with ASD parts of the human body. After the intervention, it was observed that the application was very useful for teaching and that the motivation levels of the participants increased significantly.

#### *4.2. Practical Skills*

Second, the Practical Skills category included only 8.51% of the identified studies and was subdivided into several subcategories. First, the Healthcare subcategory concerned teaching about the health care that people should have. An example of a study in this subcategory is [29]. De Urturi et al. [29] developed a system consisting of a set of serious games aimed at teaching first aid (such as what to do in certain situations and basic knowledge about medical care and medical specialties) to people with ASD. Because the application was still in development, only partial results were available, so to determine if these results were promising, the authors administered a simple questionnaire to the participants, as they obtained positive results, they decided to continue developing the project.

Another subcategory of Practical Skills is the Daily Living subcategory. The studies in this subcategory focused on building knowledge about the development of daily recurring activities, and examples are [30,31]. Pérez-Fuster et al. [30] analyzed the impact of an intervention with digital technology (DT) compared to that of a treatment-as-usual (TAU) intervention on adults with ASD. The DT intervention sought to improve daily life skills, such as washing dishes and washing clothes. The results showed that the DT intervention significantly improved the daily life skills of the participants and was more effective than the TAU intervention. Santarosa and Conforto [31] presented a tablet application for children with ASD and children with intellectual disability (ID) that seeks to teach and develop routines in the classroom and verbal communication by directly involving teachers and assistants in schools. Children with ASD successfully adapted to the application, and their socioadaptive behaviors both in the classroom and related to verbal communication improved greatly. On the other hand, children with ID did not achieve autonomous use of the application, and they only had improvements in nonverbal classroom routines.

The final subcategory within the Practical Skills category is the Transportation subcategory. The studies in this category were concerned with teaching the necessary knowledge that individuals need to be able to transport themselves effectively. Some examples of this are found in [32,33]. McKissick et al. [32] investigated the impact of a computer instruction package to teach map-reading skills to three elementary students with ASD. Very promising results were obtained for interventions that used technology with children with ASD, such as increased levels of learning and improved learning habits among students. De Los Rios [33] proposed a draft of a study to evaluate platforms and interfaces that help users transport themselves, such as Google Maps or Apple Maps with eye tracking. They compared these platforms and interfaces with a proposed system that would provide a more personalized environment that is adapted and accessible to the needs of people with ASD.

#### *4.3. Social Skills*

Third, the Social Skills category included 36.17% of the total resulting studies and was subdivided into three subcategories. The studies in the first subcategory, Communication, focused on the development of skills such as exchanging information between two or more individuals and examples from this subcategory are found in [34,35]. Milne et al. [34] investigated the use of autonomous virtual humans (self-directed) to teach and facilitate the practice of basic social skills in greetings, conversation, listening, and shifts in conversation to people with ASD. The results were positive, as users increased their knowledge and development of social skills. In addition, it has been indicated that this approach was well received by participants and caregivers. Ribeiro and Barbosa [35] developed a game called ComFiM, which aims to encourage communication between people with severe degrees of autism. The game was evaluated based on the perceptions of the interlocutors of each player and the communication intentions observed between the players to collaborate with each other and the results showed that the application positively influenced the communication intentions of the players.

The Emotions subcategory included studies that examined the development of skills such as the identification of facial emotions. Some studies from this subcategory are [36,37]. Romero [36] carried out a computer-based intervention to teach the recognition of emotions to students with communication and social skill deficits. All participants showed improvements when assessing and recognizing emotions on faces, but it was suggested that the effectiveness of the intervention should be tested in a larger population. Christinaki et al. [37] presented a serious game with a natural user interface (NUI) interaction that aims to teach young children with ASD to recognize and understand different facial emotions. The authors concluded that technological interventions with NUI improve the learning process and indicated that the emotional state of the players is directly related to their learning skills.

Additionally, the studies in the Interpersonal relationships subcategory emphasized individuals' development of relationships. Some of the studies that were assigned to this subcategory are [38,39]. Boyd et al. [38] described how collaborative assistance technologies, such as the Zody collaborative game, can be used to facilitate social relationships in children with ASD. They discussed how design can foster three levels of social relationship, i.e., membership, partnership, and friendship, even without the help of adults. The results indicate that collaborative technologies provide support for the development of social skills at different levels of intimacy between players without a mediator during the intervention. Hourcade et al. [39] conducted an intervention with multitouch tablets with children with ASD to promote their social skills and help them develop their creativity, alter their interests, and be able to understand emotions. The result of the intervention was that it increased pro-social behaviors, such as collaboration, coordination, and interest in social activities, in children with ASD.

#### *4.4. General Skills*

Finally, the General Skills category included 29.79% of the studies. As this category referred to a range of topics, we defined only one subcategory, the General subcategory; some example studies are [40,41]. Backman et al. [40] investigated a method of evaluating children on the autism spectrum through computer games, which provide an objective, motivating, and safe evaluation of the participants. Although more research was recommended, the results showed that computer games have great potential in special education as an evaluation tool to clarify the difficulties associated with ASD. Hulusic and Pistoljevic [41] presented the initial development process of the LeFCA framework, which was used to teach children with ASD basic skills and concepts. LeFCA consists of four games that focus on developing basic skills (such as labeling, pointing and pairing in reference to visual and auditory stimuli) necessary for learning. Each of the participants was constantly motivated to play, and the skills learned could be extrapolated to new media or environments without the need for any training.

After reviewing all the studies and classifying them based on their learning topics, as shown in Table 5, we can see that there are a few studies that used modern and/or complex technologies, such as virtual reality or sensors. These technological approaches are interesting examples of how this area is developing in innovative ways.

Notably, most of the studies focused on teaching Social Skills, such as Emotions (12.77%), Communication (9.57%), and Language (14.89%), which are the most important areas that people with ASD have difficulties with.

#### **RQ2. Which user experience and accessibility elements**/**methods are considered when analyzing the impact of technology on people with autism spectrum disorder?**

Although many of the studies suggested that accessibility and user experience are fundamental concepts for interventions with people who have ASD, these aspects were not treated with the importance that they should be.

Several of the studies that were reviewed from the pool of articles reported having used and/or considered user experience and/or accessibility, but most of these studies did not provide enough detail about the use of these concepts. Table A2 (available in Appendix A) shows a total of 23 studies that in some way used and/or provided "detail" on the use of these concepts in their research. We can see that the most recurrent terms used in the studies were user experience, usability, and accessibility.

For instance, many of the studies claimed to have focused on accessibility when developing touchscreen applications, such as [23,24,39,42,43]. However, the authors' affirmations were not supported by empirical evidence or other details.

On the other hand, other studies such as [27,33] proposed the evaluation of the usability and/or user experience of the systems in future works. De Los Rios [33] suggested evaluating the usability of the application based on eye tracking. Schmidt and Beck [27] proposed the use of eye-tracking, electroencephalogram (EEG) scanning, and focus group interviews to evaluate the usability of the system.

Studies such as [28,40,42,44,45] aimed to evaluate usability and user experience based on post-intervention questionnaires with users, as well as with the people around them (such as their teachers or parents). These studies worked with control and test groups of children with and without ASD. Few studies indicated the number of subjects involved in the experiments: 14 in [42], 11 in [28], and 30 in [40]. Forty teachers were also involved in the experiment described in [42]. In addition to the questionnaires, Santarosa and Conforto [45] and Backman et al. [40] carried out methods such as focus groups in their interventions to be able to evaluate the usability and user experience.

Additionally, in studies such as those by Khowaja and Salim [46] and Naziatul et al. [47], the proposed systems were evaluated based on heuristic evaluations. In these studies, the authors adapted the heuristics proposed by Nielsen [48] to the contexts of their interventions. In both cases, three experienced evaluators assessed the system usability.

In addition, in the study by Vallefuoco et al. [49], a usability user test was carried out with 10 children aged between 5 and 12 under the methodology proposed by Moreno Ger [50] to evaluate the system, its usability, and the effectiveness of the customized elements developed to fulfill the objective of the study.

Finally, Caria et al. [22] worked with children with ASD between 16 and 22 years old, and Almeida et al. [51] worked with 40 children between 3 and 13 using the "System Usability Scale" (SUS) to evaluate the usability of their applications.

As we can see, few studies provided details about how they used concepts such as usability, user experience and accessibility, how these concepts were evaluated, and what kind of users were involved in their experiments. We think that it is important to consider all these concepts when developing new solutions.

#### **RQ3. Which game elements are considered when using gamification or serious games in the education of people with autism spectrum disorder?**

Several of the identified studies described the use of game-based learning (mostly serious games), but they did not specify and/or provide details about the elements of the games that were used. However, a significant number of studies explicitly presented some game elements that allow these systems to be more attractive and engaging for users. In Table A3 (available in Appendix A), we can see the game elements used in the studies, where the most frequent elements were points, levels, and rewards. Brief definitions of the game elements, as presented by Werbach [10], are presented in Section 2.5.

For example, Vallefuoco et al. [49] analyzed a serious game that focused on improving math skills in children with ASD and for which one of the main elements was feedback. Likewise, Sorce et al. [18] used avatars in an application with Kinect to foster the interest of participants with ASD in digital representations of works of art, paintings, and sculptures. In addition, Romero [36] carried out a computer-based intervention with intrinsic rewards and points to teach the recognition of emotions. Similarly, Chen et al. [52] designed and developed a computer game with points and rewards to develop and evaluate emotional skills and conceptual comprehension skills (such as recognizing fruits) in children with autism spectrum disorder. Additionally, Harrold et al. [53] added to the concepts described above through the use of levels in CopyMe, a serious game for iPad, which provides children with ASD with a means to learn emotions through observation and mimics. In the same way, Sturm et al. [42] used stories in addition to rewards, points, and levels in a game with Kinect technology that aims to promote the recognition of emotions and encourage collaboration between people with ASD and their peers. Finally, Boyd et al. [38] described the use of Zody, as a collaborative

assistance application, to teach social relations to children with ASD through the use of collaboration, points, levels, and rewards.

Most of the studies considered in this review did not explicitly identify which game elements they used in the development of their solutions. Even when they did, they did not give enough details on the effectiveness of the specific game elements. Although some authors claimed that their users were more engaged with the solutions they proposed, they did not provide empirical evidence to support such claims.

#### **5. Conclusions and Future Work**

Our systematic literature review focuses on analyzing the impact of technology on people with autism spectrum disorder based on research published during the last 10 years and available on the relevant scientific databases. The analysis shows an increase in the papers published on this topic over the years, which indicates an increasing research interest in the area. Interestingly, the highest percentage of the papers presented are case studies (74%). The studies were categorized into four categories: Conceptual Skills, Practical Skills, Social Skills, and General Skills. Studies that focus on Social Skills are predominant (36.17%).

Regarding RQ1, we observe that new research has focused on supporting children with ASD by using technologies such as virtual reality, augmented reality, virtual agents, sensors, and geolocation through educational games. These studies emphasize teaching different skills to people with ASD in educational contexts, with a higher percentage of studies focusing on Social Skills (36.17%) than on Conceptual (25.53%) or Practical Skills (8.51%), which shows a need for more research and development of new solutions for teaching such important topics. Exploring these alternatives and expanding the technological solutions to teach skills to people with ASD seem to be promising research topics.

The results related to RQ2 show that several studies mention that aspects such as user experience, usability, and accessibility are crucial when working with people with ASD. However, these aspects are usually not considered or validated in detail. Although the use of new technologies, such as EEG scanning and eye tracking in [27], to evaluate the usability of their systems is indeed interesting, studies have shown that brain activity may be negatively correlated with the Asperger questionnaire [54] and may be weaker for individuals with ASD when observing other people's actions [55]. Future studies should be careful with the use of such technological approaches, as brain activity may be misleading when working with people with ASD, especially in tasks that require recognizing emotions from facial expressions or movements. We believe that user experience is important and that future studies should consider accessibility and usability tests to ensure positive experiences and comfort with the use of their solutions, as there is a lack of research that applies these concepts correctly and that provides details about the user groups that participate in interventions.

Regarding RQ3, we have observed in the literature that game elements are a good way to engage users with learning and enhance the effectiveness of teaching approaches for people with ASD, but our findings show that there is a lack of evidence about the effect of the use of game elements in gamification, e-learning, and serious game solutions. We believe that future studies should consider and validate the use of game elements. Werbach [10] highlighted that game elements are effective, have a positive relation with users' engagement, and have been widely used with promising results.

We think that the use of technologies in conjunction with suitable game elements and user experience and accessibility design and evaluation are promising research topics related to teaching people with ASD.

**Author Contributions:** Conceptualization, K.V.; methodology, K.V. and C.R.; validation, K.V., C.R., D.Q., and E.J.; formal analysis, K.V. and C.R.; investigation, K.V.; resources, K.V. and E.J.; writing—original draft preparation, K.V.; writing—review and editing, C.R., D.Q., and E.J.; visualization, K.V.; supervision, C.R.

**Funding:** Katherine Valencia is a beneficiary of the CONICYT PhD Scholarship in Chile 2019, number: 21191170.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**


**Table A1.** Identified Learning Topics.


**Table A1.** *Cont.*


**Table A1.** *Cont.*

**Table A2.** Identified User Experience Concepts.


**Table A2.** *Cont.*


**Table A3.** Identified Game Elements.


#### **References**


© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

### *Article* **Connected Bike-smart IoT-based Cycling Training Solution**

#### **George Catargiu 1, Eva-H. Dulf 1,2,\* and Liviu C. Miclea <sup>1</sup>**


Received: 17 February 2020; Accepted: 6 March 2020; Published: 7 March 2020

**Abstract:** The Connected Bike project combines several technologies, both hardware and software, to provide cycling enthusiasts with a modern alternative solution for training. Therefore, a trainer can monitor online through a Web Application some of the important parameters for training, more specifically the speed, cadence and power generated by the cyclist. Also, the trainer can see at every moment where the rider is with the aid of a GPS module. The system is built out of both hardware and software components. The hardware is in charge of collecting, scaling, converting and sending data from sensors. On the software side, there is the server, which consists of the Back-End and the MQTT (Message Queues Telemetry Transport) Broker, as well as the Front-End of the Web Application that displays and manages data as well as collaboration between cyclists and trainers. Finally, there is the Android Application that acts like a remote command for the hardware module on the bike, giving the rider control over how and when the ride is monitored.

**Keywords:** connected bike; smart technologies; IoT; personalized training; embedded; back-end; front-end; Android; modules; MQTT; monitoring

#### **1. Introduction**

As times change, technology evolves and people migrate from fieldwork to deskwork, a new health threat arises: the sedentary lifestyle. To fight this phenomenon, fitness seems like a handy solution, especially for the advantage of having a personal trainer that can monitor the activity and offer guidance throughout the training. The electric bicycle is a trendy solution, as it is presented in [1]. However, most cyclists either amateur or professionals do not have this opportunity, making it harder to improve and reach specific goals. This paper proposes a technical solution meant to overcome this issue, by using sensors, microcontrollers and microcomputers to read, send, save, and display data. Therefore, any trainer can monitor relevant cycling data gathered from the bike in real time.

Data collection from sensors is widely used today. However, all the data are focused on sharing system development [2–4], sensor network implementation [5,6], safety [7], smart city development [8,9] or intelligent control implementation [10–12]. A series of great results are published considering the positioning method of the sensor system, structural health monitoring, and intelligent vehicle [13–17]. No research focusing on IoT (Internet of Thinks) and smart bike fusion for training has been published.

Nowadays, in cycling, the most common training method is by sheet. This sheet can be custom made by a trainer for one or more riders and holds a training plan by days, weeks and even season periods. However, this old and rigid method has a higher chance of failing to reach its purpose.

The Connected Bike proposed in this paper retrieves and transmits over internet the speed, cadence and cyclist power output both to a server-hosted database and to a web application, from where a trainer can monitor, evaluate and notify possible errors made by the cyclist during the ride. Furthermore, by persisting data to a server, both the cyclist and the trainer can return over past rides and discuss about them. The rider also has the possibility to control the system by starting, pausing or ending the ride with the help of a mobile application and internet connection.

All the technologies used in the presented Connected Bike project serve the following purposes:


The idea of real time bicycle tracking has been around for a while, and Chris Kiefer and Frauke Behrendt tackled this in [18] with a smart e-bike monitoring system (SEMS). They use custom hardware and a mobile phone with internet connection as an access point to monitor data like the bike's current location and the motor's level of assistance. This project is aimed towards e-bicycle fleets, focusing on scalability and ease of use, thus leaving aside acquisition of training relevant data like speed or cadence. A significant drawback for SEMS is that the battery powering the system discharges quite fast because of the hardware and connected mobile phone.

Neil Savage describes in [19] a bicycle data-monitoring concept named Copenhagen Wheel, by replacing the rear wheel with an intelligent one. This wheel has a modified hub, which encapsulates in addition to a standard one, an electric motor, and an ensemble of electronic components, both powered by a battery. The system is capable of changing gears, adapting the motor's speed and monitoring the torque, all while gathering pollution, humidity, temperature data, and sending it to a server through a mobile phone connected to the internet.

In terms of gathering data, which is a key concept of this research, Maurizio Di Paolo Emilio presented in [20] the main concepts involved in the process. Therefore, he described a data acquisition system as a mean of collecting information about a physical phenomenon through sensors and other electronic components. The sensors can be both digital (1/0, ON/OFF, etc.) and analog, the latter being harder to quantify and needs additional sampling and conversion circuitry. Any data acquisition system must have a computer or microcontroller as a core in charge of processing data and a mean of transportation, storage and display for this data.

In [21], Sarah Al-Mutlaq briefly presented the working principle of a load cell. These sensors translate the load or force acting on it into an electric signal and can be of three types:


All these ideas are embedded in the present paper, leading to an easy-to-use smart bike for personalized training. The novelty of the paper consists in the design and implementation of this smart connected bike, using the interaction of future-oriented mobility and state-of-art data technology, but cheap and accessible elements. All steps are described in detail to be easy to reproduce by the reader. The presented solution has the potential to disrupt existing training solutions.

The paper is structured in three parts. After this short introductory part, Section 2 reveals the used materials and methods, divided in 12 subsections, for each discussed module. Section 3 presents the obtained results. The work ends with concluding remarks.

#### **2. Materials and Methods**

The interface for monitoring the cyclist's activity must have an increased portability and must be easy to access and interact. To meet these criteria, a web application was developed. This can be accessed by trainers, as well as by cyclists, having pages and functionalities tailored for the specific user type.

In the web application, a trainer has access to a list of the cyclists to whom they have connected, can select a rider for real time monitoring, or analyze any of the previous rides. They can also stop the collaboration with any rider or establish a new collaboration with another. Last, but not least, the user has access to a page from where they can view and modify their personal account's data. The cyclist user cannot see their performance in real time, having only access to the previous rides. However, they have the possibility to control the monitoring by creating a new ride, pausing or ending the current one, through the help of a mobile phone application that acts as a remote for the module on the bicycle.

From the hardware point of view, the project consists of three modules: a microcontroller module, a microcomputer module and a mixed one. The first one is in charge of reading data from the load cell, measuring the force applied to the pedal and sending the data to the second module through a wireless channel.

The second module consists of a microcontroller, in charge of reading and processing data both from sensors and from the first module and a microcomputer that receives this data, converts it into a standardized message format, and sends it to the server via internet. It also retrieves the current location in terms of latitude and longitude with the help of a GPS module.

Last, but not least, the third module is an always-on microcomputer which hosts the back-end of the web application and the MQTT broker. Two rechargeable batteries power the first two modules, and the mixed module requires internet access that is provided either by a mobile hotspot or by an internet USB Stick.

Taking into account the multiple IoT specific architectures, described by P.P. Ray in [22] and presented shortly in the previous section, the most suitable one for this research proved to be a hybrid architecture, including the sensors, microcontrollers and the web and mobile applications.

As seen in Figure 1, the entire system is structured on three levels: the hardware level, the application level and the network level.

The hardware level consists of two modules that both have an Arduino microcontroller with separate power banks and have wireless communication between each other. The first module is based on an Arduino Micro, having connected an amplification module HX711 and a load cell. This module reads data from the load cell and sends it to the other module via infrared communication. The second module has an Arduino Mega with two Hall Effect sensors, used to determine the speed and cadence. The microcontroller performs data manipulation, afterwards converting it to byte array and sending it to the Raspberry Pi through USB communication. Here, the data is converted again from binary format and is aggregated with latitude and longitude readings from the GPS module. The final data format is a JavaScript Object Notation (JSON) that is periodically published to the server.

The application level includes three main components in charge of the system's business logic, establishing the way the application is handled and how data is stored and displayed. The first component is an Android App that is used to command the Raspberry Pi located on the bicycle, making it easy to manage the ride monitoring. The second component is the Raspberry Pi module placed on the bicycle that implements the system's logic in the form of a state machine, deciding based on the commands from the Android App if it should start reading, managing and sending data. Finally, the third component of this layer is the web application, where users can register, login and view different rides' data, or monitor a ride in real time.

**Figure 1.** Connected Bike system architecture.

The network level is made out of the actual communication between modules through internet. All the IoT-specific communication in this system is based on the MQTT, each component being connected through an Access Point as clients, except of a Raspberry Pi device, which is the actual server, hosting both the MQTT broker and the HTTP RESTful API server for the web application. There is also HTTP protocol used to establish communication between the front-end and the back-end for the web application.

Seen as a black box, the entire hardware scheme is described in Figure 2, making it easy to observe the way modules communicate with each other. The module blocks are highlighted with blue dotted lines. The communication between the Arduino Mega, Raspberry Pi and the GPS module is realized via USB (for the GPS module, an UART to USB converter was attached). Furthermore, because there are two rotating joints from the pedal to the bicycle's frame, making it harder to use wires, the infrared transmission medium was chosen for its simplicity and robustness. The downside of this approach was the addition of another microcontroller and an extra battery to power it.

**Figure 2.** The hardware scheme.

#### *2.1. The Pedal Module*

The main functionality of this module is reading the force applied to the pedal by a cyclist. This is possible thanks to a load cell, an amplifying driver, a microcontroller, and a Hall Effect sensor. The data is sent to the frame module with an infrared LED connected to the microcontroller. A rechargeable portable battery powers the entire module. The wiring diagram is shown in Figure 3.

**Figure 3.** Pedal module wiring diagram.

The microcontroller implements all the logic in this module, being responsible for reading, processing and sending data with a sample frequency of 80 Hz. Because of the fitting space and power limitations, an Arduino Micro was chosen for this module. The communication between the Arduino and the load cell driver is based on I2C (Inter-Integrated Circuit) using the pins 2 and 3 configured as serial data and serial clock. The Hall Effect sensor was connected to General-Purpose Input/Output (GPIO) pin 7, configured as an external interrupt pin and for the infrared LED, the fifth GPIO pin was used.

For this project, a rectangular load cell from SparkFun was used. Thanks to its rectangular form and small dimensions, it fitted directly on the pedal, receiving the full foot force from the cyclist. This load's range is between 0 and 50 kgf and measures with ± 0.5%RO accuracy.

Another particularity of this sensor, in contrast to its alternatives, is the number of wires. Most of the load cells provide four wires, making it easy to use one standalone sensor because it contains a full Wheatstone bridge made out of strain gauges. However, this particular load cell uses three wires and implements half of the bridge from two strain gauges of 1000 Ω each, connected in series.

The HX711 driver was developed for general interfacing between a microcontroller and a load cell, working with a voltage supply of 2.7 to 5 V and has a very simple working principle. Two pins are sending an excitation signal to two sides of the Wheatstone bridge (from the load cell) and two other pins are reading the voltage difference between the other two sides of the bridge. In free mode when no force is applied, the bridge is balanced, thus the difference is zero. When a force is applied onto the cell, the bridge becomes unbalanced and the voltage difference increases. The HX711 reads this difference in mV, amplifies it and maps it to digital values to be sent to the microcontroller.

In order to make this module work with only one three wire load cell, the bridge had to be completed with two 220 Ω static resistors, allowing the driver to read the voltage change from the sensor.

In order for the data to be received from the HX711 driver and processed by the Arduino Micro to reach the frame module, an infrared 5 mm LED was used. Because the pedal is continuously rotating and the pedal module is functioning as a standalone one, a Hall effect sensor was used to determine the position of the pedal arm relative to the bicycle's frame so the Arduino Micro gets notified when the pedal reaches a full circle and passes the magnet placed on the frame.

#### *2.2. Bike Frame Module*

This module is used to determine the speed, cadence and rider's power during the ride activity. All this data is serialized and sent via USB to the Raspberry Pi, which acts as a master of the entire system, but also as an internet gateway. Being also a standalone module, it is powered by a rechargeable battery. The wiring diagram is presented in Figure 4.

For data processing, an Arduino Mega 2560 was used, which communicates with the Raspberry Pi via USB. The 16 MHz processing frequency of the microcontroller allows implementation of multiple tasks in a Real Time Operating System (RTOS). The necessity of using an RTOS occurred from the need to read and compute sensor values with higher precision and data transmission.

To monitor the rider's position continuously, a GPS module from Adafruit was used. This device has a sensitivity of −165 dB, 10 readings per second, 66 channels, and the possibility to receive signals from 22 satellites with its own integrated antenna. The latitude and longitude are sent via a FTDI USB-TTL converter to the Raspberry Pi.

**Figure 4.** Bike frame module wiring diagram.

#### *2.3. Software*

This project was built using multiple programming languages and technologies, addressing different parts and modules.

The users can interact with the system using two applications: An Android app and a web app. Riders can only use the first application while both riders and trainers can use the second app. The user's use cases diagram is presented in Figure 5.

In the following sections, the software implementation is explained, alongside the used technologies for each module.

**Figure 5.** Connected bike use cases.

#### *2.4. The Pedal Software Module*

The main component for this module is an Arduino Micro, detailed in the previous hardware chapter. The open source libraries used to implement the software for this microcontroller are HX711.h and IRemote.h. The first one allows interfacing with the HX711 load cell driver while the second one was used for sending encoded message with the infrared LED. The setup function from the Arduino sketch initializes the pin connected to the IR LED and attaches a rising edge interrupt as well as an interrupt service routine to be dispatched on each event. Following this, a HX711 driver initialization and calibration sequence is executed.

Once the setup is complete, the loop function is called, reading the sensors in an infinite loop. The load cell reading is performed by the get\_units function from the library mentioned above. The returned value of this function is converted from lbs. to kg and scaled by 10 for a single decimal precision as seen in Equation (1).

$$\text{val}[\text{kg}] = (\text{val}[\text{lbs}] \cdot 0.453592) \cdot 0.4 \tag{1}$$

The readings are stored in a three values circular buffer holding the highest values read during a complete pedal cycle. Once the pedal reaches a full rotation and comes next to the magnet placed on the frame, the Hall Effect sensor triggers an interrupt on pin 7, therefore triggering the corresponding interrupt routine. This function computes the average between the values from the buffer, sets the transmission flag and clears the buffer. In the loop function, on each iteration, the transmission flag is checked and if it is set, then the previously computed average is serialized and sent via USB to the Raspberry Pi.

#### *2.5. The Frame Module*

The code for the Arduino Mega 2560 in this module is structured in multiple files with the .cpp and .h extensions. The software architecture for the frame module is presented in Figure 6.

**Figure 6.** Software architecture.

The code structure and the data manipulation follow a custom-made structure, following guides and patterns from an automotive. Thus, as seen in the above figure, the data transmission and the functionalities are decoupled, data travelling from one component to another through a virtual medium named the Runtime Environment, implementing an AUTOSAR-like (Automotive Open System Architecture) architecture, adapted to the application's context and simplified. The code is structured in four abstraction layers:


The initPinout and initTimers macros initialize the pins and timers used by the program and are placed in a separate header file called ucActuatorInit\_\_hh.h using the *#define* directive.

The IRrecv library for the infrared receiver is instantiated at the beginning and is activated by the enableIRIn function. After creating an instance, the object reads in the background the data from the KY-022 module and updates the global results variable, of type decode\_results.

In order to compute the speed and cadence, the timers four and five are used. When initialized, the timers are configured to use in normal mode and the capture mode is activated. This means that each timer will increment the TCNTn (Timer Counter Register n = 4, 5) starting from 0 to the maximum value of 65535, and once it reaches this value, the register is reset. Each timer has an allocated pin for the capture mode (pins 48 and 49). Thus, by enabling this mode, the pins' functionality is also changed, allowing them to trigger interrupts on events such as a Hall Effect sensor's impulse, capturing the TCNT's current value in a separate register ICR (Interrupt Capture Register), and an interrupt service routine is dispatched. The timer's working principle is presented in Figure 7, presenting a normal mode of operation and three interrupts on Interrupt Capture Pin.

**Figure 7.** Timer's working principle.

Both the timer 4 and timer 5 routines implement the same operations. First, the tick count is read, then using the tick count from the last interrupt, the elapsed ticks between two consecutive interrupts are determined, using Equation (2), and the result is stored in a circular 10 values buffer from the runtime environment.

$$t\_{timer} = i\_2(k) - i\_1(k-1) \tag{2}$$

where *k* refers to current ticks and (*k*−1) to previous ticks.

In the case where the current timer value reading is lower than the previous one, the tick count will be determined with Equation (3).

$$\mathbf{t}\_{\text{timer}} = 65535 - \mathbf{i}\_2(\mathbf{k} - 1) + \mathbf{i}\_3(\mathbf{k}) \tag{3}$$

The SensorsModule.cpp file holds the functions to calculate the speed, cadence and power. These functions take the value buffers as parameters and compute the average for the entire set. The value buffers from RTE are defined as volatile ensuring atomic access and data consistency. The speed computing function determines the bicycle's speed implementing as in Equation (4).

$$v\_{\text{wheel}} = \frac{\mathbb{C}\_{\text{wheel}}}{\text{t}} \cdot 3.6 \cdot 10 \tag{4}$$

where *vwheel* represents the actual wheel speed, *Cwheel* is the wheel circumference in [mm], and t is the time necessary for rotation in [ms]. The value 3.6 is the 1 m/s converted in 3.6 km/h, and the value 10 is the scaling preventing from working with float values.

Similar to the previous function, the cadence calculus follows Equation (5).

$$\text{Cadence} = \frac{\text{time}}{\frac{(\text{time} \text{Ticks} \text{Avg} \cdot \text{64})}{1000}} \tag{5}$$

where time is measured in [ms], while timerTicksAvg in [μs].

The power computing function, in contrast to the previous ones presented above, receives both the 10 values buffer holding the load cell readings and the current cadence, necessary to determine the angular speed used in the final formula. The instantaneous power is determined with Equation (6).

$$P\_{\text{predal}} = F\_{\text{predal}} \cdot \text{l} \cdot \Omega \tag{6}$$

where *Ppedal* represents the pedal power, *Ppedal* the pedal force, *l* is the crank leg length measured in [cm], and Ω is the angular speed.

At the microcontroller side, the implementation of the power computation is presented in Figure 8.

#### **Figure 8.** C code implementation of power computing.

The most important software part implemented on the Arduino Mega is the real time operating system. It completely replaces the implicit function loop with periodically executed tasks. At the core of this framework stands the task scheduler. This component dispatches tasks based on their period and priority.

For this project, the FreeRTOS library was used, developed as open source under the MIT license [23]. In order for the library to work on most microcontrollers available on the market, the timer used to determine when a task should be dispatched is the watchdog timer. In addition, because some of the microcontrollers including Arduino only have a single priority level, the library implemented an abstraction layer exposing four virtual priority levels.

In the ucActuator.ino file, inside the setup function, three tasks are created with different periods and priorities, using the xTaskCreate from the FreeRTOS library as follows:

	- 17.5 ms execution period
	- Highest priority
	- Continuously interrogates the infrared receiver and saves the values to the RTE buffer and performs the same for the pedal load cell
	- Operations implemented in this task have the shortest execution time
	- 175 ms execution period
	- Lowest priority
	- Checks if the timers used for speed and cadence overflowed more than twice, meaning the movement stopped and sets the corresponding values to 0
	- Calls the function that transmits the serialized data via USB to the Raspberry Pi
	- Data serialization to byte array and data transmission take the longest time

Due to the software priorities, the tasks are executed preemptively. Thus, the medium task is interrupted five times by the shortest task, while the longest task is interrupted 10 times by the short task and twice by the medium task. This allows for a so-called parallel execution on a single core processor given also the reasonable 16 MHz processor frequency of the Arduino Mega. A visual representation of tasks can be observed in Figure 9.

**Figure 9.** (**a**) Medium (purple) and short task (yellow); (**b**) Long (purple) and short task (yellow).

To facilitate code maintenance, a configuration header was created named ucActuator\_kh.h containing pin names and constant values used throughout code. In the event a pin or a parameter shall be changed, all the modifications can be performed into a single file.

#### *2.6. The Raspberry Pi Frame Module*

The Raspberry Pi from the frame module is in charge of processing data from Arduino Mega, structuring it into messages and sending them over the internet to the server and front-end for storing and visualization.

The software for the Raspberry Pi application is written in Python, and the communication wireless communication is performed through the MQTT (Message Queueing Telemetry Transport) protocol. As presented on the official web page [24], this protocol is a simple way of sending data between nodes as clients via TCP or Web sockets, used mainly in Internet of Things. The two main entities that are at the core of this protocol are the client and the broker. The latter manages the communication between clients that can publish messages to a topic or subscribe to a topic to consume messages. The Python library that was used to work with this protocol is Paho, which was developed as open source by the Eclipse foundation.

The code was structured by functionalities, in multiple files with the .py extension, also called Python modules. The entry point of the application is the piBrain.py module where two methods are called from the beginning. The first one blocks the program's execution waiting for an internet connection, returning when one is established. The next method is the main one, where the configuration file appconfig.cfg is parsed and the constants are retrieved. In addition, this method instantiates the MqttRemoteThread class and starts the thread.

The MqttRemote.py module contains the MqttRemoteThread class, which extends the Threading class, allowing it to run in a separate thread. The class implements a state machine, listens to commands coming from the Android App through MQTT, and based on them changes the state of the application. Therefore, the cyclist can start, pause or stop the ride using their mobile phone, the thread modifying the entire application's state. A visual representation of the state machine can be seen in Figure 10.

**Figure 10.** State machine of Raspberry Pi application.

When instantiating the MqttRemoteThread, a new MQTT client is also created that subscribes to the command topics that are sent from the Android App. Paho allows for attaching callback methods to events triggered on-message. The first method of such a kind is named on\_connect and is dispatched right after a new connection is established between the client and the broker. Here, the client subscribes to the specific topic. The second callback method is on\_message and it is called each time a new message is published to the topic the client is subscribed to. The library also provides a loop\_start method that creates a new non-blocking thread that will get messages by polling on the subscribed topics, calling the on\_message method once a new message is detected. In this application, the commands that the Android App can send are:

• "NEW"—creates a new ride;


A new ride is created by the attemptNewRide method, which performs five tries as POST requests to the server. If the server returns 201 success codes together with the newly created ride's primary key, then the program saves this id that is afterwards used to save the ride data to the corresponding ride id. If the server fails to create a new ride, then the application changes its state to "STOP", issuing an event through MQTT back to the Android App to inform the user about the failure.

The run method of the MqttRemoteThread class runs in an infinite loop and consists of if statements that check if a new command was received, changing the application's state accordingly. Once in a new state, the application will perform the required operation and then enter a listening state, ready to accept further commands. This thread is also responsible of sending information back to the Android App to inform the user about the current application's state.

When a user starts the application and sends the "START" command, the startApp method is called from the main thread, which starts three new threads and a thread safe queue object for passing data between them: serialReadThread, gpsPollingThread, mqttPublisherThread.

The SerialReadThread is implemented in the SerialReader.py module, which extends the Threading class and serves the purpose of reading and decoding the data received on the USB bus. On each loop iteration, if the application's state is "RUNNING", the serReader method is called to read six bytes from the input buffer. The raw data is converted into the actual values for speed, cadence and power, which in turn are formatted into a dictionary object that is added to the thread safe queue from where it can be read by other threads.

Inside the GpsPolling.py module there is the GpsPollingThread that is also a separate execution thread where values for latitude and longitude are read using the gps library for Python. The values are updated periodically and stored into a dictionary object, being available through the getLocation method.

Last but not least, the MqttPublisherThread from the MqttPublisher.py module is the execution thread where the final message is created and is sent over MQTT. The speed, cadence and power values as well as the latitude, longitude and the current date-time are retrieved from the thread safe queue object and put together in a JSON message. This message is sent to the cyclist-dependent topic. The way this application's classes interact is presented in Figure 11, which contains the UML classes diagram in Pynthon, used in Raspberry Pi.

In order to start the application at boot time, a Linux shell script was created called raspmodule.sh. Here, the Python App's main module, piBrain.py, is started using its own virtual environment, using the local interpreter and libraries. This shell script is triggered when the Raspberry Pi boots thanks the Cron utility available also on Debian that reads a crontab file where the script was added.

**Figure 11.** UML classes diagram.

#### *2.7. Android Application*

The Android Application has been designed to provide a user-friendly interface and usage. The sole purpose of this application is to inform the user about the state of the bicycle module and to allow them to interact with the system using the phone as a remote control to create a new ride, to stop or resume the current one.

In terms of technical implementation, this app includes two main activities, two classes and one interface. The first activity consists of a TextView that displays in the form of a label the name of the application "Connected Bike Trainer", two EditText elements that allow the user to enter the username and password. It should be noted that in this application, a user could only access with an account that was previously created in the web application. Thus, the last element is a login-only button that triggers the user authentication and if it proves successful, it creates the next activity.

The second activity consists of four buttons and three graphical elements of type TextView. Here, the user can interact with the application, can create a new ride, start, pause, resume, or end the current one.

Of the two classes, MqttHandler is responsible for sending commands and receiving status messages from the Raspberry Pi, via the MQTT communication protocol, while the Service class is responsible for communicating with the server via HTTP protocol.

When a user accesses the application, the first activity to be started is the Login one. Attached to the login button in this activity, is a method that will be called when the button is pressed. This method, attemptLogin, performs the necessary operations to validate the account data end and prevents sending erroneous data to the server such as blank credentials, or special characters. If the validation passes, the UserLoginTask class is instantiated and the execute method is called. This class, representing an asynchronous task, extends the AsyncTask class and contains four methods. The onPreExecute method is automatically called first and it makes the loading component available on the phone's screen. Next, the doInBackground method is called, which has the role of logging the user into the application asynchronously. This is the main component of this task, executing the instructions in a separate thread without blocking the main thread or the graphical user interface. Here, the RequestLogin method is called from the Services class, sending the username, password and an instance of the RestCallback as parameters. onSuccess, onFailure and onNotRider are the three interface methods that are asynchronously called based on the server's response. If the login I is successful, the current activity is replaced by the RiderActivity, which is the core of this application.

The Services class is utilized just by the login activity, implementing the RequestLogin and RequestUserData methods. The first one creates a JSON object with the user's credentials and makes a POST request to the server. The returned response is valid and successful if it has the "token" field, representing the authorization key. Moreover, if the logged-in user is a cyclist, the onSuccess method is called, otherwise the onNotRider is dispatched, displaying a message to inform the user that only cyclists can use this Android Application to control their rides. The second method performs a GET request to the server, using the token received from the previous request and the response is valid if it contains the "topic" field. The synchronization between the asynchronous tasks and the execution threads is done with a CountDownLatch instance that has also been configured with a 5 s timeout to prevent blocking connections.

The methods by which the user can interact with the bicycle system are implemented in RiderActivity. The four buttons available to the user have those methods attached in order to send messages and change the state of the frame module application, running on the Raspberry Pi. This is possible by calling the sendCommand method implemented in the MqttHandler class. Here, the command is converted to JSON and sent on the dedicated command topic to the frame module. Also, being a two-way communication between the Android App and the Raspberry Pi on the bicycle, in order to receive and display information about the state of the frame module, when the eclipse.paho library is initialized, event handlers are attached for events such as lost connection or successful message delivery. When a new message is received from the frame module, the messageArrived is called, retrieving it and sending it for displaying in the RiderActivity to inform the cyclist regarding the application's state. The MQTT connection to the broker is established in the constructor of the MqttHandler class. For a better understanding of the Android Application, a UML class diagram was developed, shown in Figure 12.

**Figure 12.** UML class diagram in Android application.

#### *2.8. The Server*

As most IoT systems, the Connected Bike project also has a server, which in this case consists of two main parts. The first part is the MQTT broker, which is necessary for the intermediation of messages between clients, while the second part is the Back-End for the web application, having both the role of persisting data transmitted from the bicycle and providing functionality to the Front-End.

Being a vital component in the operation of the entire system, the server must be switched on and have an uninterrupted Internet connection. The optimal solution was to use a Raspberry Pi 3 Model B+ to host the server for this project, being both low-cost and reliable. It is always powered on and connected to the internet, serving both the Back-End and the MQTT broker. In order to be accessible to the other modules, port forwarding rules were set on the router to which the Raspberry Pi is connected and the device received a static IP in the Local Area Network. Therefore, the other components of the system are able to access the services available on the server using the public IP of the router and the specified port. Next, are presented the two components that run on the server explaining them in more detail.

#### *2.9. Back-End*

The Back-End component is the one that directly accesses the database and implements the logic behind the web interface so that the data requested in the application is formatted and sent to the requester. In this project, the Back-End serves both the Front-End of the application and the Android Application. The communication protocol is HTTP, currently being one of the most used data transmission protocols over the internet.

The programming language in which the Back-End part was implemented is Python 3.6, using the open-source flask and flask-restful libraries. The design pattern used is Model-View-Controller (MVC), but only the Model and Controller are implemented on the Back-End side, because the Front-End is in charge of the View. In the development of the Back-End, the steps in the tutorial [25] have been followed and adapted.

The database used is PostgreSQL where all the users' data as well as the rides' data are stored. The tables and the relationships can be seen in Figure 13.

The database consists of two tables that maintain records of all the rides for each cyclist (ride\_table) and the data recorded for each ride (ride\_data). General user data is kept specific to all users, regardless of profile, such as username, password, and role. Because users may have different roles, namely cyclist or trainer, it means that the specific data for each category will be different. Therefore, the rider\_info table was created to keep specific information for cyclists such as name, weight, bicycle, and topic, the last one being used to send commands from the Android App to the rider's bike frame module. The trainer\_info keeps information about coaches such as name, discipline, level, and certificate.

Because a cyclist can have multiple trainers and a trainer can collaborate with multiple cyclists, there is a many-to-many relationship between the two entities. This was translated into a relationship table trainer\_to\_rider. This table keeps in each record the relation between a trainer and a cyclist through their primary keys.

**Figure 13.** Database structure.

The database was defined first by writing the tables and relationships as model classes in separate Python modules, all centralized in a package named models. After this, the database was created directly from the defined classes using the SQLAlchemy library with the Code-First approach. This library is an Object Relational Mapper (ORM) designed for Python that allows users to map database tables and relationships to classes and work with entities to perform CRUD operations (Create Read Update Delete). Everything from database access to creating queries are performed in the background, allowing the developer to write complex database operations that are translated into the database-specific queries.

The other Python package is named controller. Here, the Back-End is divided into two functionalities. The first one is dedicated to users. The UserView.py module defines all the methods that provide functionality for mobile and web applications using the table dedicated to users. These methods can create a new account, modify user data, and validate login credentials by querying the user, trainer\_info, rider\_info and trainer\_to\_rider tables with the help of SQLAlchemy. Similar to UserView.py, the RideView.py module contains all the methods dedicated to rides and rides data respectively, performing queries on the ride and ride\_data tables. All the functions described above have a decorator that specifies the URL path and the type of REST (Representational state transfer) method through which they can be accessed remotely. Also, some methods may have more decorators depending on the access level. For example, the delete\_ride method has both the *@Auth.auth\_required* decorator, which allows it to be called only if the request has a token corresponding to an authenticated user, and the *@Auth.rider\_required* decorator, which checks if the requester is a cyclist. These decorators are in fact other methods called implicitly to verify various requirements, such as whether the user is authenticated or has a certain role to access a resource. If the access criteria are satisfied, the decorated method will perform its task. All the decorators as well as the methods used to generate and decode JWT tokens are in the Authentication.py module. The encryption algorithm is SHA-256 and the Unicode standard is UTF-8.

In order for all the Back-End methods written in Python and described above to be remotely accessible over the Internet, a Web Server Gateway Interface (WSGI) is required. This mechanism handles all the HTTP requests coming from the Internet, routing them to the appropriate methods from the local machine. For this application, a WSGI server called Gunicorn was used, which is specifically designed to work with the Flask framework. It was configured to work as a daemon with three workers, so it did not interfere with the other running processes on the machine. These workers are actually threads in a pool, that are assigned per request to route to the corresponding method.

#### *2.10. MQTT Broker*

The communication between the MQTT clients was made possible by the Mosquitto broker, developed by Eclipse Foundation and released for public use for free. Once installed and configured, the broker starts as a background process when the Raspberry Pi finishes booting. The settings were configured in the mosquitto.conf file, enforcing the use of a username and password for all MQTT clients in this project for increased security. In addition, two listeners are configured here. The first uses the 1883 port for TCP/IP connection, which is used both for communication with the frame module and for sending commands and status information between the frame module and the Android Application. The second listener exposes the 8083 port for websockets connection, used by the Web Application for receiving online ride data from the frame module. Moreover, in this configuration file, the SSL certificates are also added for a more secure connection and the location where logs from Mosquitto broker will be stored.

#### *2.11. Other Components*

In addition to the Back-End and the MQTT broker, there are other auxiliary components that fulfill various roles on the Raspberry Pi server.

The first component is the Python service that subscribes to the data topic and retrieves ride data messages transmitted by the bicycle's frame module and saves it to the database based on the ride id from the message. This service started at the Raspberry Pi's boot time is waiting for the internet connection to be established, then it initializes the MQTT client and starts an infinite loop thread waiting for messages on the *connectedbike*/*data*/*#* topic. The hashtag character is a wildcard, meaning that all messages that are sent on the topic *connectedbike*/*data*/*{user\_id}* will be received, regardless of the user id. This is advantageous as the ride\_data table does not require a user id but only ride id, and this id is sent along with the data in the MQTT message.

Another important part of running the Back-End is the reverse proxy component. Its purpose is to hide implementation details and protect the Back-End component. All requests from clients are received by the reverse proxy that routes them to the WSGI and then routes back the response to the caller. For this project, NGINX was used as a reverse proxy, being a free and open-source software, supported by numerous contributors. After installation, it has been configured in the connectedbike.conf. Here were defined the access domains, the path to the Back-End, the files for storing logs, and the server that contains the parameters and the proxy and access control headers. Certbot certificates for SSL encryption are also configured here. Thus, all requests received on the public domain connectedbike.cf or www.connectedbike.cf are forwarded by the proxy to the Gunicorn WSGI serving the Back-End. The operation of this assembly can be seen in Figure 14.

After installation, the NGINX server automatically starts as a daemon service when the Raspberry Pi is turned on. Also, at the time of activation, NGINX takes over and sets all the configurations identified in the file with the .conf extension from the /etc/nginx/sites-available/ folder, in this case being connectedbike.conf.

The third server component is the Cron utility. This is a simple service offered on most Linux operating systems. To configure the crontab file, which the service uses as a look-up file for scheduling tasks, the *crontab -e* command line is called from the command line. For this application, a command is added to the crontab to run at boot time on the Raspberry Pi and to call the mqttdataposter.sh script. This shell script changes the current working directory to the root of the Application folder containing the MqttDataPoster.py. Afterwards, the Application is started from inside its Python virtual environment.

**Figure 14.** NGINX reverse proxy operation scheme.

Both the MQTT Mosquitto broker and the NGINX reverse proxy server are automatically started at boot time and run as background services. Gunicorn, however, does not offer this possibility, which is why a forth auxiliary component is required, namely the creation of a Systemd unit with the .service extension, that handles the activation of the Gunicorn server at boot time. Systemd is a utility offered within the Debian operating system that allows the creation of user-defined services, running in the background. The service created for Gunicorn is called connectedbike.service and consists of three parts.

The first part, defined by the [Unit] tag, contains a description of the service and specifies the dependencies required to run the service. In this case, they are network-online.target and network.target.

The second part represented by the [Service] tag specifies the details for starting the service. Here the following are defined:


The third section defined by the [Install] tag allows the service to be enabled or disabled. So, using the WantedBy=multi-user.target directive, a folder with the same name is created that persists even if the device is restarted and activates the service at boot time. In order for the service to be active and running every time the Raspberry Pi is started, the shell command *sudo systemctl enable connectedbike.service* is required. The same command, but replacing the enable keyword with start or stop, can be called to begin or terminate the service.

Last but not least, the Raspberry Pi server has allowed firewall access to the ports required by the application. Also, for a more efficient and user-friendly use, a free domain name (connectedbike.cf) has been assigned to the server's IP.

#### *2.12. Web Application*

In this project, the web application has the role of displaying in a clear and easy way the data read by the module on the bicycle, giving the trainer the possibility to monitor the evolution of a cyclist online. In addition, the application offers the possibility of both categories of users to analyze previous rides. Also from the app, users can register a new account, login or connect with each other for collaboration. To achieve this, the ReactJS Javascript framework, developed and released free by Facebook, was used.

The application uses multiple components and elements to achieve the desired graphical user interface and user experience. For an easier state management, the react-redux library was used. It allows for centralizing the state for the entire application and passing only parts of it to components that specifically need them. The global application state can be modified only by functions named reducers. They receive as a parameter the current state of the application and the action based on which the decision to change the global state is taken.

For the implementation of this application, the tutorial presented by Jason Watmore in [26] is followed. The structure is divided into pages, auxiliary components and Redux components. In total, there are five pages: LoginPage, RegisterPage, RiderPage, TrainerPage and MonitoringPage. These in turn contain sub-pages, components rendered in the main page, which change according to the selected tab in the left-side menu. This allows conditional display of several different information, using the same URL. The user interface (buttons, labels, graphic elements) was created with the help of the MaterialUI library, which offers a wide range of free graphic elements. The monitoring page is accessible by both the trainer and the cyclist, being the place where the data is displayed, whether they come online or are uploaded from the Back-End. When the monitoring page is rendered in the browser, a props check is performed. If a ride id is found, then a function is called to bring the data through a GET request to the server. In the absence of a ride id, riderTopic is received in the props and the component subscribes to this topic to receive the data from the bicycle frame module in real time. To receive data through MQTT, the paho-mqtt library for React is used and the protocol supported by JavaScript is Websockets, which is why the broker on the server exposes both TCP/IP connection port and a Websocket connection port.

Because the data from the bicycle is transmitted and records each second, the number of entries in the database increases drastically. Thus, for the data to be fully displayed and analyzed, without occupying a large portion of the graphical interface, the Highcharts library for React was used. Three Highcharts charts were configured with this, one for speed, one for cadence and one for power. To each chart was attached a Navigator element available in the library that allows selection of a portion from a large data set and the graphical visualization of that part only, thus facilitating a thorough study of the ride in more detail. Also on the monitoring page was added a map using the GoogleMaps library for React, that displays markers with latitude and longitude received from the GPS module on the bicycle.

The components folder also includes AlertBar.js, which is used to display labels with various user information about in-app events such as failed login or failed connection to Back-End. Also here have been defined a set of constants for enforcing private routes, respectively limiting access to the application for users who have not logged in. So, anyone can access the Login or Register page, but only users who have logged in and received an authorization token from the server can navigate through the rest of the application. In addition, these constants decide based on the response from the server whether the user is a cyclist or coach and redirects it to the appropriate page.

Finally, yet importantly, the Redux part is composed of three parts:


The engine behind these three parts is the store. It is created using the createStore function in the redux library. This function is called with a set of parameters such as persistedReducer or middleware. In order to maintain the entire state of the application, even after the user refreshes the browser page, the redux-persist library is used. With it, the entire status hierarchy of the application is saved in Local Storage as JSON, where it can be retrieved again after refreshing the web page.

In the application development stage, the Node.js server was used. It has the role of running a local instance (localhost) at a predefined port, in this case, port 3000, where the developer can observe the changes made to the code. To expose the application on the internet and for anyone to access it, the npm run build command was run, creating a new build folder containing all the files and packages needed to run the application in production. This build was uploaded for free to Netlify, making it accessible to everyone on the internet.

#### **3. Results**

The main distinguishing features of the system are presented in the following:


The pedal module, as mounted on the bicycle, can be seen in Figure 15.

**Figure 15.** The final pedal module.

Also, the frame module, consisting of Arduino Mega 2560, Raspberry Pi 3 Model B+, Hall sensors, infrared receiver, and GPS module from Adafruit is shown in Figure 16.

**Figure 16.** The resulted bicycle frame module.

In its final form, the hardware part of the frame module has been placed inside a bicycle water bottle and the connection with the sensors is made through an RS-232 connector as shown in Figure 17. This allows the bottle to be disconnected from the electronic components on the frame, leaving the sensors in place.

**Figure 17.** Final form of the Connected Bike

The Android App that allows the cyclist to manage the ride data monitoring is presented in Figure 18.


**Figure 18.** Android application interface

Last but not least, the web application by which users can create their account, monitor the rides or create connections with each other is shown in Figure 19.

**Figure 19.** Web application of the Connected Bike

Finally, using the above modules, both hardware and software, the following requirements were achieved:


#### **4. Conclusions**

This connected bike project was born out of a real issue encountered among both amateur and professional cyclists. This problem consists of the limitations existing in the current training methods. In this project, a wide range of technologies have been used, starting from the electronics and hardware side and to the web and mobile applications. Putting all these together, a prototype of a complete solution has been created to help cyclists and facilitate trainers.

**Author Contributions:** Conceptualization and methodology, G.C.; software, G.C.; validation, E.-H.D.; formal analysis, L.C.M.; writing—original draft preparation, G.C.; writing—review and editing, E.-H.D.; supervision, E.-H.D.; funding acquisition, L.C.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by a grant of the Romanian Ministery of Research and Innovation, CCCDI – UEFISCDI, project number PN-III-P1-1.2-PCCDI-2017-0734 / ROBIN – "Robot,ii s,i Societatea: Sisteme Cognitive pentru Robot,i Personali s,i Vehicule Autonome", within PNCDI III. E.H.D. was funded by Hungarian Academy of Science, Janos Bolyai Grant (BO/ 00313/17) and the ÚNKP-19-4-OE-64 New National Excellence Program of the Ministry for Innovation and Technology.

**Conflicts of Interest:** The authors declare no conflict of interest

#### **Abbreviations**


#### **References**


International Conference on Mobile Computing and Networking, Los Cabos, Mexico, 21–25 October 2019; ACM Digital Library, Article No.: 60. pp. 1–3. [CrossRef]


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **High Precision Positioning with Multi-Camera Setups: Adaptive Kalman Fusion Algorithm for Fiducial Markers**

**Dragos Constantin Popescu 1,2,\*, Ioan Dumitrache 1,3, Simona Iuliana Caramihai <sup>1</sup> and Mihail Octavian Cernaianu 2,\***


Received: 31 March 2020; Accepted: 7 May 2020; Published: 11 May 2020

**Abstract:** The paper addresses the problem of fusing the measurements from multiple cameras in order to estimate the position of fiducial markers. The objectives are to increase the precision and to extend the working area of the system. The proposed fusion method employs an adaptive Kalman algorithm which is used for calibrating the setup of cameras as well as for estimating the pose of the marker. Special measures are taken in order to mitigate the effect of the measurement noise. The proposed method is further tested in different scenarios using a Monte Carlo simulation, whose qualitative precision results are determined and compared. The solution is designed for specific positioning and alignment tasks in physics experiments, but also, has a degree of generality that makes it suitable for a wider range of applications.

**Keywords:** critical infrastructures; positioning system; optical measurements; fiducial markers; adaptive kalman; measurement fusion

#### **1. Introduction**

The evolution of technology has led to an increasing demand for solving complex problems which may be viewed as attempts to control and direct system behaviours towards desired states. The inherent complexity of problems and processes requires new approaches both in system modelling and in defining the emergent interaction with a highly dynamical and sparsely defined environment.

Cognitive approaches are successfully used in contexts where the boundary between the systems and the environment is fuzzy. However, they exhibit strong interrelation and interconnection, assisted by specific perception mechanisms [1]. Advances in complex control applications can be achieved only by considering adequate design approaches for sensory systems, especially in domains like environmental applications [2], health [3], industrial control, agriculture, etc.

Although new technologies such as wireless sensor networks or Internet of Things (IoT) are providing valuable solutions for appropriate perception mechanisms, complex issues are raised with the inclusion of data fusion, reliability, flexibility, reconfigurability, and cost of the measurement system. If some of these problems can be addressed during the modelling process, there are others (e.g., sensor positioning and sensor dynamics) that have a bigger impact on the generality of the overall solution.

These issues are specific in critical infrastructures, research institutes, power systems, e-health, mining, and traffic control, where the multitude of concurrent dynamics generating a large amount

of information requires adequate solving methods that cover aspects such as relevance, efficiency, and cost optimisation.

This paper follows the development of a measurement method used in a very precise and yet flexible and portable positioning system. Firstly, the method is an answer to a specific application: Automatic alignment with high precision of various instruments which have to be remotely operated in highly flexible, open, and dynamic configurations for physics experiments. However, it can be used in any application where high precision positioning over a large working area is required, where no absolute reference can be defined, or when it is necessary to simultaneously align multiple features in relative positions, with high reliability. Such applications include large scale construction sites where multiple components need to be positioned in precise locations, manufacturing facilities where large machines are assembled or spacecraft docking in zero gravity environments [4].

The article has the following structure. The specific application and context addressed in this work are presented in Section 2. The theoretical context behind the proposed algorithm is described in Section 3. It it then used to build two essential procedures for the positioning system. The precision of the proposed method is assessed in the Section 4 using a Monte Carlo simulation. The conclusions and future developments are detailed in Section 5.

#### **2. Description of the Initial Problem**

The large number of high repetition rate, ultrashort pulse, and high power laser facilities that will come online all around the world will require state-of-the-art tools to allow the harnessing of their full potential [5]. The high power lasers of the ELI-NP (Extreme Light Infrastructure-Nuclear Physics) user facility will be employed for a wide range of research topics like studies of nuclear induced reactions, dark matter search, material irradiation, or medical applications [6]. The development of the experimental setups for studying these topics requires specific instrumentation, while also having strict needs in terms of positioning and alignment, in order to ensure optimal experimental conditions.

As the setups are continuously changing, absolute position referencing is hard to achieve. This is a necessity, as multiple instruments need to be precisely positioned relative to each other during the experiment. Figure 1 displays an example of a setup for solid target alignment in which multiple optical diagnostics are positioned using motorised manipulators that have 3 to 6 degrees of freedom (DOF).

**Figure 1.** Solid target alignment setup example where multiple instruments are positioned using motorised manipulators.

Apart from the precision requirements, additional ones should be taken into consideration. The alignment should be done remotely because the experiments take place inside vacuum chambers and behind concrete walls used for radiation shielding. Moreover, to take advantage of the high

repetition rate of the lasers and to maximise the beam-time available for the users, the positioning has to be performed in a limited amount time. In a previous work [7], we addressed this series of requirements and constraints by developing an automatic alignment system that is based on relative position measurements using imaging cameras and compact and flat fiducial markers, due to space and visibility constraints. The alignment algorithm was based on a real-time optimisation procedure which is the subject of a patent [8].

Although the fiducial markers were initially developed and used in augmented reality applications [9], due to their versatility in determining their position in a non-invasive manner (by imaging them with a camera), they were quickly adopted for a different range of applications. These include kinematic calibration and visual servoing for industrial robots [10,11], robot localisation and navigation [12–14], SLAM (simultaneous localisation and mapping) [15–17] and sensor fusion [18–20].

The motivation behind this work is to combine the measurements from multiple cameras in order to increase the working area of the system and to maximise the positioning precision. Simultaneously, the main focus is to present and test the precision of the proposed methods and algorithm, comparing the results with those recorded while using only one camera [21].

#### **3. Method**

The basic approach behind the proposed method is to use multiple cameras in order to improve the precision of estimating the pose of fiducial markers and to extend the working area of the system. It begins with the assumption that each camera provides measurements that are erroneous and noisy. The problem can be conceived through an analogy with the GPS system where distance measurements from multiple satellites are used to estimate the position of the receiver module with high precision. The key ingredient in the GPS system is to know very precisely the position of the satellites. The proposed positioning system needs to meet the same requirement for its cameras, but wasting too much effort on this task diminishes its practicality. Consequently it is only assumed that the cameras have unknown but fixed positions.

A related approach that has similar objectives can be found in [22]. The main benefit of the method is the possibility to calibrate the setup of cameras using a 3D feature with fiducial markers having unknown configuration. However, it is not meant for estimating the pose of single markers. In order to achieve this, additional methods are required.

Our method is developed using the ArUco fiducial markers [23,24]. The ArUco library detects and estimates the pose of the marker with respect to the camera using the solvepnp algorithm in which the pinhole camera model is data fitted using a Levenberg–Marquardt non-linear optimisation procedure. The output is represented by the extrinsic camera parameters which can be expressed in terms of a homogenous transformation *T<sup>M</sup> <sup>C</sup>* (the transformation between the camera attached reference frame and the marker attached one) detailed in [7].

#### *3.1. Adaptive Kalman Fusion Algorithm for Multiple Cameras and Fiducial Markers*

The method consists of fusing noisy measurements from multiple cameras, in order to improve the precision of estimating the exact position of the fiducial marker. In the scientific literature, multiple sensor fusion and noise filtering methods have been developed over the years. Among them there is the Kalman filter [25], which is the most widely used. The noise effect is mitigated by using a dynamical model for the physical process involved and a statistical model (covariance matrix) for the noise in the measurement process.

For a discrete linear state-space dynamical model in Equation (1), the Kalman filter estimates the value of the state vector *x* using noisy measurements for the input *u* and the output *y*.

$$\begin{cases} \mathbf{x}\_k = A \cdot \mathbf{x}\_{k-1} + B \cdot \mathbf{u}\_{k-1} + q\_{k-1} \\ y\_k = \mathbb{C} \cdot \mathbf{x}\_k + r\_k \end{cases} \tag{1}$$

In standard data fusion applications that use the Kalman filter, the pose of various objects is estimated using the input from accelerometer, gyroscope, and magnetometer sensors [26,27] and the output from distance measurement devices [28,29]. Our application makes impossible to use any type of attached sensors and, hence, we only rely on the "non-invasive" pose measurement using the solvepnp algorithm and imaging cameras.

Our approach is to employ a state-space model with free dynamics (where *u* is zero) and with an identity state matrix (*A*). In this respect, each measurement is made using only one snapshot image from all the cameras that are synchronised with the help of an electrical trigger signal. In this way, the position of the fiducial marker during the measurement is considered fixed. On the same set of images, the solvepnp algorithm is applied multiple times and, thus, the evolution of the measurement is caused only by the noise and not by the movement of the fiducial marker. Furthermore, the measurements are used to iteratively estimate the real position using the proposed algorithm.

In order to improve the results, we propose a procedure designed to update the statistical model of the noise. The *Q* and *R* covariance matrices (which are discussed below) are continuously adjusted using the newly acquired data. Any change of the noise behaviour and any existing correlations are captured and thus, the effect of the noise is mitigated more efficiently. Consequently, the proposed algorithm is an adaptive Kalman version.

The homogeneous transformation representation (*T<sup>M</sup> <sup>C</sup>* ) has numerous practical benefits, especially for pose composition and inversion operations, but it is not suitable in this circumstance because it is redundant (16 numerical values for expressing a 6 DOF pose). The solution is to use an equivalent representation which is composed of the set of translation coordinates (*X*, *Y*, and *Z*) and the set of Euler angles (*AX*, *AY*, and *AZ*).

In this particular case, the structure of the state vector is the real pose of the fiducial marker defined in Equation (2), where the subscript *k* denotes the present discrete-time sample of the state vector.

$$\mathbf{x}\_k = \begin{bmatrix} X\_k \\ Y\_k \\ Z\_k \\ AX\_k \\ AY\_k \\ AZ\_k \end{bmatrix}. \tag{2}$$

The output vector is composed of the pose elements measured using all the cameras available. The structure of the output vector is defined in Equation (3), where the superscript *i* is indicating the index of the camera considered. .

$$y\_k = \begin{bmatrix} X\_k^1 \\ Y\_k^1 \\ Z\_k^1 \\ AX\_k^1 \\ AY\_k^1 \\ AZ\_k^1 \\ \vdots \\ AZ\_k^1 \end{bmatrix} \qquad \begin{array}{c} \vdots \\ X\_k^i \\ Y\_k^i \\ Z\_k^i \\ AX\_k^i \\ AY\_k^i \\ AZ\_k^i \\ \vdots \end{array} \tag{3}$$

The discrete-time state-space model considered is defined in Equation (4), where *I*<sup>6</sup> is the identity matrix of rank 6, *qk*−<sup>1</sup> is a random noise signal with normal distribution (white noise) that is quantifying the false evolution induced by the noisy measurement, the *H* is the measurement model matrix defined in Equation (5), and *rk* is the measurement noise also considered white noise.

$$\begin{cases} \mathbf{x}\_k = I\_6 \cdot \mathbf{x}\_{k-1} + q\_{k-1} \\ y\_k = H \cdot \mathbf{x}\_k + r\_k \end{cases} \tag{4}$$

$$H = \begin{bmatrix} I\_6^1 \\ \vdots \\ I\_6^i \\ \vdots \end{bmatrix}. \tag{5}$$

The equations that describe the Kalman filter are presented in Equation (6). The first two equations give a rough state estimate using the dynamical model while the last 5 equations are used for improving the estimate using the newly acquired output sample. *P* is a covariance matrix that expresses the confidence degree of the state estimation, which is updated during the algorithm iterations, *Q* and *R* are the covariance matrices associated with the noises *q* and *r* respectively, *vk* is the rough estimation error (difference between the measured output and the predicted one using the first two equations), *Sk* is the covariance matrix associated with the predicted output, and *Kk* is the Kalman state update.

#### **Predict the state**

1. *<sup>x</sup>*ˆ*k*|*k*−<sup>1</sup> = *<sup>A</sup>* · *<sup>x</sup>*ˆ*k*−1|*k*−<sup>1</sup> 2. *Pk*|*k*−<sup>1</sup> <sup>=</sup> *<sup>A</sup>* · *Pk*−1|*k*−<sup>1</sup> · *<sup>A</sup><sup>T</sup>* <sup>+</sup> *Qk*−<sup>1</sup>

#### **Update the prediction**

3. *vk* = *yk* − *Hk* · *<sup>x</sup>*ˆ*k*|*k*−<sup>1</sup> 4. *Sk* <sup>=</sup> *Hk* · *Pk*|*k*−<sup>1</sup> · *<sup>H</sup><sup>T</sup> <sup>k</sup>* + *Rk* 5. *Kk* <sup>=</sup> *Pk*|*k*−<sup>1</sup> · *<sup>H</sup><sup>T</sup> <sup>k</sup>* · *<sup>S</sup>*−<sup>1</sup> *k* 6. *<sup>x</sup>*ˆ*k*|*<sup>k</sup>* = *<sup>x</sup>*ˆ*k*|*k*−<sup>1</sup> + *Kk* · *vk* 7. *Pk*|*<sup>k</sup>* <sup>=</sup> *Pk*|*k*−<sup>1</sup> <sup>−</sup> *Kk* · *Sk* · *<sup>K</sup><sup>T</sup> k*

(6)

The filter requires good estimates for the initial state *x*<sup>0</sup> and the covariance matrices *Q* and *R*. In the proposed algorithm this is achieved using an initialisation procedure. The position of the fiducial marker is measured for *Ni* number of sampling times. The result is the set of samples defined in Equation (3) for *k* = 1, ... , *Ni* (the length of the initialisation) and *i* = 1, ... , *n* (the number of cameras available).

Averaging the samples gives a good estimate for the initial state, which is built according to Equation (7), where *E*[·] is the mean operator (expected value). The *Q* and *R* matrices are computed from the same set of samples, assuming that there is no correlation between the noises affecting the measurements.

$$\mathbf{x}\_{0} = \begin{bmatrix} X\_{0} \\ \begin{bmatrix} X\_{0} \\ Y\_{0} \\ \end{bmatrix} & \begin{bmatrix} X\_{0} \\ \end{bmatrix} \\ \begin{bmatrix} X\_{0} \\ Y\_{0} \\ Z\_{0} \\ \end{bmatrix} & \begin{bmatrix} E\_{0} \\ \end{bmatrix} \\ AX\_{0} \\ AX\_{0} \\ \begin{bmatrix} AY\_{0} \\ AZ\_{0} \\ \end{bmatrix} & \begin{bmatrix} E\_{0} \\ \end{bmatrix} \\ AX\_{0} \\ AZ\_{0} \\ \end{bmatrix} & \begin{bmatrix} AX\_{k}^{i} \\ \end{bmatrix} \\ \begin{bmatrix} E\_{0} \\ AZ\_{0} \\ \end{bmatrix} & \begin{bmatrix} E\_{0} \\ \end{bmatrix} \\ \begin{bmatrix} E\_{0} \\ 1 \le i \le n \\ \end{bmatrix} \\ \begin{bmatrix} E\_{0} \\ 1 \le i \le n \\ \end{bmatrix} & \begin{bmatrix} AZ\_{k}^{i} \\ 1 \le i \le n \\ \end{bmatrix} \end{bmatrix}$$

(7)

$$Q\_0 = \operatorname{diag} \left( \underset{1 \le i \le n}{\operatorname{var}} \left[ \underset{1 \le k \le N\_i}{\operatorname{var}} \left( \mathbf{X}\_k^i \right) \right], \underset{1 \le i \le n}{\operatorname{var}} \left[ \underset{1 \le k \le N\_i}{\operatorname{var}} \left( Y\_k^i \right) \right], \underset{1 \le i \le n}{\operatorname{E}} \left[ \underset{1 \le k \le N\_i}{\operatorname{var}} \left( Z\_k^i \right) \right] \right),$$

$$\underset{1 \le i \le n}{\operatorname{E}} \left[ \underset{1 \le k \le N\_i}{\operatorname{var}} \left( AX\_k^i \right) \right], \underset{1 \le i \le n}{\operatorname{E}} \left[ \underset{1 \le k \le N\_i}{\operatorname{var}} \left( AY\_k^i \right) \right], \underset{1 \le i \le n}{\operatorname{E}} \left[ \underset{1 \le k \le N\_i}{\operatorname{var}} \left( AZ\_k^i \right) \right] \right) \tag{8}$$

$$R\_0 = \operatorname{diag} \left( R\_0^1, \dots, R\_0^i, \dots, R\_0^n \right) \tag{9}$$

$$R\_0^i = \text{diag}\left(\max\_{1 \le k \le N\_i} \left(X\_k^i\right)\_{\substack{1 \le k \le N\_i \end{pmatrix}}, \max\_{1 \le k \le N\_i} \left(Y\_k^i\right)\_{\substack{1 \le k \le N\_i \end{pmatrix}},$$

$$\max\_{1 \le k \le N\_i} \left(AX\_k^i\right)\_{\substack{n \le x \le N\_i \\ 1 \le k \le N\_i}} \left(AY\_k^i\right)\_{\substack{1 \le k \le N\_i \\ 1 \le k \le N\_i}} \left(AY\_k^i\right)\_{\substack{1 \le k \le N\_i \\ 1 \le k \le N\_i}} \left(AZ\_k^i\right). \tag{10}$$

*Q* is a diagonal matrix defined in Equation (8) built using the mean variance of the pose elements (*X*, *Y*, *Z*, *AX*, *AY*, and *AZ*) along all the cameras. The *R* matrix is defined in Equation (9) where *R<sup>i</sup>* is defined in Equation (10), built using the variance for each pose element measured by each camera.

After the initialisation is finished, the Kalman algorithm is iterated for *Ne* number of sampling times, while at each step, a new set of samples *yk* in the form of Equation (3) is measured and an estimated state trajectory is built (*x*ˆ*k*|*<sup>k</sup>* for *<sup>k</sup>* = *Ni* + 1, . . . , *Ni* + *Ne*).

Newly measured system outputs can be used to improve the statistical models of the noises for increased performance. Thereby, at each iteration, every new set is added to the initialisation set and the covariance matrices *Q* and *R* are updated using Equations (11)–(13) for *k* = 1, . . . , *Ne*.

$$Q\_k = \operatorname{diag} \left( \underset{1 \le i \le n}{\operatorname{E}} \left[ \underset{1 \le j \le N\_i + k}{\operatorname{var}} \left( X^i\_j \right) \right] \; \underset{1 \le i \le n}{\operatorname{E}} \left[ \underset{1 \le j \le N\_i + k}{\operatorname{var}} \left( Y^i\_j \right) \right] \; \underset{1 \le j \le n}{\operatorname{E}} \left[ \underset{1 \le j \le N\_i + k}{\operatorname{var}} \left( Z^i\_j \right) \right] \right)$$

$$\underset{1 \le i \le n}{\operatorname{E}} \left[ \underset{1 \le j \le N\_i + k}{\operatorname{var}} \left( A X^i\_j \right) \right] \; \underset{1 \le i \le n}{\operatorname{E}} \left[ \underset{1 \le j \le N\_i + k}{\operatorname{var}} \left( A Y^i\_j \right) \right] \; \underset{1 \le i \le n}{\operatorname{E}} \left[ \underset{1 \le j \le N\_i + k}{\operatorname{var}} \left( A Z^i\_j \right) \right] \tag{11}$$

$$R\_k = \operatorname{diag} \left( R^1\_{k'}, \dots, R^i\_{k'}, \dots, R^n\_k \right) \tag{12}$$

$$R\_k^i = \operatorname{diag} \left( \max\_{1 \le j \le N\_i + k} \left( X\_j^i \right)\_{\substack{1 \le j \le N\_i + k}} \left( Y\_j^i \right)\_{\substack{1 \le j \le N\_i + k \end{pmatrix}}} \left( Z\_j^i \right)\_{\substack{1 \le j \le N\_i + k \end{pmatrix}},$$

$$\operatorname\*{var}\_{1 \le j \le N\_i + k} \left( AX\_j^i \right)\_{\substack{1 \le j \le N\_i + k \end{pmatrix}}} \left( A Y\_j^i \right)\_{\substack{1 \le j \le N\_i + k \end{pmatrix}}} \left( A Z\_j^i \right)\_{\substack{1 \le j \le N\_i + k \end{pmatrix}}} \left( A Z\_j^j \right). \tag{13}$$

Considering that the estimated state trajectory (*x*ˆ*k*|*k*) belongs to a system with free dynamics where, in the absence of the noise effects, the state should be constant, a final estimate with a better precision can be achieved by averaging the values of the estimated state trajectory according to Equation (14). The timeline of all the procedures that are involved in the proposed algorithm is presented in Figure 2. In Algorithm 1 the proposed procedure is summarised.

$$\mathfrak{X} = \underset{N\_i + 1 \le k \le N\_i + N\_\ell}{E} \left[ \mathfrak{X}\_{k|k} \right] \tag{14}$$

#### *Sensors* **2020**, *20*, 2746

**Algorithm 1:** The proposed algorithm for multi-camera pose fusion.


#### *3.2. Setup Calibration Procedure*

Given a set of *n* cameras in a pre-defined fixed configuration, having unknown absolute positions, the purpose of the setup calibration procedure is to determine, in a precise manner, their relative positions. As depicted in Figure 3, the goal is to determine the set of homogenous transformations *TCj Ci* where *i* = 1, . . . , *n* and *j* = *i* + 1, . . . , *n*.

**Figure 3.** The calibration of a set of *n* cameras requires to determine their relative positions *TCj Ci* .

This requires the use of a precision gauge, which can be manufactured in the form of a cube of fiducial markers like in [21] or in any 3D shape where the markers can be viewed all around. The gauge needs to be manufactured or calibrated with increased precision. It can be seen as the absolute precision reference used to calibrate the entire positioning system. In Figure 4, a conceptual diagram of a gauge containing *m* fiducial markers is presented. The set of homogenous transformations between all the markers (*TMj Mi* where *i* = 1, . . . , *m* and *j* = *i* + 1, . . . , *m*) is precisely known.

**Figure 4.** A setup of cameras is calibrated using a precision gauge. It is a manufactured feature having *m* fiducial markers where all their relative positions are precisely known.

The camera setup calibration procedure is done simultaneously for all the camera pairs *Ci*, *Cj* where *i* = 1, ... , *n* and *j* = *i* + 1, ... , *n*, with the goal of estimating the homogenous transformation *TCj Ci* (as presented in Figure 5). It starts with placing the precision gauge inside the environment. Depending on the orientation, each camera sees a different portion of the gauge, i.e., from all *n* fiducial markers, *Ci* can measure the position of *ni* markers and *Cj* can measure only *nj*, where *ni* ≤ *n* and *nj* ≤ *n*. It is preferable that at least one fiducial marker *Mk* is seen by both cameras, otherwise, it can be virtually determined using the gauge transformations.

The proposed algorithm estimates the *TMk Ci* homogenous transformation using the noisy measurements from all *ni* markers using the *Ci* camera. This transformation is expressed multiple times in terms of *TMl Ci* pose measurement (where *<sup>l</sup>* = 1, ... , *ni*) using Equation (15). The *<sup>T</sup>Mk Ml* transformation is precisely known from the gauge.

$$T\_{Ci}^{\text{Mk}} \approx T\_{Ci}^{\text{Mk}}{}\_{I} = T\_{Ci}^{\text{Ml}} \cdot T\_{MI}^{\text{Mk}} \tag{15}$$

The resulting <sup>ˆ</sup> *<sup>T</sup>Mk Ci <sup>l</sup>* pose is converted to translations and Euler angles which are used for building the output measurement vector of Equation (3).

**Figure 5.** The process of calibrating all the pairs of *n* cameras, which illustrates all the homogenous transformations involved in finding the relative position of *Ci* and *Cj* cameras.

The proposed algorithm is further applied and the result is an estimation of *TMk Ci* having a higher degree of precision and being closer to the real value. For the *Cj* camera, *TMk Cj* is estimated in a similar manner. Consequently, *TCj Ci* transformation is computed using Equation (16).

$$T\_{Ci}^{Cj} = T\_{Ci}^{Mk} \cdot \left( T\_{Mk}^{Cj} \right)^{-1} \tag{16}$$

#### *3.3. Position Estimation Procedure*

Having a calibrated camera setup, the aim of this procedure is to estimate the position of a fiducial marker attached to a specific instrument, according to the application.

In Figure 6, the conceptual diagram of this procedure is depicted. Depending on the configuration, the *M* marker can be seen only by a number of *nm* out of *n* cameras (*nm* ≤ *n*). An arbitrary camera (*Cr*) is chosen, which is considered the positioning reference.

**Figure 6.** The homogenous transformations involved in estimating the position of a fiducial marker placed in the environment of a calibrated camera setup.

The proposed algorithm is estimating the *T<sup>M</sup> Cr* homogenous transformation using the noisy measurements from each of the *nm* cameras. In a similar manner to the setup calibration procedure, the *T<sup>M</sup> Cr* transformation is expressed multiple times in terms of *<sup>T</sup><sup>M</sup> Ck* pose measurement (where *k* = 1, . . . , *nm*) using Equation (17). The *TCk Cr* is precisely known from the setup calibration.

$$T\_{\rm Cr}^{M} \approx T\_{\rm Crk}^{\tilde{M}} = T\_{\rm Cr}^{\rm Ck} \cdot T\_{\rm Ck}^{M}.\tag{17}$$

The resulting ˆ *T<sup>M</sup> Cr <sup>k</sup>* pose is converted to translations and Euler angles and used for building the output measurement vector of Equation (3). The proposed algorithm is further applied and the result is an estimation of *T<sup>M</sup> Cr* which has a higher degree of precision and is closer to the real value.

This approach can be used in a similar manner for measuring the relative position of multiple fiducial markers in the environment, which is very useful in alignment tasks as presented in [7].

#### **4. Simulation Results**

#### *4.1. Monte Carlo Setup*

There are two factors that contribute and affect the precision of estimating the pose of the fiducial marker. First, there are the physical and environment-related aspects, which include the optical specifications of the imaging system (the sensor and optical resolution, the focal length, the depth of field, and the field of view), the environment illumination conditions (how strong, uniform, and consistent the lighting is) and if the cameras are optimally positioned so as to achieve good viewing angles and maintain consistent accuracy along the working area. Secondly, there are

factors related to the algorithms regarding how precise the pose can be estimated and how much the effect of the noise can be mitigated during the data fusion. In this paper we decouple the two types of factors and only consider the behaviour of our proposed method, so as to have a first qualitative assessment regarding the precision.

Consequently, we developed a simulation environment that aims to replicate how our method performs in a real setup, considering the noisy pose estimations it receives from the solvepnp algorithm. For an increased confidence in the results, we adopted a Monte Carlo approach in which we statistically analysed how the noise is propagated through our method and how the precision is affected. The simulation is implemented in MATLAB where positions of multiple cameras, precision gauges, and fiducial markers are virtually defined. In order to simulate the noisy input from the solvepnp, each position (that is supposed to be measured in the real environment) is disturbed with additive random noise having normal distribution. The noise is configured considering the precision limits we determined experimentally for one camera in our previous work [21]: for *X* and *Y*, 75 μm, for *Z*, 300 μm, and for *AX*, *AY* and *AZ*, 0.02◦. The mean value of the noise is 0 while the standard deviation (*σ*) was configured in such a way that 95.45% of the noise values are within the experimental limits (inside [−2*σ*, 2*σ*]).

The simulation puts in place three scenarios which are additionally used to assess the different contributions between the number of cameras and the number of the fiducial markers from the gauge:


The results are compared against estimating the pose using only one camera in the same environment. The Monte Carlo simulation performs 5000 iterations where, in each run and for each of the scenarios, the setup is calibrated using the gauge. The calibration is further used to estimate the pose of the fiducial marker which is compared with the true, predefined one. The *Ni* and *Ne* parameters are configured to 20 and 50 respectively. In a real application, the choice of these two parameters is a matter of cost optimisation, considering the computational effort, the resources available, and the required measuring frequency.

Compared with a real setup, we adopted one simplifying hypothesis. The angle of incidence of the marker relative to the camera, as we showed in [21], has an influence on the precision. Close to normal angles of incidence tend to bring more noise in the estimation. In this study, we only consider that the precision of the solvepnp algorithm to be consistent, regardless of the angle. However, in other respects, the simulation is considering the worst case scenario because of the following arguments:


#### *4.2. Results*

Figures 7–12 presents the simulation results for estimating *X*, *Y*, and *Z* coordinates and *AX*, *AY*, and *AZ* orientation angles, which are given in terms of the probability density function (PDF) and

the standard deviation (SD) of the error. The first plot from each Figure depicts the estimation error achieved using only one camera. The following three plots give the results achieved using the setups from each of the above-mentioned scenarios.

The results show that the proposed algorithm achieves an increase in precision which is close to an order of magnitude. It can also be observed that it is of greater importance to have more fiducial markers in the precision gauge instead of more cameras. In scenario #3, a slight decrease of precision is experienced in comparison to the #1 scenario. This is to be expected as each added camera is an additional noise source. However, the benefit of achieving a larger working area is far more important. In Table 1 the results are summarised and compared with regards to the limits of the variation interval [−2*σ*, 2*σ*] where it is expected that 95.45% of the errors will occur. This can be considered the precision that the positioning system is expected to achieve when using th proposed method.

**Figure 7.** The probability density function (PDF) and the standard deviation (SD) of the error for estimating the *X* coordinate.

**Figure 8.** The probability density function (PDF) and the standard deviation (SD) of the error for estimating the *Y* coordinate.

**Figure 9.** The probability density function (PDF) and the standard deviation (SD) of the error for estimating the *Z* coordinate.

**Figure 10.** The probability density function (PDF) and the standard deviation (SD) of the error for estimating the *AX* angle.

**Figure 11.** The probability density function (PDF) and the standard deviation (SD) of the error for estimating the *AY* angle.

**Figure 12.** The probability density function (PDF) and the standard deviation (SD) of the error for estimating the *AZ* angle.

**Table 1.** The simulation results for estimating the pose using only one camera and multiple cameras in three scenarios. The value is the boundary of the error variation interval (2*σ*).


In order to further emphasise the simulation results achieved by the proposed method, Figures 13–18 depict a set of examples from the 3rd scenario which show how the estimation of the state vector elements is evolving. Each of the figures presents: The noisy measurement from the five cameras, the value of the state vector computed after the initialisation procedure, the evolution of the state vector during the Kalman estimation, and the final state estimation which falls close to the true value. Although the noise in the measurements is amplified because the reconstruction of the position of the marker in different cameras, the evolution of the Kalman estimation shows a clear noise damping effect.

**Figure 13.** The evolution of the estimation of *X*.

**Figure 14.** The evolution of the estimation of *Y*.

**Figure 15.** The evolution of the estimation of *Z*.

**Figure 16.** The evolution of the estimation of *AX*.

**Figure 17.** The evolution of the estimation of *AY*.

**Figure 18.** The evolution of the estimation of *AZ*.

#### **5. Conclusions**

With respect to the considered case study, the results show that the proposed method and algorithm have managed to successfully meet the objectives. The working area could be increased in accordance with the number of cameras in the setup. This is a decision-making procedure that needs to take into account the cost relative to the working area and the precision required. For our simulation scenarios, the precision increase was close to an order of magnitude, which was around 10–15 μm for *X* and *Y* coordinates, 30 μm for *Z* and 0.002◦ for *AX*, *AY*, and *AZ* orientation angles.

The cost is an important benefit of the system compared to other solutions like laser trackers which tend to be extremely expensive. In addition to that, our proposed method could achieve relative and simultaneous positioning of multiple fiducial markers, which supports the development of advanced applications.

Future work will include a complete analysis of the method in a real environment where all physical and algorithm-related factors are considered, and a precision comparison against other methods presented in the literature.

**Author Contributions:** Conceptualisation, D.C.P. and I.D.; methodology, I.D., S.I.C., and M.O.C.; software, D.C.P. and M.O.C.; validation, D.C.P. and M.O.C.; formal analysis, D.C.P. and I.D.; investigation, D.C.P., S.I.C., and M.O.C.; resources, M.O.C.; writing—original draft preparation, D.C.P., S.I.C., and M.O.C.; visualisation, D.C.P.; supervision, I.D. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was supported by Extreme Light Infrastructure Nuclear Physics Phase II, a project co-financed by the Romanian Government and the European Union through the European Regional Development Fund and the Competitiveness Operational Programme (No. 1/07.07.2016,COP, ID 1334). It was also supported by POC contract 53/05.09.2016, ID 40 270, MySMIS 105976 Ecosystem for ICT Research, Innovation, Product and Service Development for a Internet of Things Interconnected Society—NETIO, subsidiary contract 1265/22.01.2018, Development of an Innovative Solution for Efficient Communication in an Industrial Environment—DIVE and by Romanian Ministry of Research through project PN 19 06 01 05/2019.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


c 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

#### *Article*
