# **Assistive Technology and Biomechatronics Engineering**

Edited by Kimiyasu Kiyota, Akira Shionoya and Mitsuteru Inoue Printed Edition of the Special Issue Published in *Sensors*

www.mdpi.com/journal/sensors

## **Assistive Technology and Biomechatronics Engineering**

## **Assistive Technology and Biomechatronics Engineering**

Editors

**Kimiyasu Kiyota Akira Shionoya Mitsuteru Inoue**

MDPI Basel Beijing Wuhan Barcelona Belgrade Manchester Tokyo Cluj Tianjin

*Editors* Kimiyasu Kiyota Department of Human-Oriented Information Systems Engneering National Institute of Technology (KOSEN) Kumamoto College Koshi Japan

Akira Shionoya Physical Education and Health Care Center Nagaoka University of Technology Niigata Japan

Mitsuteru Inoue National Institute of Technology (KOSEN) Kumamoto College Koshi Japan

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Special Issue published online in the open access journal *Sensors* (ISSN 1424-8220) (available at: www.mdpi.com/journal/sensors/special issues/KOSEN ISATE2021 sensors).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-6413-5 (Hbk) ISBN 978-3-0365-6412-8 (PDF)**

© 2023 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

## **Contents**


### **Kimiyasu Kiyota, Takaaki Ishibashi, Manabu Shimakawa and Kazuyuki Ito** Effects of Social Implementation Education for Assistive Device Engineers at NIT (KOSEN) through the Development of a Digital Reading Device for the Visually Impaired Reprinted from: *Sensors* **2022**, *22*, 1047, doi:10.3390/s22031047 . . . . . . . . . . . . . . . . . . . . **139 Carlos Wellington P. Gon¸calves, Rog´erio A. Richa and Antonio P. L. Bo** Tracking and Classification of Head Movement for Augmentative and Alternative

Reprinted from: *Sensors* **2022**, *22*, 435, doi:10.3390/s22020435 . . . . . . . . . . . . . . . . . . . . . **153**

Communication Systems

## **About the Editors**

#### **Kimiyasu Kiyota**

Kimiyasu Kiyota is a professor at NIT, Kumamoto CollegeKOSEN). His areas of expertise are welfare engineering and assistive technology education. In particular, he has developed assistive devices for the visually impaired, such as pen-input systems and tracing-reading systems.

#### **Akira Shionoya**

Akira Shionoya is a head of Physical Education and Health Care Center at Nagaoka University of Technology. His area of research expertise is sport engineering, bio-mechanics, human dynamics, sport material, and applied physiology.

#### **Mitsuteru Inoue**

Mitsuteru Inoue is a board member of head quarter at National Institute of Technology (KOSEN). His area of research expertise is magnetophotonics, spin systems, spin waves, hologram memory, and hologram display.

## **Preface to "Assistive Technology and Biomechatronics Engineering"**

This Special Issue focuses on assistive technologies (AT) to solve biomechanical and motor control problems in people with health impairments due to disability, illness, or injury. All over the world, technologies are being developed to make human life richer and more comfortable. However, some people are unable to benefit from these technologies. For these people, we have identified new advanced science and technology related to AT from around the world and published a wide range of papers related to assistive technologies based on previous basic engineering, such as sports engineering, human dynamics, biomechanics, and motion control using IoT sensors. The papers were published in the *Journal of Biomechatronics and Sports Engineering*.

We hope that this Special Issue will be one of the proposals to contribute to a symbiotic society by bridging the digital divide in IoT technology for the elderly and the disabled.

> **Kimiyasu Kiyota, Akira Shionoya, and Mitsuteru Inoue** *Editors*

## *Article* **Impedance Characteristics of Monolayer and Bilayer Graphene Films with Biofilm Formation and Growth**

**Ryoichi Nakagawa 1,†, Kai Saito 1,‡, Hideyuki Kanematsu 1,\* , Hidekazu Miura <sup>2</sup> , Masatou Ishihara <sup>3</sup> , Dana M. Barry 4,5 , Takeshi Kogo <sup>1</sup> , Akiko Ogawa <sup>1</sup> , Nobumitsu Hirai <sup>1</sup> , Takeshi Hagio <sup>6</sup> , Ryoichi Ichino <sup>6</sup> , Masahito Ban <sup>7</sup> , Michiko Yoshitake <sup>8</sup> and Stefan Zimmermann <sup>9</sup>**


**Abstract:** Biofilms are the result of bacterial activity. When the number of bacteria (attached to materials' surfaces) reaches a certain threshold value, then the bacteria simultaneously excrete organic polymers (EPS: extracellular polymeric substances). These sticky polymers encase and protect the bacteria. They are called biofilms and contain about 80% water. Other components of biofilm include polymeric carbon compounds such as polysaccharides and bacteria. It is well-known that biofilms cause various medical and hygiene problems. Therefore, it is important to have a sensor that can detect biofilms to solve such problems. Graphene is a single-atom-thick sheet in which carbon atoms are connected in a hexagonal shape like a honeycomb. Carbon compounds generally bond easily to graphene. Therefore, it is highly possible that graphene could serve as a sensor to monitor biofilm formation and growth. In our previous study, monolayer graphene was prepared on a glass substrate by the chemical vapor deposition (CVD) method. Its biofilm forming ability was compared with that of graphite. As a result, the CVD graphene film had the higher sensitivity for biofilm formation. However, the monolayer graphene has a mechanical disadvantage when used as a biofilm sensor. Therefore, for this new research project, we prepared bilayer graphene with high mechanical strength by using the CVD process on copper substrates. For these specimens, we measured the capacitance component of the specimens' impedance. In addition, we have included a discussion about the possibility of applying them as future sensors for monitoring biofilm formation and growth.

**Keywords:** biofilm; sensors; impedance; EPS; Raman spectroscopy

#### **1. Introduction**

Biofilms are inhomogeneous film-like matters formed by bacterial activities. Bacteria in various environments attach to materials' surfaces and form biofilms on them,

**Citation:** Nakagawa, R.; Saito, K.; Kanematsu, H.; Miura, H.; Ishihara, M.; Barry, D.M.; Kogo, T.; Ogawa, A.; Hirai, N.; Hagio, T.; et al. Impedance Characteristics of Monolayer and Bilayer Graphene Films with Biofilm Formation and Growth. *Sensors* **2022**, *22*, 3548. https://doi.org/10.3390/ s22093548

Academic Editors: Kimiyasu Kiyota, Akira Shionoya and Mitsuteru Inoue

Received: 11 March 2022 Accepted: 11 April 2022 Published: 6 May 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

which could result in various industrial, hygiene, and medical problems. These problems cover broad areas including corrosion, scale problems (in pipe works, heat exchangers, toilets, kitchens, and bathrooms), and medical fronts in hospitals [1–7]. Therefore, it is very important to solve problems induced by biofilms. As for the evaluation of biofilms, many methods have already been proposed. They are mainly classified into two types. One of them is visualization by using expensive cutting-edge apparatus such as electron microscopes [8–12], laser microscopes [13–17], Raman or infrared analytical apparatus and a variety of their combinations [18–29]. These evaluation methods could provide valuable quantitative information to help us improve our evaluation of biofilms. However, the qualitative analyses have been difficult to use for obtaining quantitative information. Moreover, from a practical point of view, the method would be too professional/sophisticated for everyone to use. The other category includes biological methods such as measuring the number of bacteria, staining, gene analyses, etc., [7,30–35]. Among them, measuring the number of bacteria is not always compatible with biofilm formation and growth. The reason for this is that biofilms are formed by bacterial activities in the initial stage and their behaviors are affected by EPS during the growth stage. Gene analyses require special apparatus and professional knowledge. On the other hand, staining methods are visible, convenient, and do not require lots of professional skills to obtain a quantitative evaluation. However, they are not suitable for in situ monitoring. Based on this background information, it is desirable to sense (detect) biofilms by using electronic devices to overcome the weak points of conventional evaluation methods.

In this experiment, we focused on the impedance behaviors of materials forming biofilms. When biofilms form on materials, the impedance behaviors change due to the different dielectric characteristics of the substrates. However, the susceptibility of materials (electrodes) to biofilm formation and growth is critical. In the past, we confirmed the relatively high susceptibility to biofilm formation [31,36–39]. In those cases, we used monolayer graphene on copper substrates produced by CVD [40], since the process could produce reproducible and high quality (high covering ratio) graphene films. However, the mechanical strength of monolayer graphene is relatively weak and the coverage is generally not so high. Therefore, in this experiment, we used double-layer graphene and carried out an investigation to find out if its impedance characteristics (mainly the capacitance components) would provide measurements suitable for practical sensing in the future.

#### **2. Materials and Methods**

#### *2.1. Specimens—Graphene Film Formation*

Copper foils of 0.21 × 0.247 m were heated to 600 ◦C by resistance heating. Then plasma formed in the gas mixture (of methane and hydrogen) and was irradiated on the specimens to form monolayer and bilayer graphene on copper substrates. The display of surface layers is shown in Figure 1, schematically. The graphene layer forms on 10 µm copper foil. The thickness of graphene films were confirmed by their transparency, since the transparency for monolayer and bilayer graphene were already fixed [40,41]. On the other hand, the 100 µm polyester film (detachable by heating) exists on the other side of the copper foil. Those sheets were cut into small coupon of 10 mm × 10 mm and were used for various tests.

#### *2.2. Biofilm Formation and Freeze Dehydration*

To produce biofilms on specimens, we used Gram-negative *Escherichia coli* (*E. coli*, K-12, G6). The bacteria were cultivated in LB solution at 37 ◦C for 24 h in advance, so that the number of bacteria would reach 10<sup>8</sup> CFU/mL (CFU stands for colony formation unit and corresponding to the colony numbers). Then graphene film specimens were immersed into each of 12 plastic wells and 1.6 mL bacterial solution (precultured as mentioned above) were added into each well. The plastic wells were put into an incubator at 25 ◦C for 24 h.

**Figure 1.** The structure of the specimen used in this experiment. **Figure 1.** The structure of the specimen used in this experiment.

*2.2. Biofilm Formation and Freeze Dehydration*  To produce biofilms on specimens, we used Gram-negative *Escherichia coli* (*E. coli*, K-12, G6). The bacteria were cultivated in LB solution at 37 °C for 24 h in advance, so that the number of bacteria would reach 108 CFU/mL (CFU stands for colony formation unit and corresponding to the colony numbers). Then graphene film specimens were immersed into each of 12 plastic wells and 1.6 mL bacterial solution (precultured as mentioned above) were added into each well. The plastic wells were put into an incubator at 25 °C for 24 h. After 24 h, the specimens immersed in LB culture were freeze dried in the following way. The mixed solutions of distilled water and ethanol were prepared at various ratios After 24 h, the specimens immersed in LB culture were freeze dried in the following way. The mixed solutions of distilled water and ethanol were prepared at various ratios (water: ethanol = 7:3, 5:5, 3:7, 2:8, 1:9, 0.5:9.5, 0.2:9.8, and 0:1) to replace water components of biofilms completely with ethanol. Specimens were immersed in each solution for 15 min in the order of increasing ethanol ratios. Then the specimens were immersed into each of mixed solutions of ethanol and t-butyl alcohol in the order of 7:3, 5:5, 3:7, 2:8, 1:9, 0.5:9.5, 0.2:9.8, 0:1 of ethanol and t-butyl solutions to completely replace ethanol in biofilms with t-butyl solution. Finally, the specimens were kept frozen in a freezer for 30 min and then dried in a vacuum desiccator by using a rotary pump. Then the specimens were analyzed using Raman spectroscopy.

#### (water: ethanol = 7:3, 5:5, 3:7, 2:8, 1:9, 0.5:9.5, 0.2:9.8, and 0:1) to replace water components *2.3. Raman Spectroscopy*

of biofilms completely with ethanol. Specimens were immersed in each solution for 15 min in the order of increasing ethanol ratios. Then the specimens were immersed into each of mixed solutions of ethanol and t-butyl alcohol in the order of 7:3, 5:5, 3:7, 2:8, 1:9, 0.5:9.5, 0.2:9.8, 0:1 of ethanol and t-butyl solutions to completely replace ethanol in biofilms with t-butyl solution. Finally, the specimens were kept frozen in a freezer for 30 min and then The apparatus used for Raman spectroscopy was NRS-3100 made by JASCO Corporation, Tokyo, Japan. The optical microscope was combined with the spectroscopy. A green laser beam (100 mW, 532 nm) was irradiated on the specimens' surfaces and Raman shifts were observed between 500 and 4000 cm−<sup>1</sup> . The measurements were repeated several times at certain locations for each of three specimens (N = 3).

dried in a vacuum desiccator by using a rotary pump. Then the specimens were analyzed

#### using Raman spectroscopy. *2.4. Absorbance Measurement*

*2.3. Raman Spectroscopy* The apparatus used for Raman spectroscopy was NRS-3100 made by JASCO Corporation, Tokyo, Japan. The optical microscope was combined with the spectroscopy. A green laser beam (100 mW, 532 nm) was irradiated on the specimens' surfaces and Raman shifts were observed between 500 and 4000 cm**−**1. The measurements were repeated several times at certain locations for each of three specimens (N = 3). *2.4. Absorbance Measurement* After the impedance measurements, specimens were immediately stained by 0.1% crystal violet (CV) solution. The 1.6 mL CV solution was prepared for each plastic well and specimens were immersed into them for 30 min. Then the CV solution was removed, and the specimens were washed three times with pure water. Next, they were immersed into 3 mL ethanol solution for 30 min at room temperature to extract CV-stained biofilms. Then 200 µL of extracted solutions underwent absorbance measurements by a plate reader (Multi Scan FC, Thermofisher Scientific Co, Yokohama, Japan.). Light (570 nm−<sup>1</sup> ) was irradiated to each extracted solution and the absorbance was measured to evaluate the quantitative amount of biofilm.

#### After the impedance measurements, specimens were immediately stained by 0.1% *2.5. Impedance Measurement*

crystal violet (CV) solution. The 1.6 mL CV solution was prepared for each plastic well and specimens were immersed into them for 30 min. Then the CV solution was removed, and the specimens were washed three times with pure water. Next, they were immersed into 3 mL ethanol solution for 30 min at room temperature to extract CV-stained biofilms. Then 200 μL of extracted solutions underwent absorbance measurements by a plate reader (Multi Scan FC, Thermofisher Scientific Co, Yokohama, Japan.). Light (570 nm**−**1) was irradiated to each extracted solution and the absorbance was measured to evaluate the quantitative amount of biofilm. *2.5. Impedance Measurement* As for the impedance measurement, specimens were removed from the plastic wells after being immersed in bacterial solutions. They dried naturally for 15 min in the air and then had impedance measurements. The electrode part and the schematic circuit are shown in Figure 2. Specimens were put between the electrode parts of the measurement system and sandwiched between two copper plate electrodes. The contact part between the copper electrodes and specimens was composed of vinylidene chloride films (10 micrometer thickness) and the sandwiched electrodes were secured by clips, so that a certain pressure was softly applied to make the electric contact. The electrode part was connected to a chemical impedance Analyzer (IM 3590, Hioki Co., Ueda, Japan). The impedance characteristics were measured when the frequencies from 100 Hz to 20 kHz were applied to the specimens.

As for the impedance measurement, specimens were removed from the plastic wells after being immersed in bacterial solutions. They dried naturally for 15 min in the air and then had impedance measurements. The electrode part and the schematic circuit are

**Figure 2.** The electrode structure for impedance measurements. **Figure 2.** The electrode structure for impedance measurements.

#### **3. Results**

the specimens.

#### **3. Results**  *3.1. Raman Spectroscopy with Optical Microscopy*

*3.1. Raman Spectroscopy with Optical Microscopy* Generally, bilayer graphene films covered the copper foil substrate better and the sensitivity of graphene to biofilm formation was expected to be higher. The mechanical strength for two-layer graphene films was also expected to be higher than that for mono-Generally, bilayer graphene films covered the copper foil substrate better and the sensitivity of graphene to biofilm formation was expected to be higher. The mechanical strength for two-layer graphene films was also expected to be higher than that for monolayer graphene films.

shown in Figure 2. Specimens were put between the electrode parts of the measurement system and sandwiched between two copper plate electrodes. The contact part between the copper electrodes and specimens was composed of vinylidene chloride films (10 micrometer thickness) and the sandwiched electrodes were secured by clips, so that a certain pressure was softly applied to make the electric contact. The electrode part was connected to a chemical impedance Analyzer (IM 3590, Hioki Co. Ueda, Japan). The impedance characteristics were measured when the frequencies from 100 Hz to 20 kHz were applied to

layer graphene films. Figure 3 shows the Raman spectroscopic results for monolayer graphene film specimen and two-layer graphene film specimen. For one of the representative monolayer graphene specimens (Figure 3a), the remarkable peak was found at 2700 cm−1 and it could be attributed to 2D band peak of graphene. A G band peak was observed at 1580cm-1. It shows the existence of graphite. However, the peak was weak. On the other hand, one of the representative bilayer graphene specimens (Figure 3b) showed the G-band peak at 1580 cm−1 more clearly and suggests that the crystallographic characteristics of graphene Figure 3 shows the Raman spectroscopic results for monolayer graphene film specimen and two-layer graphene film specimen. For one of the representative monolayer graphene specimens (Figure 3a), the remarkable peak was found at 2700 cm−<sup>1</sup> and it could be attributed to 2D band peak of graphene. A G band peak was observed at 1580 cm−<sup>1</sup> . Itshows the existence of graphite. However, the peak was weak. On the other hand, one of the representative bilayer graphene specimens (Figure 3b) showed the G-band peak at1580 cm−<sup>1</sup> more clearly and suggests that the crystallographic characteristics of graphene film would have been lost from the viewpoint of crystallography to some extent.

film would have been lost from the viewpoint of crystallography to some extent.

(**a**) (**b**) **Figure 3.** Raman peaks for graphene film specimens without biofilms. (**a**) Monolayer graphene film, (**b**) bilayer graphene film.

Figure 4 shows the Raman spectroscopic peaks for specimens forming biofilms. Figure 4a shows a typical results for monolayer graphene film specimens, while Figure 4b shows the results for a typical bilayer graphene film specimen. For specimens forming biofilms on monolayer graphene, Raman peaks derived from biofilms were found. For example, the peak(s) around 2500 cm−<sup>1</sup> could be attributed to lipids and/or proteins. From 1000 cm−<sup>1</sup> to 1600 cm−<sup>1</sup> , the peaks could be attributed to proteins, lipids, and/or nucleic acids. On the other hand, the 2D band peak around 2700 cm−<sup>1</sup> became relatively weak and almost disappeared. Moreover, for specimens forming biofilms on bilayer graphene

films, some peaks attributed to biofilms appeared in addition to the 2D and G-band Raman peaks, as shown in Figure 4b. For example, lipid and/or protein peaks were found at 2800 cm−<sup>1</sup> . Furthermore, peaks from 1400 cm−<sup>1</sup> to 1500 cm−<sup>1</sup> (peaks assigned to proteins and/or lipids) were found. They clearly show that biofilms were qualitatively formed on graphene film specimens.

**Figure 4.** Raman peaks for graphene film specimens with biofilms. (**a**) Monolayer graphene film, (**b**) two-layer graphene film.

#### *3.2. Biofilm Assay Using CV Staining*

It was hard and impossible for us to quantitatively evaluate biofilm amounts by using Raman Spectroscopy. Then we measured and evaluated stained biofilms quantitatively, using the CV staining method. Figure 5 shows the results of CV staining. The vertical axis shows the absorbance, which corresponds to the amount of biofilm formed on the specimens. As shown in Figure 5, the amount of biofilm (absorbance values) was higher for monolayer graphene film specimens (Figure 5a) than for bilayer graphene film specimens (Figure 5b), even though the graphene coverage of the monolayer graphene film specimen was lower than that of the bilayer film.

#### *3.3. Impedance Measurements*

Figure 6 shows the impedance behaviors for specimens of graphene films without biofilms. Figure 6a shows the change of impedance behavior with increasing frequencies. Figure 6b shows the differential curve of impedance.

As shown in Figure 6, the impedance value decreased with increased frequencies due to dielectric relaxation. The differential curve shows that the tendency and the differential value increased with frequencies monotonously. However, the increment gradually became smaller with increased frequency. In the range of lower frequencies, the differential curve was vibrated. We presume, it could be attributed to superfluous noises sometimes seen in the region.

Figure 7 shows the impedance curve with frequencies (Figure 7a) for the monolayer specimen on which biofilms formed. The impedance values increased in the range of higher frequencies. The differential curve (Figure 7b) shows the complicated peaks at higher frequencies. We presume the differential peaks could be attributed to the existence of various polymeric substances derived from biofilms.

*Sensors* **2022**, *22*, x FOR PEER REVIEW 6 of 12

**Figure 5.** Absorbances of specimens stained by 0.1% crystal violet solution. Figure 6b shows the differential curve of impedance.

**Figure 5.** Absorbances of specimens stained by 0.1% crystal violet solution.

As shown in Figure 6, the impedance value decreased with increased frequencies due to dielectric relaxation. The differential curve shows that the tendency and the differential

**Figure 6.** Impedance behavior for specimens of monolayer graphene film (**a**) impedance curve, (**b**)

As shown in Figure 6, the impedance value decreased with increased frequencies due to dielectric relaxation. The differential curve shows that the tendency and the differential

differential curve.

(**a**) (**b**)

of various polymeric substances derived from biofilms.

of various polymeric substances derived from biofilms.

*Sensors* **2022**, *22*, x FOR PEER REVIEW 7 of 12

seen in the region.

curve, (**b**) differential curve.

seen in the region.

**Figure 7.** Impedance behaviors of monolayer graphene film specimens with biofilms. (**a**) Impedance curve, (**b**) differential curve. **Figure 7.** Impedance behaviors of monolayer graphene film specimens with biofilms. (**a**) Impedance curve, (**b**) differential curve. Figure 8a shows the impedance behavior with frequencies for the bilayer graphene

value increased with frequencies monotonously. However, the increment gradually became smaller with increased frequency. In the range of lower frequencies, the differential curve was vibrated. We presume, it could be attributed to superfluous noises sometimes

value increased with frequencies monotonously. However, the increment gradually became smaller with increased frequency. In the range of lower frequencies, the differential curve was vibrated. We presume, it could be attributed to superfluous noises sometimes

Figure 7 shows the impedance curve with frequencies (Figure 7a) for the monolayer specimen on which biofilms formed. The impedance values increased in the range of higher frequencies. The differential curve (Figure 7b) shows the complicated peaks at higher frequencies. We presume the differential peaks could be attributed to the existence

Figure 7 shows the impedance curve with frequencies (Figure 7a) for the monolayer specimen on which biofilms formed. The impedance values increased in the range of higher frequencies. The differential curve (Figure 7b) shows the complicated peaks at higher frequencies. We presume the differential peaks could be attributed to the existence

Figure 8a shows the impedance behavior with frequencies for the bilayer graphene film specimen without biofilms. The tendencies for the impedance curve and the differential curve (Figure 8b) were almost the same, even though the absolute values were different from each other. Figure 8a shows the impedance behavior with frequencies for the bilayer graphene film specimen without biofilms. The tendencies for the impedance curve and the differential curve (Figure 8b) were almost the same, even though the absolute values were different from each other. film specimen without biofilms. The tendencies for the impedance curve and the differential curve (Figure 8b) were almost the same, even though the absolute values were different from each other.

(**a**) (**b**) **Figure 8.** Impedance behaviors for specimens of bilayer graphene film. (**a**) Impedance curve, (**b**) **Figure 8.** Impedance behaviors for specimens of bilayer graphene film. (**a**) Impedance curve, (**b**) differential curve. **Figure 8.** Impedance behaviors for specimens of bilayer graphene film. (**a**) Impedance curve, (**b**) differential curve.

differential curve. Figure 9a shows the impedance curve for a specimen of bilayer graphene film and Figure 9b shows the differential curve. The bilayer graphene film shows the same tendencies for the impedance curve and the differential curve with those for specimens of monolayer graphene film. Therefore, the various peaks observed in the range of high frequencies should be attributed to EPS in the same way.

Figure 10 shows the impedance behaviors for freeze-dried specimens of bilayer graphene film. Figure 10a shows the impedance curve, while Figure 10b shows the differential curve. In this case, the impedance and the differential curves were very similar to those of specimens without biofilms.

cies should be attributed to EPS in the same way.

cies should be attributed to EPS in the same way.

*Sensors* **2022**, *22*, x FOR PEER REVIEW 8 of 12

**Figure 9.** impedance behaviors of bilayers graphene film specimens with biofilms, (**a**) impedance curve, (**b**) differential curve. **Figure 9.** impedance behaviors of bilayers graphene film specimens with biofilms, (**a**) impedance curve, (**b**) differential curve. tial curve. In this case, the impedance and the differential curves were very similar to those of specimens without biofilms.

Figure 9a shows the impedance curve for a specimen of bilayer graphene film and Figure 9b shows the differential curve. The bilayer graphene film shows the same tendencies for the impedance curve and the differential curve with those for specimens of monolayer graphene film. Therefore, the various peaks observed in the range of high frequen-

Figure 9a shows the impedance curve for a specimen of bilayer graphene film and Figure 9b shows the differential curve. The bilayer graphene film shows the same tendencies for the impedance curve and the differential curve with those for specimens of monolayer graphene film. Therefore, the various peaks observed in the range of high frequen-

**Figure 10.** Impedance behaviors of freeze-dried bilayer graphene film specimens with biofilms. (**a**) Impedance curve, (**b**) differential curve. **Figure 10.** Impedance behaviors of freeze-dried bilayer graphene film specimens with biofilms. (**a**) Impedance curve, (**b**) differential curve.

#### **4. Discussion 4. Discussion**

(**a**) (**b**) **Figure 10.** Impedance behaviors of freeze-dried bilayer graphene film specimens with biofilms. (**a**) Impedance curve, (**b**) differential curve. **4. Discussion**  We found in our previous studies that the monolayer graphene would be sensitive to biofilm formation, when compared with graphite specimens. It could be attributed to the existence of π electrons perpendicular to the surface in the resonance state. Therefore, biofilms formed on graphene films rather easily. This was also true for bilayer graphene We found in our previous studies that the monolayer graphene would be sensitive to biofilm formation, when compared with graphite specimens. It could be attributed to the existence of π electrons perpendicular to the surface in the resonance state. Therefore, biofilms formed on graphene films rather easily. This was also true for bilayer graphene films. The existence of biofilms was shown concretely by Raman Spectroscopy. As We found in our previous studies that the monolayer graphene would be sensitive to biofilm formation, when compared with graphite specimens. It could be attributed to the existence of π electrons perpendicular to the surface in the resonance state. Therefore, biofilms formed on graphene films rather easily. This was also true for bilayer graphene films. The existence of biofilms was shown concretely by Raman Spectroscopy. As described in the previous section, the impedance measurement shows that the increase of values and peaks of differential curves were found in higher frequencies. We presume that those increases, and peaks could be attributed to the biofilm formation and growth. Particularly, we expect that the differential peaks would be utilized as biofilm sensors for the future.

films. The existence of biofilms was shown concretely by Raman Spectroscopy. As The comparison between non-freeze dried and freeze-dried specimens suggests that the peaks in the higher frequency area could be related closely to the adsorption of hydroxyl groups to EPS of biofilms. The information was lost for the freeze-dried specimens.

Figure 11 shows the schematic diagram for biofilm on specimens and the electric path change with frequencies. At lower frequencies, the current tends to flow in the lower concentration areas of EPS. Then the ionic conduction in the aqueous solution of biofilms would occur more easily. However, the current tends to move in higher concentration areas of EPS when the frequency increases. Therefore, the impedance must change in higher frequencies and the differential peaks could be found there. Therefore, we could conclude that the differential peaks at higher frequencies are essential to indicate the existence of biofilms and could be used as biofilm sensors in the future. areas of EPS when the frequency increases. Therefore, the impedance must change in higher frequencies and the differential peaks could be found there. Therefore, we could conclude that the differential peaks at higher frequencies are essential to indicate the existence of biofilms and could be used as biofilm sensors in the future.

described in the previous section, the impedance measurement shows that the increase of values and peaks of differential curves were found in higher frequencies. We presume that those increases, and peaks could be attributed to the biofilm formation and growth. Particularly, we expect that the differential peaks would be utilized as biofilm sensors for

The comparison between non-freeze dried and freeze-dried specimens suggests that the peaks in the higher frequency area could be related closely to the adsorption of hydroxyl groups to EPS of biofilms. The information was lost for the freeze-dried specimens. Figure 11 shows the schematic diagram for biofilm on specimens and the electric path change with frequencies. At lower frequencies, the current tends to flow in the lower concentration areas of EPS. Then the ionic conduction in the aqueous solution of biofilms would occur more easily. However, the current tends to move in higher concentration

**Figure 11.** Schematic diagram of biofilms and current paths. **Figure 11.** Schematic diagram of biofilms and current paths.

*Sensors* **2022**, *22*, x FOR PEER REVIEW 9 of 12

the future.

As for the impedance characteristics of electrodes, Figure 12 shows the corresponding equivalent circuits for biofilms. We presume at this point that the conductive solution in biofilms could be attributed to the part composed of only a resistance. The part composed of only a capacitance could correspond to the electrode part. Then the central part composed of parallel capacitance and resistance corresponds to the biofilm itself. The resistance component represents the ionic conductive aqueous part, and the capacitance is the insulated part composed of EPS. Therefore, the multiple peaks found in the higher frequency area would correspond to various EPSs. Examples include polysaccharides, proteins, lipids, and nucleic acids, respectively. This presumption should be confirmed by further quantitative and qualitative investigations. As for the impedance characteristics of electrodes, Figure 12 shows the corresponding equivalent circuits for biofilms. We presume at this point that the conductive solution in biofilms could be attributed to the part composed of only a resistance. The part composed of only a capacitance could correspond to the electrode part. Then the central part composed of parallel capacitance and resistance corresponds to the biofilm itself. The resistance component represents the ionic conductive aqueous part, and the capacitance is the insulated part composed of EPS. Therefore, the multiple peaks found in the higher frequency area would correspond to various EPSs. Examples include polysaccharides, proteins, lipids, and nucleic acids, respectively. This presumption should be confirmed by further quantitative and qualitative investigations. *Sensors* **2022**, *22*, x FOR PEER REVIEW 10 of 12

**Figure 12.** Equivalent circuit of this model.

#### **5. Conclusions**

mation.

graphene films.

of biofilms.

in Japan.

**Figure 12.** Equivalent circuit of this model. **5. Conclusions**  By using monolayer and bilayer graphene films on copper foils and applying impedance spectroscopy to the specimens, we investigated the possibility of sensing (detecting) for biofilm formation and growth. The graphene film specimens were sandwiched by insulating electrodes and the impedance behaviors. We observed their changes with frequencies. The following results were obtained.


(2) As for graphene film specimens with biofilms, the increase of impedance was ob-

(4) The increase of impedance values and the differential peaks found in relatively higher frequencies were not observed for the freeze-dried specimens. It suggests that the peaks would be related closely to the adsorption of hydroxyl groups to the EPS

(5) The differential curves of impedance behaviors show some peaks in the higher frequency area. The peaks would correspond to the formation of EPS components.

**Author Contributions:** Original draft preparation: R.N. and K.S.; project administration: H.K., H.M., and S.Z.; review and editing: D.M.B. and H.K.; funding: H.M. and N.H.; resources: M.I. and M.B.; investigation: T.K., A.O., and M.Y.; supervision: T.H. and R.I. All authors have read and

**Funding:** This work was supported by JSPS KAKENHI (Grants-in-Aid for Scientific Research from the Japan Society for the Promotion of Science, Grant Number 20K05185 and 21K12739). A part of this work was supported by the GEAR 5.0 Project of the National Institute of Technology (KOSEN)

Therefore, the peaks could be utilized for the sensing of biofilms.

agreed to the published version of the manuscript.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable. **Data Availability Statement:** Not applicable.


**Author Contributions:** Original draft preparation: R.N. and K.S.; project administration: H.K., H.M. and S.Z.; review and editing: D.M.B. and H.K.; funding: H.M. and N.H.; resources: M.I. and M.B.; investigation: T.K., A.O. and M.Y.; supervision: T.H. and R.I. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by JSPS KAKENHI (Grants-in-Aid for Scientific Research from the Japan Society for the Promotion of Science, Grant Number 20K05185 and 21K12739). A part of this work was supported by the GEAR 5.0 Project of the National Institute of Technology (KOSEN) in Japan.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** The authors appreciate the Advanced Technology R&D Center of Mitsubishi Electric Co., Japan Food Research Laboratories (JFRL), and The Society of International Sustaining Growth for Antimicrobial Articles (SIAA) for their very useful advice and information about biofilms.

**Conflicts of Interest:** The authors declare no conflict of interest The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


## *Communication* **Image Correction Methods for Regions of Interest in Liver Cirrhosis Classification on CNNs**

**Yoshihiro Mitani 1,\* , Robert B. Fisher <sup>2</sup> , Yusuke Fujita <sup>3</sup> , Yoshihiko Hamamoto <sup>3</sup> and Isao Sakaida <sup>4</sup>**


**Abstract:** The average error rate in liver cirrhosis classification on B-mode ultrasound images using the traditional pattern recognition approach is still too high. In order to improve the liver cirrhosis classification performance, image correction methods and a convolution neural network (CNN) approach are focused on. The impact of image correction methods on region of interest (ROI) images that are input into the CNN for the purpose of classifying liver cirrhosis based on data from B-mode ultrasound images is investigated. In this paper, image correction methods based on tone curves are developed. The experimental results show positive benefits from the image correction methods by improving the image quality of ROI images. By enhancing the image contrast of ROI images, the image quality improves and thus the generalization ability of the CNN also improves.

**Keywords:** B-mode ultrasound images; liver cirrhosis classification; convolution neural networks; image correction; image quality improvement; inverse of tone curves; tone curves

#### **1. Introduction**

In the medical imaging field, a computer-aided diagnosis (CAD) (e.g., [1–3]) system which gives a second opinion is strongly needed. Ultrasound imaging is non-invasive and widely used for the diagnosis of liver cirrhosis [4]. Cirrhosis of the liver is expected to progress to liver cancer in the worst case. Therefore, we are investigating a CAD system to diagnose liver cirrhosis sooner [5,6]. Many believe that using machine learning and artificial intelligence is effective for designing CAD systems. In general, there are two types of ultrasound images: B-mode and M-mode. The B-mode shows the whole of the breast imaging, including the liver. On the other hand, the M-mode provides motion images of the liver aortic vessels [7].

In this study, B-mode liver ultrasound imaging is focused on. B-mode ultrasound image data was provided by the collaborator. These data include 12 cirrhosis patients and 8 normal subjects. There were five ultrasound images per person. Figure 1 shows B-mode liver ultrasound images. Figure 1a,b show a normal and a cirrhosis liver, respectively. In this study, we focus on classifying regions of interest (ROIs) from the B-mode ultrasound images. Figure 2 shows examples of ROI images. The ROI images were manually cut out from the liver areas by a physician. A total of 200 normal and 300 cirrhosis ROI images were collected. Figure 2a,b show normal and cirrhosis livers. The size of each ROI image was 32 by 32 pixels. The grey level was 8 bits. Thus, the values ranged from 0 to 255. This is a typical two-class problem, normal or cirrhosis. From this figure, it seems difficult to visually classify a liver as normal or cirrhosis if one is not a physician because of noisy ultrasound images.

**Citation:** Mitani, Y.; Fisher, R.B.; Fujita, Y.; Hamamoto, Y.; Sakaida, I. Image Correction Methods for Regions of Interest in Liver Cirrhosis Classification on CNNs. *Sensors* **2022**, *22*, 3378. https://doi.org/10.3390/ s22093378

Academic Editor: Francesco Lamonaca

Received: 31 March 2022 Accepted: 26 April 2022 Published: 28 April 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

**Figure 2.** ROI images.

In a previous study [5], we explored liver cirrhosis classification using an approach based on higher-order local auto-correlation (HLAC) features, which are known as handcrafted features in the pattern recognition field. The HLAC feature approach produced the best performance among our experimental results. However, this experimental result showed that the average error rate was over 40%. Even the best performance from the conventional approach was not very good.

Convolution neural networks (CNNs) are widely used in the medical imaging field (e.g., [8]). In recent studies, as shown in (e.g., [9–11]), issues of liver diseases as captured in ultrasound images have been addressed. Deep learning approaches were adopted in [9,11], and a combination of hand-crafted features and the classical classifier was used in [10]. Each of these approaches was reported to result in better accuracy. Note, however, that the datasets themselves were all different. Each of the datasets used in [9,11] was for liver fibrosis staging classification. The dataset in [11] was for normal and fatty liver, not cirrhosis. On the other hand, the dataset we used was for normal and cirrhosis. Cirrhosis is the most dangerous liver disease, as mentioned above. Therefore, it is difficult to collect a sufficient amount of data on it. The limitation on the available cirrhosis data increases the difficultly of the pattern recognition problem. The average error rate on the limited dataset may thus be high compared to another type of dataset.

We expect that the deep neural nets will be able to yield a good performance in this liver cirrhosis classification problem. One defect of CNNs is the need for many training samples. This leads to an over-training problem. In a previous study [6], we showed the effectiveness of the ROI image augmentation method by a perspective transformation.

Apart from the ROI image data augmentation, we would also like to improve the average error rate of liver cirrhosis classification using CNNs. By contrast with the conventional pattern recognition system, the CNN could work well by learning a combination of input images and output labels or class names. Thus, we may overlook the importance of obtaining better image quality. In the pattern recognition field in general, the richer the features that are obtained, the better the classification performance of the pattern recognition system. Most classifiers work well only if the features are good. Therefore, there has always been much effort devoted to obtaining better features, with both the classical and the deep learning approaches. However, deep learning can reduce the efforts because the CNN learns automatically by feeding up a combination of input images and output labels. Therefore, our focus is on improving the image quality of ROI images with the goal of improving the classification of liver cirrhosis. The method called CEUS (contrast-enhanced ultrasound) (e.g., [12]) has been expected to obtain more contrast-enhanced images than conventional ultrasound imaging. In this study, on the other hand, the aim is to apply image processing to the already available images to obtain higher-quality images.

Livers with cirrhosis are known to be physically harder. This would make the image regions slightly lighter. On the other hand, a normal liver is not so hard. In this case, the image regions are slightly darker. Therefore, we expect that the image contrast of ROI images is one of the important factors in classifying liver cirrhosis. In order to highlight the image contrast of ROI images, we adopted the approach of enhancing the image contrast of the ROI images. The image quality was expected to be better when using the image correction methods of tone curves (e.g., [13]). We hoped that the improvement in the image quality would lead to a decrease in the average error rate in liver cirrhosis classification.

In the experiments, we used grey-level transformation functions, i.e., tone curves, for image quality improvement. In this study, in order to enhance the image contrast of ROI images, we used two line-type tone curves and one curved line-type tone curve. Furthermore, we also expected that the darker regions of the ROI images would have better features as well as lighter regions. In order to implement this, we used the inverses of the tone curves. In the experiments, we also used three types of inverses of tone curves. In this paper, we examined the effect of the changes in image quality with image correction methods classifying cirrhosis of the liver on B-mode ultrasound images. The experimental results show the effectiveness of the image correction methods in improving the image quality of ROI images. By enhancing the image contrast of the ROI images, the image quality improved, and thus the generalization ability of the CNN also improved.

In the Discussion section, classical classifiers, *k*-NN (*k* = 1, 3, 5), SVM, LDA, and RF, as well as the transfer learning method, VGG16, are compared to investigate the effects of the proposed method. The *k*-NN (nearest neighbour) classifier [14,15] classifies a test sample based on *k* nearest neighbour samples. The decision is basically made by *k* majority votes. In particular, the nearest neighbour classifier (*k* = 1) is very well known and used in the pattern recognition field and in ultrasound liver classification. The SVM (support vector machine) [16] is known as an effective classifier and it is frequently used for medical imaging. The basic idea of SVM is to find a hyperplane with a margin that maximizes the distance to each sample. The LDA (linear discriminant analysis) classifier [14,15] classifies a test sample based on the statistics, mean vectors, and the same covariance matrix for each class. The RF (random forest) [17] is one of the ensemble learning approaches which are based on decision trees. The RF classifies a test sample based on the multiple outputs performed by each decision tree. The RF with multiple decision trees was expected avoid over-training. The VGG16 [18] is one of the transfer learning approaches. A VGG16 such as CNN (convolution neural network) only requires combinations of inputs or the image itself and its class label. Transfer learning was expected for a small training sample size problem such as one with the current dataset.

#### **2. Materials and Methods**

The generalization ability of CNNs has been improved by adding more layers to deeply train the networks [19,20]. We hoped that the deep nets would improve the generalization ability for this difficult cirrhosis pattern recognition problem. From our preliminary

experimental results, the deeper nets could memorize all the training samples. However, they seemed not to address the unknown test samples. This is known as over-training. In particular, the number of available samples, such as in this set of medical data, is limited. This is known as the small sample size problem in the pattern recognition field [21,22]. With small training sample sizes, deeper nets do not work well. Therefore, we conducted our investigation using shallow nets. The generalization ability of CNNs depends on their network structure and the parameters to be determined. First, we showed the CNN architecture, which is the same as in [6]. Figure 3 shows the structure of the CNN. The input of the CNN is the ROI image of a 32 by 32 size. Firstly, we convolved the ROI image using 32 filters with a 3 by 3 filter size. Furthermore, through 2 by 2 max-pooling, we reduced the ROI image size to a half-sized image, 16 by 16 in size. Secondly, we repeatedly convolved and performed max-pooling in the same manner. Then, we obtained 32 images 8 by 8 in size. Thirdly, we flattened this image into a 2048 (=32 by 8 by 8) dimensional vector. To implement the classification stage, we used a fully connected artificial neural network, with one hidden layer and 2 output neurons for {cirrhosis, healthy}. We explored how performance varied with the number of neurons, with 100 neurons in the final configuration, which was 2048-100-2. The intermediate activation functions were ReLU, and the output layer used softmax. The dropout rate was 0.5, with a batch size of 400. The network was optimized using ADAM over 100 time periods.

**Figure 3.** CNN structure.

Secondly, we showed the image correction methods used. We expected that the image contrast of the ROI images would be important in classifying liver cirrhosis. Thus, we adopted the approach of emphasizing the image contrast of the ROI images. In this study, we used 2 line-type tone curves and 1 curved line-type tone curve as image correction methods [13]. We also used their inverted types. Figures 4 and 5 show the tone curves and the inverse tone curves. In the figures, each of the image correction processing techniques from type 0 to III corresponds to that from type IV to VII. We can see the relationships among the tone curves and the inverse tone curves. The aim of type I to III and type V to VII is to obtain enhanced-contrast images using the parameter values *t* and *γ*. By controlling the parameter values of *t* and *γ*, we expected to obtain enhanced-contrast images. The image correction techniques we used are as follows. The notation *fmax* means the maximum value of the grey level. The grey level was 8. The values ranged from 0 to 255. Then, we could read *fmax* as 255. In each of Figures 4 and 5, the figure is drawn according to *fmax* = 255. For 12 grey levels, the values ranged from 0 to 4095. Then, we could read *fmax* = 4095. The notations *f* and *g* are the intensities of input and output images, respectively.

#### *2.1. Type 0 Original Line*

The type 0 original line is the same for both input and output values. The value of output *g* flows is the same as the input value *f* .

$$\mathbf{g} = f \tag{1}$$

This means that the intensities of the input and the output images are the same.

#### *2.2. Type I Linear Line*

The type I linear line behaves in the same manner as a linear line except for having a lower threshold point. When pixel values are less than threshold *t*, the intensity values are zero. This means they turn black and will be ignored.

$$\mathbf{g} = \begin{cases} \ 0, & f < t \\ \ f, & \text{otherwise} \end{cases} \tag{2}$$

We could vary the threshold value *t* from 0 to *fmax*. If the value of *t* is 0, this means that the intensities of the input and the output images are the same. The higher the value of *t* is, the larger the dark regions are.

**Figure 4.** Tone curves.

**Figure 5.** Inverse of tone curves.

#### *2.3. Type II Linear Line*

The type II line is similar to the type I line. When it becomes equal to or greater than a threshold, the values of the intensity linearly increase. Otherwise, the intensities are zero.

$$\mathbf{g} = \begin{cases} \mathbf{0}, & f < t \\ \ f\_{\text{max}}(f - t) / (f\_{\text{max}} - t), & \text{otherwise} \end{cases} \tag{3}$$

We could vary the threshold value *t* from 0 to *fmax*. If the value of *t* is 0, the intensities of the input and the output images are the same. These are the same as those for type I. The higher the value of *t* is, the larger the dark regions are. At the same time, the contrast of the image is enhanced.

#### *2.4. Type III Gamma Curve*

The gamma curve is one of the curved line-type tone curves. It is not a linear line but a non-linear curve.

$$\mathbf{g} = f\_{\text{max}} (\frac{f}{f\_{\text{max}}})^{\frac{1}{\gamma}} \tag{4}$$

We could control the degree of non-linearity with the value of *γ*. When the value of *γ* is equal to 1, the intensities of the input and the output images are the same. If the value of *γ* is less than 1, the image is darker. Otherwise, the image is brighter.

#### *2.5. Type IV Inverse of Linear Line*

The type IV inverse of linear line behaves in the same manner as a linear type 0 line. The value of output *g* flows is the same as the inverse of the input value *f* .

$$
\mathbf{g} = f\_{\text{max}} - f \tag{5}
$$

This means that the intensities of the input and the output images are inverse. The darker regions become lighter. By contrast, the lighter regions become darker.

#### *2.6. Type V Inverse of Linear Line*

The type V inverse of linear line behaves in the same manner as the inverse of a linear line except for values larger than a threshold, where the values are set to zero. That means that they turn black and will be ignored.

$$\mathbf{g} = \begin{cases} \ f\_{\text{max}} - f\_{\prime} & f < t \\ \mathbf{0}, & \text{otherwise} \end{cases} \tag{6}$$

We could vary the threshold *t* from 0 to *fmax*. If the value of *t* is *fmax*, this means that the intensities of the input and the output images are inverse. The lower the value of *t* is, the larger the dark regions are.

#### *2.7. Type VI Inverse of Linear Line*

The type VI linear line behaves in the same manner as the inverse of a linear line except for a greater threshold. When the pixel values are greater than the threshold, the intensity values are zero. This means that they turn black and will be ignored.

$$\mathbf{g} = \begin{cases} \, \, f\_{\text{max}}(\mathbf{1} - f/t), & f < t \\ \, \mathbf{0}, & \, \text{otherwise} \end{cases} \tag{7}$$

We could vary the threshold *t* from 0 to *fmax*. If the value of *t* is *fmax*, this means that the intensities of the input and the output images are inverse. The lower the value of *t* is, the larger the dark regions are. Simultaneously, the contrast of the image is emphasized.

#### *2.8. Type VII Inverse of Gamma Curve*

The type VII inverse of gamma curve behaves as a type III gamma curve.

$$g = f\_{\max} (\frac{f\_{\max} - f}{f\_{\max}})^{\frac{1}{\gamma}} \tag{8}$$

We could control the degree of non-linearity with the value of *γ*. When the value of *γ* is equal to 1, the intensities of the input and the output images are inverse. If the value of *γ* is less than 1, the image is darker. Otherwise, the image is lighter.

In the experiments, we used ROI images modified by these image correction methods, from type I to VII and including type 0. We tried to investigate the effectiveness with and without the image correction methods on CNNs in terms of the average error rate.

#### **3. Results**

The training used 500 ROI images, with 200 healthy and 300 cirrhosis examples. The effectiveness of the image correction methods was evaluated by comparing the classification error rates, which are the defined as the ratio of misclassified test images to the total number of test images. The holdout method was used to estimate the error rate, as the evaluation images were independent of the training and test images [14,15]. We report the average error rate, which gives an indication of the generalization of the classifier. The algorithm for error rate estimation is shown in Figure 6. In each trial, the 500 ROI images were split into 400 training (160 normal and 240 cirrhosis) and 100 test (40 normal and 60 cirrhosis) images. The training and test images were then modified using the tone curves as described above. The CNN was trained using the training images and the error rate was computed using the test images. This process was repeated 10 times with different random splits of the training and test images to estimate the average error rate and 95% confidence interval.

We conducted the liver cirrhosis classification experiments with tone curves and inverse tone curves. The following experiments were carried out as Experiment 1 and

Experiment 2. Experiment 1 was conducted for the use of the tone curves. On the other hand, Experiment 2 was conducted for the inverse tone curves.

**Figure 6.** Error rate estimation with the holdout method.

#### *3.1. Experiment 1*

The purpose of the experiment was to investigate the generalization ability of the CNN without and with tone-curved image correction methods of ROI images in terms of the average error rate. For the type I and II linear lines, the values of *t* varied from 20 to 160 every 20. For the type III gamma curve, the values of gamma ranged among 1/10, 1/5, 1/3, 1/2, 1, 2, 3, and 5. Table 1 shows the average error rates of the CNN without and with image correction methods of type I, II, and III. In the table, the upper and the lower values show the average error rate and the width of the 95% confidence interval, respectively. From the results, we chose values which gave the minimum average error rates. Then, the optimal values of type I, II, and III were *t* = 20, *t* = 20, and *γ* = 1/2, respectively. Table 2 shows the details of Table 1. It provides the experimental results for values of *t* and *γ*. When it comes to the use of the image correction methods, type I, II, and III, we see that these methods were slightly superior to the method without image correction, type 0. In particular, the type II method gave the minimum error rate of 31.60%. It was more effective for classifying liver cirrhosis.

**Table 1.** Average error rates of the CNN without (type 0) and with image correction methods of type I, II, and III.



**Table 2.** Detail of the average error rates of the CNN with image correction methods of type I, II, and III.

#### *3.2. Experiment 2*

The purpose of the experiment was to investigate the generalization ability of the CNN without and with inverse tone-curved image correction methods of ROI images in terms of the average error rate. For the type V and VI inverse of linear lines, the values of *t* varied from 80 to 220 every 20. For the type VII inverse of gamma curve, the values of *γ* ranged among 1/10, 1/5, 1/3, 1/2, 1, 2, 3, and 5. Table 3 shows the average error rates of the CNN without and with image correction methods of type IV, V, VI, and VII. From our experimental results, we selected values which gave the minimum average error rates. Then, the optimal values of type V, VI, and VII were *t* = 160, *t* = 200, and *γ* = 1/2, respectively. Table 4 shows details of Table 3. It gives the experimental results for values of *t* and *γ*. Our experimental results showed that the average error rates of type IV, V, VI, and VII were slightly superior to those without (type 0) image correction.

From all the experimental results shown in Tables 1 and 3, the average error rate of the type VI image correction method was the best. Its minimum average error rate was 30.60%. The *p*−value was 0.057681, compared to type 0. This means that type VI is superior to type 0 in terms of *p* = 0.10. However, type VI is not as good in terms of *p* = 0.01 and *p* = 0.05.

**Table 3.** Average error rates of the CNN without (type 0) and with image correction methods of type IV, V, VI, and VII.



**Table 4.** Detail of the average error rates of the CNN with image correction methods of type V, VI, and VII.

#### **4. Discussion**

Enhancing the image contrast of ROI images is considered to lead to improved image quality, at least when considering human viewing. Here, we investigated whether the contrast enhancement would also help reduce classification error rates. Type VI (*t* = 200) showed the best average error rate in our limited experiments. Thus, we tried to observe the modified ROI image with the image correction method of type VI. Figure 7 shows the difference between the original and the modified ROI images for normal and cirrhosis livers. From the appearance of the modified images and the experimental results, we draw the following conclusions:


For the type VI method (*t* = 200), the modified ROI images were much lighter because the modified ROI images were inverted and their intensities of greater than 200 were ignored. The image contrast was slightly enhanced because the gradient of the line was steeper. Through this method of image processing, we could obtain richer features for classifying liver cirrhosis. As a result, the features may be richer and the generalization ability of the CNN may improve.

**Figure 7.** Difference between the original and the modified ROI image for normal and cirrhosis livers.

On the other hand, the generalization ability of the images corrected through the type II (*t* = 20) method yielded the second best average error rate in our limited experiments. For the type II method (*t* = 20), the modified ROI images were much darker because the intensities of less than 100 were ignored. By cutting the darker regions and inclining the gradient of the linear line, we could obtain richer features for classifying liver cirrhosis. Thus, the features may richer and the generalization ability of the CNN may improve.

Furthermore, the effect of the classifiers which are well known and widely used as machine learning was also investigated on original images and modified images (type VI (*t* = 200)). The classifiers were *k*-nearest neighbour, support vector machine (SVM), linear discriminant analysis (LDA), random forest (RF), and transfer learning [23,24]. The same experiments as shown in the Results section were conducted. The feature we used was the image itself. In the experiments, *k*-NN (*k* = 1, 3, 5) [14,15], the linear-type SVM [16], the RF [17] with 100 decision trees, and VGG16 [18] as the transfer learning approach were used. In the VGG16, a fully artificially connected network of the same kind as previously mentioned in the CNN was added. The VGG16 was retrained by 5 out of 20 layers by fine-tuning. That is, the 15 remaining layers were frozen. Table 5 shows the average error rates of the classifiers on original images and modified images. The average error rates of the classifiers except for VGG16 were very poor. They were over 40%. There was almost no difference between the original images and the modified images. These results are in line with our previous findings [5]. In principle, these classifiers cannot avoid using one-dimensional data flattening from two-dimensional image data. They could not use the data as a 2D image. On the other hand, the VGG16 was better. A VGG16 such as the CNN could use the data as a 2D image for image pattern recognition. Slight improvement when using modified images also seems to have occurred with VGG16.


**Table 5.** Average error rates of classifiers on original images and modified images.

Finally, the limitations of the experiments are discussed. From the experimental results, the image correction method with type VI (*t* = 200) for ROIs of liver cirrhosis classification on the CNN was found to work better on our dataset. When it comes to using the transfer learning VGG16, the effectiveness also seemed clear. There is a possibility for attaining an appropriate image quality through a modified method for each classifier and for each dataset. The actual selection of parameters *t* and *γ* for each image quality correction method should be conducted as follows, for example:

**Step 1** Prepare several candidates for *t* or *γ*.


#### **5. Conclusions**

In this paper, we examined the effect on the image quality of the ROI image-byimage enhancement methods on the CNN in classifying a cirrhosis of the liver on Bmode ultrasound images. The experimental results showed the effectiveness of the image enhancement methods in improving the classification of ROI images. By modifying the image contrast of ROI images, the image quality was improved and the generalization ability of the CNN improved. Thus, as the proposed method improves image quality for the

pre-processing of ROI images, it is expected that the methods proposed by researchers such as [9–11] will also be improved. Furthermore, considering the computational cost of tone curves, we should adopt a look-up table instead of a calculation of the tone curve function. In particular, the computational cost of a non-linear function, such as a gamma correction, could be cheap. As we mentioned above, emphasizing the image contrast of ROI images is effective for improving the image quality. This can be carried out through image correction methods using tone curves. We plan to explore other types of image modification methods. One possibility is to highlight both darker and lighter regions of ROI images to improve the image quality. There are many common image correction methods. Therefore, trying these image correction methods should be considered for cirrhosis classification in future work. The validation of the proposed method for another dataset should also be addressed in future work. A CNN different from the network architecture in Figure 3 could further improve the classification performance. Furthermore, considering the network architecture of CNNs is also needed. The transfer learning approach [24] should also be considered for further improvement in liver cirrhosis classification.

**Author Contributions:** Conceptualization, Y.M., R.B.F., Y.F., Y.H. and I.S.; methodology, Y.M. and R.B.F.; software, Y.M.; validation, Y.M., R.B.F., Y.F. and Y.H.; formal analysis, Y.M. and R.B.F.; investigation, Y.M.; resources, Y.F. and I.S.; data curation, Y.F. and I.S.; writing—original draft preparation, Y.M.; writing—review and editing, Y.M. and R.B.F.; visualization, Y.M.; supervision, R.B.F., Y.H., and I.S.; project administration, Y.H. and I.S.; funding acquisition, Y.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Posture Estimation Using Surface Electromyography during Wheelchair Hand-Rim Operations**

**Satoshi Ohashi 1,\* , Akira Shionoya <sup>1</sup> , Keiu Harada <sup>2</sup> , Masahito Nagamori <sup>1</sup> and Hisashi Uchiyama <sup>1</sup>**


**Abstract:** This study examined competitive wheelchairs that facilitate sports participation. They can be moved straight ahead using only one arm. Our designed and developed competitive wheelchairs have a dual hand-rim system. Their two hand-rims, attached to a drive wheel on one side, can be operated simultaneously for straight-ahead movement. Specifically, based on integrated electromyography (iEMG) data calculated from surface electromyography (sEMG), we examined the wheelchair loading characteristics, posture estimation, and effects on body posture during one-arm propulsion movement. The first experiment yielded insights into arm and shoulder-joint muscle activation from iEMG results obtained for two-hand propulsion and dual hand-rim system propulsion. Results suggest that muscle activation of one arm can produce equal propulsive force to that produced by two arms. The second experiment estimated the movement posture from iEMG during one-arm wheelchair propulsion. The external oblique abdominis is particularly important for one-arm wheelchair propulsion. The iEMG posture estimation validity was verified based on changes in the user body axis and seat pressure distribution. In conclusion, as confirmed by iEMG, which is useful to estimate posture during movement, one-arm wheelchair use requires different muscle activation sites and posture than when using two arms.

**Keywords:** parasports; assistive technology; competitive wheelchair; sEMG; iEMG; muscle activation; seat pressure

#### **1. Introduction**

As audiences across the world tuned in to watch the Paralympic Games, they saw athletes using impressive para-sport equipment such as high-technology wheelchairs, prosthetic limbs, and other assistive technology [1–4]. For sports played by people with disabilities, research and development of sports equipment according to the events, the sites and degrees of disability, physique, and other conditions are accelerating worldwide [5–11]. Although the characteristics of such research and development are rarely disclosed to the public, the results are expected to be developed eventually into commercial products [12,13].

The research and development for competitive wheelchairs has also considered lighter and stronger materials such as aluminum, titanium, and carbon [14]. Differences in frame materials and hand-rim shape alone have led to great differences in mechanical workload and exercise physiology [15–17]. Furthermore, competition-specific sports wheelchairs must be configured and adapted in numerous ways to suit an athlete's physical impairments and to improve performance and comfort. Physical disability types include limb deficiency, impaired muscle power, impaired range of movement, ataxia, and athetosis. Wheelchair hand-rim operation requires arm and hand muscle function. Individuals with disabilities in one arm or hand exhibit markedly lower performance than users of two arms [18]. Conventionally used manual wheelchairs require long and intensive periods of use and

**Citation:** Ohashi, S.; Shionoya, A.; Harada, K.; Nagamori, M.; Uchiyama, H. Posture Estimation Using Surface Electromyography during Wheelchair Hand-Rim Operations. *Sensors* **2022**, *22*, 3296. https:// doi.org/10.3390/s22093296

Academic Editors: James F. Rusling and Andrea Cataldo

Received: 4 February 2022 Accepted: 22 April 2022 Published: 25 April 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

control of two arms until proper hand-rim operation can be achieved. Therefore, some adaptation is necessary for users with asymmetrical arm use. Moreover, propelling a wheelchair with one arm during competition is an even more difficult task when pushing a hand-rim with two hands. Some such individuals with disabilities often require onearm drive wheelchairs [19,20]. Nevertheless, only a few reports of the relevant literature describe studies related to one-arm driven wheelchairs; even these reports have contents related to driving systems or persons with hemiplegic disabilities [21–26]. No research report describes such a system for competitive wheelchairs.

This study was conducted to examine a competitive wheelchair for practical application, operable by moving straight ahead with only one arm, which will allow participation in sports. The wheelchair with a dual hand-rim system we developed has two hand-rims attached to the drive wheel on one side [27,28]. A user moves in a straight line by gripping both hand-rims simultaneously with one hand. Alternatively, a user can turn using a single hand-rim. The wheelchair presents the added benefit that it can be operated with one arm, even for movements during competition, to execute sudden stops and starts: stop-and-go motion. Such maneuverability enables one-arm movements such as going forwards, backwards, turning, and stopping, which can be done using propulsion with two arms. The results can support many possibilities: for example, if a competitor who still has function of two arms uses this wheelchair, then the wheelchair could be used to drive straight ahead using the same sports motion as that of an athlete holding a racket in one hand while operating the dual hand-rim; also, athletes who can move only one arm can operate the wheelchair just as one might operate a regular dual-arm wheelchair. Sports motions of this kind were not possible with conventional hand-rim operation. The benefits obtained from this study can support and improve athletes' operations and competitive skills.

Despite the benefits explained above, no report of the relevant literature has described a study of competitive wheelchairs equipped with dual hand-rim systems. Consequently, many unanswered questions remain about loads on the body during hand-rim operation and the loads' effects on user posture. Once these issues are resolved, it will clarify the conditions of compatibility with the competition and will also allow for training according to the site and degree of disability. Our competitive wheelchair with dual hand-rim system will have a different hand-rim operation than existing competition wheelchairs. Therefore, as a preliminary step in the investigation under subdivided conditions, the first step is necessary to clarify the differences in general muscle activation and movement posture by comparing one-arm with two-arm during the simplest straight-ahead movement. This is because differences in seat height and axle position are known to affect propulsion efficiency, stability, and wheelchair manageability [29], and estimation of biomechanical parameters during straight-line driving and evaluation of the operability of a competitive wheelchair are important issues for users with disabilities [30]. An earlier study used findings from surface electromyography (sEMG) of users to characterize and elucidate wheelchair propulsion, because a correlation exists between sEMG data and muscle strength [31]. The strength of the primary muscles in the user's upper limb musculature strongly influence the propulsive force transmitted to the wheelchair [32]. In the case of onearm operation, there should be a difference in muscle activation between the left and right sides. In other words, this difference in muscle activation is related to changes in the motion of the upper limb. As described herein, we thought that by identifying the site of muscle activation by sEMG it would be possible to estimate approximate movement posture. This knowledge is important for the development of new competitive wheelchairs and their use in sports. Many methods have been proposed for detecting motion posture, including two-dimensional and three-dimensional video analyses based on computer vision and motion capture technology [33,34]. However, our methods do not require great resources of equipment, cost, or time, in addition to burdensome preparation for experiments.

As described herein, we present an estimate of movement posture during one-arm operation of a wheelchair based on changes in integrated electromyography (iEMG) data calculated from sEMG data. The first experiment is designed to elicit insights into differ-

ences in muscle activation of a user's arm and shoulder joint muscles when using one arm and when using two arms with competitive wheelchair equipped with a dual hand-rim system. The experimentally obtained muscle activation results suggest that one arm use can produce equal propulsive force to that produced using two arms. The second experiment was conducted using iEMG data to estimate the movement posture during wheelchair propulsion with one arm. The external oblique abdominis play an important role in producing the movement posture for wheelchair propulsion force with one arm. Finally, using iEMG data, the posture estimation was verified by assessing the amount of change in the user's body axis and seat pressure distribution. Results demonstrate that one-arm operation uses different muscle activation sites than the body posture used for propulsion by two-arm, indicating the body posture differences estimated from surface EMG.

#### **2. Materials and Methods**

This study was conducted to develop competitive wheelchairs that will allow participating in sports by facilitating straight-ahead movement with one arm. For example, our wheelchair is intended for use in situations where the athlete grips a racket in one hand and operates the hand-rim with the other hand, or when the athlete must propel the competitive wheelchair with only one arm. However, the differences in muscle activation and body posture between one-arm and two-arm operations have not been clarified. Once these issues are resolved, it will be possible to determine the muscles needed to strengthen one-arm operation and the approximate range of adaptability to the site and degree of disability. Preliminary test results confirmed great differences in body postures during straight-line motion with one-arm propulsion. In the case of one-arm operation, there should be a difference in muscle activation between the left and right sides. In addition, this difference in muscle activation is related to changes in the motion of the upper limb. We thought that by identifying the site of muscle activation by sEMG, it would be possible to estimate approximate movement posture. The first experiment was conducted to elicit insights into differences in muscle site activity from iEMG results obtained for one-arm and two-arm propulsion of a wheelchair equipped with a dual hand-rim system. Based on results of the first experiment, the second experiment tested whether body posture can be estimated from iEMG data during one-arm wheelchair propulsion based on the results of seat pressure distribution and changes in body axis.

#### *2.1. Competitive Wheelchair with a Dual Hand-Rim System*

Figure 1 shows a competitive wheelchair with a dual hand-rim system designed for our study. Such wheelchairs are driven by a double-ring drive shaft structure [27,28]. Two wheelchairs are used for this research: (a) one with a right-hand drive with a camber angle and (b) one with a left-hand drive without a camber angle. These wheelchairs are de-signed to be interchangeable between the right and left sides by reassembly of parts. Figure 1c shows a driving force transmission axle (DFTA) and universal joint that were developed to transmit the driving force from the operation of the outer hand-rims to the oppo-site drive wheel in a competitive wheelchair with a camber angle. The material used for the DFTA is standard internal iron with specific gravity of 7.87 g/cm<sup>3</sup> and Young's modulus of 192.08 GP. A steel universal joint of the same standard is attached to the DFTA on each side. This universal joint has a structure in which the rotational transmission speed is not constant with the rotation angle, but which repeats the speed increase and decrease in a 180-degree rotation cycle. Therefore, by installing two universal joints with rotational phases that are 90 degrees apart, the rotational speed to the opposite drive wheel can be set to a constant speed. The one-arm drive wheelchair developed for bowling competition, as shown in Figure 1b, has a structure incorporating no camber angle because it must specialize in straight-line driving based on the movement characteristics associated with competition. These two competitive wheelchairs, each of which can be driven with one arm, were manufactured by Ox Engineering Co., Ltd. (Funabashi, Japan).

**Figure 1.** Competitive wheelchair with the dual hand-rim system: (**a**) right-hand drive with a camber angle, (**b**) left-hand drive without a camber angle, and (**c**) a double-ring drive shaft structure. **Figure 1.** Competitive wheelchair with the dual hand-rim system: (**a**) right-hand drive with a camber angle, (**b**) left-hand drive without a camber angle, and (**c**) a double-ring drive shaft structure.

competition, as shown in Figure 1b, has a structure incorporating no camber angle because it must specialize in straight-line driving based on the movement characteristics associated with competition. These two competitive wheelchairs, each of which can be driven with one arm, were manufactured by Ox Engineering Co., Ltd. (Funabashi, Japan).

Following is a description of straight-line operation of the wheelchair using one arm. The x and y shown as markers in Figure 1a represent the two hand-rims. Regarding the two-handed rims attached to the right-hand drive side, the outer hand-rim shown in "x" operates the opposite left-hand drive wheel. The inner hand-rim shown in "y" operates the right-hand-drive wheel. When the dual hand-rims are operated simultaneously, the driving force is transmitted to both drive wheels. Thereby, the vehicle can move straight ahead. The manual propulsion action necessitates that one arm and hand exert repetitive force to the dual hand-rims accordingly. Following is a description of straight-line operation of the wheelchair using one arm. The x and y shown as markers in Figure 1a represent the two hand-rims. Regarding the two-handed rims attached to the right-hand drive side, the outer hand-rim shown in "x" operates the opposite left-hand drive wheel. The inner hand-rim shown in "y" operates the right-hand-drive wheel. When the dual hand-rims are operated simultaneously, the driving force is transmitted to both drive wheels. Thereby, the vehicle can move straight ahead. The manual propulsion action necessitates that one arm and hand exert repetitive force to the dual hand-rims accordingly.

#### *2.2. Participants 2.2. Participants*

The research participants in these experiments gave informed consent to serve as a study subject in the experiment. This study and use of the experimentally obtained data were approved by the Ethics Committee of the Nagaoka University of Technology (H30- 1, H30-2). Research participants in the first experiment were two healthy men (173.5 ± 1.5 cm height; 70.5 ± 2.5 kg weight). Research participants in the second experiment were seven healthy men (173.1 ± 4.2 cm height; 65.4 ± 5.6 kg weight). All research participants, for whom the right hand was dominant had experienced adequate training in wheelchair manipulation. We want to explain changes and differences in muscle activity in one arm and two arms for a competitive wheelchair with a dual hand-rim system. Accordingly, we recruited healthy athletes as research participants. We understand that the inclusion of able-bodied athletes is a limitation affecting the generalizability of the study results. Research participants with a disability might show clear differences in muscle activity between one-arm and two-arm propulsion. The research participants adjusted the footrest and seat of the wheelchair before starting the experiments. The seating was secured by placing a towel between the gap on either side of the seat surface. The research participants in these experiments gave informed consent to serve as a study subject in the experiment. This study and use of the experimentally obtained data were approved by the Ethics Committee of the Nagaoka University of Technology (H30-1, H30-2). Research participants in the first experiment were two healthy men (173.5 ± 1.5 cm height; 70.5 ± 2.5 kg weight). Research participants in the second experiment were seven healthy men (173.1 ± 4.2 cm height; 65.4 ± 5.6 kg weight). All research participants, for whom the right hand was dominant had experienced adequate training in wheelchair manipulation. We want to explain changes and differences in muscle activity in one arm and two arms for a competitive wheelchair with a dual hand-rim system. Accordingly, we recruited healthy athletes as research participants. We understand that the inclusion of able-bodied athletes is a limitation affecting the generalizability of the study results. Research participants with a disability might show clear differences in muscle activity between one-arm and two-arm propulsion. The research participants adjusted the footrest and seat of the wheelchair before starting the experiments. The seating was secured by placing a towel between the gap on either side of the seat surface.

#### *2.3. Experiment Protocol*

#### *2.3. Experiment Protocol*  2.3.1. First Experiment

2.3.1. First Experiment The purpose of this experiment is to use iEMG results to gain insight into differences in arm and shoulder joint muscle activation of a user during one-arm and two-arm use of our competitive wheelchair equipped with a dual hand-rim system. The driving force provided with the push rims is defined as the delivery of propulsion to the wheelchair. The wheelchair speed is related directly to the magnitude and frequency of the propulsive action. In fact, competitive wheelchair propulsion techniques are divisible into two phases: drive and recovery [35,36]. The most important factor affecting this propulsive The purpose of this experiment is to use iEMG results to gain insight into differences in arm and shoulder joint muscle activation of a user during one-arm and two-arm use of our competitive wheelchair equipped with a dual hand-rim system. The driving force provided with the push rims is defined as the delivery of propulsion to the wheelchair. The wheelchair speed is related directly to the magnitude and frequency of the propulsive action. In fact, competitive wheelchair propulsion techniques are divisible into two phases: drive and recovery [35,36]. The most important factor affecting this propulsive force is the drive phase. Five measurement points are presented in Figure 2a: (1) flexor digitorum profundus (pinky side), (2) triceps brachii, (3) deltoid, (4) pectoralis major, and (5) latissimus dorsi. These muscles were selected for their well-known contribution to the drive and recovery phases. Figure 2b is a schematic diagram depicting the experiment. The procedure used for this experiment is the following.

force is the drive phase. Five measurement points are presented in Figure 2a: (1) flexor digitorum profundus (pinky side), (2) triceps brachii, (3) deltoid, (4) pectoralis major, and (5) latissimus dorsi. These muscles were selected for their well-known contribution to the drive and recovery phases. Figure 2b is a schematic diagram depicting the experiment.

1. As shown in Figure 3a,b, research participants hold a dumbbell with a load of 5 kg for 30 s before running. All subjects are loaded to produce identical muscle fatigue because it inhibits the possibility of inducing muscle fatigue bias through individual

force is the drive phase. Five measurement points are presented in Figure 2a: (1) flexor digitorum profundus (pinky side), (2) triceps brachii, (3) deltoid, (4) pectoralis major, and (5) latissimus dorsi. These muscles were selected for their well-known contribution to the drive and recovery phases. Figure 2b is a schematic diagram depicting the experiment.

2. The start is made from a stationary position. During running, the hand-rim operation is performed once per second (for one rotation) for eight repetitions (for eight rotations) at full speed. The minimum number of cycles required for the wheelchair propulsion to reach its maximum value from the stationary position is assumed because

1. As shown in Figure 3a,b, research participants hold a dumbbell with a load of 5 kg for 30 s before running. All subjects are loaded to produce identical muscle fatigue because it inhibits the possibility of inducing muscle fatigue bias through individual

3. After completing running, the research participants take sufficient rest to recover

2. The start is made from a stationary position. During running, the hand-rim operation is performed once per second (for one rotation) for eight repetitions (for eight rotations) at full speed. The minimum number of cycles required for the wheelchair propulsion to reach its maximum value from the stationary position is assumed because

4. Steps 1–3 are performed alternately: three times in the case of one-handed running

many competitions necessitate rapid acceleration in fewer cycles.

many competitions necessitate rapid acceleration in fewer cycles.

and three times in the case of two-handed running.

The procedure used for this experiment is the following.

The procedure used for this experiment is the following.

*Sensors* **2022**, *22*, 3296 5 of 18

differences.

from fatigue.

differences.

**Figure 2.** Measurement position of sEMG and schematic diagram I: (**a**) five measurement points and (**b**) schematic diagram of the experiment. **Figure 2.** Measurement position of sEMG and schematic diagram I: (**a**) five measurement points and (**b**) schematic diagram of the experiment.

1. As shown in Figure 3a,b, research participants hold a dumbbell with a load of 5 kg for 30 s before running. All subjects are loaded to produce identical muscle fatigue because it inhibits the possibility of inducing muscle fatigue bias through individual differences. (**a**) (**b**) **Figure 2.** Measurement position of sEMG and schematic diagram I: (**a**) five measurement points and (**b**) schematic diagram of the experiment.

ture during wheelchair propulsion with one-arm operation. Based on the hypothesis produced in light of the first experiment results, this experiment also measured the seat pres-**Figure 3.** Posture of research participants under loading: (**a**) posture of research participants holding the dumbbells Part 1 and (**b**) posture of research participants holding the dumbbells Part 2. **Figure 3.** Posture of research participants under loading: (**a**) posture of research participants holding the dumbbells Part 1 and (**b**) posture of research participants holding the dumbbells Part 2.

The purpose of this experiment was to use iEMG data to estimate the movement pos-


#### 2.3.2. Second Experiment

The purpose of this experiment was to use iEMG data to estimate the movement posture during wheelchair propulsion with one-arm operation. Based on the hypothesis produced in light of the first experiment results, this experiment also measured the seat pressure distribution and three-dimensional movement of the upper limb synchronized with the sEMG findings. These results corroborate evidence obtained for body posture effects during one-arm propulsion of the wheelchair. The wheelchair used for this experiment had no camber angle, as shown in Figure 1b because the experiment specifically examines the operating posture during straight-line operation. Three measurement points are shown in Figure 4a: (1) erector-spinae, (2) external oblique abdominis, and (3) triceps brachii. Not all measurement points are shown in Figure 4a, but six measurement points were used for these surface EMGs because they were prepared for the left and right sides of the body. These muscles were selected for their well-known contributions to the drive phase [37]. A schematic diagram of the experiment is presented in Figure 4b. The procedures used for this experiment were the same as those described in 1–4 of Section 2.4.1. body. These muscles were selected for their well-known contributions to the drive phase [37]. A schematic diagram of the experiment is presented in Figure 4b. The procedures used for this experiment were the same as those described in 1–4 of Section 2.4.1.

had no camber angle, as shown in Figure 1b because the experiment specifically examines the operating posture during straight-line operation. Three measurement points are shown in Figure 4a: (1) erector-spinae, (2) external oblique abdominis, and (3) triceps brachii. Not all measurement points are shown in Figure 4a, but six measurement points were used for these surface EMGs because they were prepared for the left and right sides of the

*Sensors* **2022**, *22*, 3296 6 of 18

**Figure 4.** Measurement position of sEMG and schematic diagram II: (**a**) three measurement points and (**b**) schematic diagram showing the experiment. **Figure 4.** Measurement position of sEMG and schematic diagram II: (**a**) three measurement points and (**b**) schematic diagram showing the experiment.

#### *2.4. Data Recording and Analysis 2.4. Data Recording and Analysis*

#### 2.4.1. Measuring Instruments 2.4.1. Measuring Instruments

The following is a description of the measuring instruments used for data collection, which included a surface electromyogram (PolymatePro MP6000 biological signal system; Miyuki Giken Co., Ltd., Bunkyo, Japan), a pressure-detecting sheet (SR Soft Vision; Sumitomo Riko Co., Ltd., Nagoya, Japan), and a three-axis acceleration sensor (MyBeat; Union Tool Co., Tokyo, Japan). The sampling frequencies used for sEMG were 20–2000 Hz, with impedance of 250 GΩ and 24 input channels. For EMG data, surface electrodes were attached to the measurement position of the agonist muscle necessary for hand-rim operation. Details of its position are presented in the experiment protocol. The electrical signals obtained from the electrodes were recorded using a biological signal system (PolymatePro MP6000; Miyuki Giken Co., Ltd.) to a PC connected to the system at a sampling frequency of 1 kHz. The 450 × 450 mm pressure-detecting sheet included 256 pressure sensor elements. The sampling frequency was 5 Hz. The measurement range of pressure values was 0–200 mmHg. The right side of the lateral direction of the wheelchair was the X-axis positive direction. The front direction was the Y-axis positive direction. The three-axis acceleration sensor measured body axis movement. The right side of the lateral direction of the wheelchair was the X-axis positive direction. The vertically upward direction was the Yaxis positive direction. The front direction was the Z-axis positive direction. The sampling frequency was 128 Hz. The acceleration range was ±4 G. The following is a description of the measuring instruments used for data collection, which included a surface electromyogram (PolymatePro MP6000 biological signal system; Miyuki Giken Co., Ltd., Bunkyo, Japan), a pressure-detecting sheet (SR Soft Vision; Sumitomo Riko Co., Ltd., Nagoya, Japan), and a three-axis acceleration sensor (MyBeat; Union Tool Co., Tokyo, Japan). The sampling frequencies used for sEMG were 20–2000 Hz, with impedance of 250 GΩ and 24 input channels. For EMG data, surface electrodes were attached to the measurement position of the agonist muscle necessary for hand-rim operation. Details of its position are presented in the experiment protocol. The electrical signals obtained from the electrodes were recorded using a biological signal system (PolymatePro MP6000; Miyuki Giken Co., Ltd.) to a PC connected to the system at a sampling frequency of 1 kHz. The 450 × 450 mm pressure-detecting sheet included 256 pressure sensor elements. The sampling frequency was 5 Hz. The measurement range of pressure values was 0–200 mmHg. The right side of the lateral direction of the wheelchair was the X-axis positive direction. The front direction was the Y-axis positive direction. The three-axis acceleration sensor measured body axis movement. The right side of the lateral direction of the wheelchair was the X-axis positive direction. The vertically upward direction was the Y-axis positive direction. The front direction was the Z-axis positive direction. The sampling frequency was 128 Hz. The acceleration range was ±4 G.

#### 2.4.2. Data Analysis

2.4.2. Data Analysis Muscle activation, which is described as the linear envelope of the EMG signal [38], has been studied quantitatively using iEMG [39]. In an earlier study, sEMG findings of wheelchair users were used as indicators of wheelchair propulsion because a correlation exists between sEMG data and muscle strength [31]. For this study, sEMG measurements were taken at the arm, shoulder, and trunk muscles related to hand-rim operations. Research participants were prepared for placement of EMG electrodes at the measurement position by wiping the skin with alcohol and by lightly abrading it. Next, sufficient electrode paste (Ten20 Conductive; Weaver and Co., Aurora, CO, USA) was applied inside the surface Ag/AgCl EMG electrodes (MA-C001-15; Fukuda M-E Kogyo Co., Ltd., Muscle activation, which is described as the linear envelope of the EMG signal [38], has been studied quantitatively using iEMG [39]. In an earlier study, sEMG findings of wheelchair users were used as indicators of wheelchair propulsion because a correlation exists between sEMG data and muscle strength [31]. For this study, sEMG measurements were taken at the arm, shoulder, and trunk muscles related to hand-rim operations. Research participants were prepared for placement of EMG electrodes at the measurement position by wiping the skin with alcohol and by lightly abrading it. Next, sufficient electrode paste (Ten20 Conductive; Weaver and Co., Aurora, CO, USA) was applied inside the surface Ag/AgCl EMG electrodes (MA-C001-15; Fukuda M-E Kogyo Co., Ltd., Nagareyama, Japan) to slightly overfit it. Then the electrode was placed onto the measurement position and pressed firmly. The electrodes were secured with surgical tape to minimize displacement during movement. A ground electrode was placed on a bony site over the iliac bone. For this study, sEMG data from each muscle were collected at a sampling frequency of 1 kHz during eight cycles for each research participant. One cycle defined here is one stroke of the hand-rim operation (to recovery phase from drive phase). The raw sEMG data were

exported (BIMUTAS II; Kissei Comtec Co., Ltd., Matsumoto, Japan) for signal analysis and post-acquisition processing. A high-pass filter was used to remove noise. The integrated EMG (iEMG) was calculated using full-wave rectification smooth of sEMG data for each muscle for each research participant. In this study, the integrated value per second of iEMG measured at rest was normalized by the average value per stroke of iEMG measured during eight strokes of driving. These iEMG data quantitatively represent the total work-load of electrical activity of the muscles, meaning that the data quantify the amount of muscle activity for one drive during wheelchair operation.

The inclination of the body axis during wheelchair propulsion was measured using a three-axis acceleration sensor mounted in the center of the chest. The sampling frequency was 128 Hz. First, the sensor values in the three-axis (X-axis, Y-axis, Z-axis) were recorded when the research participants were held stationary in a competitive wheelchair for 10 s. We adopted the average of these values as our reference value at rest. Next, the difference between the results of all eight cycles and the reference value was then determined. Finally, we integrated the difference values for each cycle interval. The cycling interval is one second. However, individual differences occur. The samplings per cycle were adjusted by individually checking the peak values of the measured data. The result is presented as the inclination of the body axis during the wheelchair propulsion cycle. The reason for grouping them into eight cycles is to synchronize them with the iEMG results.

The seat pressure distribution during wheelchair propulsion was measured using a pressure-detecting sheet. The sampling frequency was 5 Hz. The sensor elements in a pressure-detecting sheet are arranged longitudinally and horizontally (16 × 16). The sheet was divided into nine grids because we wanted to classify the direction of the pressure distribution horizontally, vertically, and diagonally. The total sensor elements inherent in one grid were set to 32 (6 × 6). First, the sensor values in the seat pressure were recorded when the research participants were held stationary in a competitive wheelchair for 10 s. We adopted the average in the one grid as each reference value of static seat pressure. Next, the difference between the results of eight cycles in each grid and the reference values was then determined. Finally, we add the difference values for eight cycles. The result was recorded as the seat pressure distribution during the wheelchair propulsion cycle. These values are used as validation data for the body posture estimation.

#### 2.4.3. Statistical Analysis

Data were analyzed using software (RStudio, ver. 1.4.1106: GNU Affero General Public License). For the first experiment, iEMG data from five muscles were analyzed: flex-or digitorum profundus (pinky side), triceps brachii, deltoid, pectoralis major, and latissimus dorsi. For the second experiment, iEMG data from six muscles were analyzed: erector spinae (right and left side), external oblique abdominis (right and left side), and triceps brachii (right and left side). Shapiro–Wilk's test revealed normality of the iEMG data in each experiment [40]. To assess significance of differences, a Wilcoxon signed-rank test by non-parametric data was selected for the corresponding two groups (one arm and two arms) [41]. The Friedman test was selected for the four groups of the corresponding non-parametric data. Furthermore, differences in means between the two groups in the four groups of data were selected with the Tukey honestly significant difference test. All thresholds for significance were set at the *p* < 0.05 level of confidence. The effect size was based on Cohen's report [42].

#### **3. Results**

#### *3.1. Results of iEMG for Muscle Activation during One-Arm Propulsion*

The graph portrayed in Figure 5 depicts the transition per stroke for one-arm and two-arm driving up to eight strokes. The average of iEMG data obtained for all trials is shown. The results are iEMG data for (a) flexor digitorum profundus (pinky side), (b) triceps brachii, (c) deltoid, (d) pectoralis major, and (e) latissimus dorsi. Error bars in the figure represent the standard error.

**3. Results** 

figure represent the standard error.

*3.1. Results of iEMG for Muscle Activation during One-Arm Propulsion* 

The graph portrayed in Figure 5 depicts the transition per stroke for one-arm and two-arm driving up to eight strokes. The average of iEMG data obtained for all trials is shown. The results are iEMG data for (a) flexor digitorum profundus (pinky side), (b) triceps brachii, (c) deltoid, (d) pectoralis major, and (e) latissimus dorsi. Error bars in the

**Figure 5.** iEMG calculation results for eight strokes performed with one-arm and two-arm driving: (**a**) flexor digitorum profundus (pinky side), (**b**) triceps brachii, (**c**) deltoid, (**d**) pectoralis major, and (**e**) latissimus dorsi. **Figure 5.** iEMG calculation results for eight strokes performed with one-arm and two-arm driving: (**a**) flexor digitorum profundus (pinky side), (**b**) triceps brachii, (**c**) deltoid, (**d**) pectoralis major, and (**e**) latissimus dorsi.

As shown in Figure 6, a Wilcoxon signed-rank test using non-parametric data found significant differences in muscle activation between one-arm and two-arm propulsion were found for the triceps brachii (*p* = 0.0000, *r* = 1.04), deltoid (*p* = 0.0013, *r* = 0.80), and pectoralis major (*p* = 0.0002, *r* = 0.95), with one-arm data indicating greater muscle activation for the three previously described muscles compared to propulsion using two arms. The flexor digitorum profundus deltoid (*p* = 0.1167, *r* = 0.39) and latissimus dorsi (*p* = As shown in Figure 6, a Wilcoxon signed-rank test using non-parametric data found significant differences in muscle activation between one-arm and two-arm propulsion were found for the triceps brachii (*p* = 0.0000, *r* = 1.04), deltoid (*p* = 0.0013, *r* = 0.80), and pectoralis major (*p* = 0.0002, *r* = 0.95), with one-arm data indicating greater muscle activation for the three previously described muscles compared to propulsion using two arms. The flexor digitorum profundus deltoid (*p* = 0.1167, *r* = 0.39) and latissimus dorsi (*p* = 0.8603, *r* = 0.04)

0.8603, *r* = 0.04) muscle activation iEMG data were not significantly different between data

obtained for one arm and for two arms.

muscle activation iEMG data were not significantly different between data obtained for one arm and for two arms. *Sensors* **2022**, *22*, 3296 10 of 18

**Figure 6.** Distribution of iEMG results obtained at each muscle site with one-arm and two-arm driving. **Figure 6.** Distribution of iEMG results obtained at each muscle site with one-arm and two-arm driving.

#### *3.2. Results of iEMG for Body Posture Estimated during One-Arm Propulsion*  3.2.1. Differences in Muscle Activation Because of Different Driving Patterns *3.2. Results of iEMG for Body Posture Estimated during One-Arm Propulsion* 3.2.1. Differences in Muscle Activation Because of Different Driving Patterns

The iEMG results obtained for the erector spinae, external oblique abdominis, and the triceps brachii of the research participants' collaborators in the experiment are presented in Figure 7. The graph presents iEMG findings obtained up to a total of eight strokes with one-arm and two-arm driving, divided into those for the left-side measurement site (a) and for the right-side measurement site (b), centered on the body axis. It is noteworthy that the iEMG data at each stroke are average values of all trials. The error bars in the figure represent the standard error. Results for the distribution of iEMG data at each muscle site are presented in Figure 8 with a box plot. As shown in Figure 8, nonparametric tests were performed using the Friedman test (*p* < 0.05) for four groups of target muscle sites: left side of one arm, left side of two arms, right side of one arm, and right side of two arms. Results showed differences in the representative values among the four groups: erector spinae was *p* = 0.0002, external oblique abdominis was *p* = 0.0002, and The iEMG results obtained for the erector spinae, external oblique abdominis, and the triceps brachii of the research participants' collaborators in the experiment are presented in Figure 7. The graph presents iEMG findings obtained up to a total of eight strokes with one-arm and two-arm driving, divided into those for the left-side measurement site (a) and for the right-side measurement site (b), centered on the body axis. It is noteworthy that the iEMG data at each stroke are average values of all trials. The error bars in the figure represent the standard error. Results for the distribution of iEMG data at each muscle site are presented in Figure 8 with a box plot. As shown in Figure 8, nonparametric tests were performed using the Friedman test (*p* < 0.05) for four groups of target muscle sites: left side of one arm, left side of two arms, right side of one arm, and right side of two arms. Results showed differences in the representative values among the four groups: erector spinae was *p* = 0.0002, external oblique abdominis was *p* = 0.0002, and triceps brachii was *p* = 0.0001.

triceps brachii was *p* = 0.0001.

(**c**) **Figure 7.** iEMG calculation results for eight strokes during one-arm and two-arm driving: (**a**) erector **Figure 7.** iEMG calculation results for eight strokes during one-arm and two-arm driving: (**a**) erector spinae, (**b**) external oblique abdominis, and (**c**) triceps brachii.

spinae, (**b**) external oblique abdominis, and (**c**) triceps brachii.

**Figure 8.** Distribution of iEMG results obtained at each muscle site with one-arm and two-arm driving (by left and right side). **Figure 8.** Distribution of iEMG results obtained at each muscle site with one-arm and two-arm driving (by left and right side).

Next, Table 1 presents test results for the difference in means between the two groups for the four groups of data. The Tukey honestly significant difference test was applied. The results obtained for one-arm operation showed a significant difference in muscle activity between the left and right. The results also confirmed that one-arm and two-arm operations produced differences in the erector spinae and external oblique abdominis. Next, Table 1 presents test results for the difference in means between the two groups for the four groups of data. The Tukey honestly significant difference test was applied. The results obtained for one-arm operation showed a significant difference in muscle activity between the left and right. The results also confirmed that one-arm and two-arm operations produced differences in the erector spinae and external oblique abdominis.


**Table 1.** Tukey honestly significant difference test. **Table 1.** Tukey honestly significant difference test.

RO–RT 0.5735 0.1956 0.0000 LO, left side of one arm; LT, left side of two arms; RO, right side of one arm; and RT, right side of two arms.

LO, left side of one arm; LT, left side of two arms; RO, right side of one arm; and RT, right side of

two arms. 3.2.2. Rate the Body Axis and Seat Pressure Distribution

Results obtained during one-arm and two-arm operation and then presented in Figure 9 show changes in the body axis from the three-dimensional acceleration sensor attached to a user's chest for eight cycles. Figure 9a presents results obtained for the lateral direction of the body (X-axis), (b) results obtained for the vertical direction of the body (Y-axis), and (c) results obtained for the front–back direction of the body (Z-axis). Error bars in the figure represent the standard error. Wilcoxon signed-rank test results obtained from non-parametric data indicated significant differences in muscle activation found for the X-axis (*p* = 0.0078, *r* = 0.94), Y-axis (*p* = 0.0078, *r* = 0.94), and Z-axis (*p* = 0.0156, *r* = 0.85). in the figure represent the standard error. Wilcoxon signed-rank test results obtained from non-parametric data indicated significant differences in muscle activation found for the X-axis (*p* = 0.0078, *r* = 0.94), Y-axis (*p* = 0.0078, *r* = 0.94), and Z-axis (*p* = 0.0156, *r* = 0.85).

Results obtained during one-arm and two-arm operation and then presented in Figure 9 show changes in the body axis from the three-dimensional acceleration sensor attached to a user's chest for eight cycles. Figure 9a presents results obtained for the lateral direction of the body (X-axis), (b) results obtained for the vertical direction of the body (Yaxis), and (c) results obtained for the front–back direction of the body (Z-axis). Error bars

*Sensors* **2022**, *22*, 3296 13 of 18

3.2.2. Rate the Body Axis and Seat Pressure Distribution

**Figure 9.** Body axis change results obtained for one-arm and two-arm driving for eight repetitions: (**a**) X-axis, (**b**) Y-axis, and (**c**) Z-axis. **Figure 9.** Body axis change results obtained for one-arm and two-arm driving for eight repetitions: (**a**) X-axis, (**b**) Y-axis, and (**c**) Z-axis.

Results shown in Figure 10 were obtained by dividing 256 seat pressure distribution values with measurement resolution of 16 × 16 into nine sections, then calculating the difference from the seat pressure reference value for each experiment. Thereafter, the results of one arm to two-arm operation were subtracted from the total pressure values for each divided area. The reason for dividing the data into nine parts was to elucidate the direction of the user's center of gravity movement. Figure 10a presents an example of a heat map visualization of the seat pressure distribution at rest, which is the reference value for seat pressure. Figure 10b shows the results of differencing the seat pressure distribution values from one-arm to two-arm operation. Results shown in Figure 10 were obtained by dividing 256 seat pressure distribution values with measurement resolution of 16 × 16 into nine sections, then calculating the difference from the seat pressure reference value for each experiment. Thereafter, the results of one arm to two-arm operation were subtracted from the total pressure values for each divided area. The reason for dividing the data into nine parts was to elucidate the direction of the user's center of gravity movement. Figure 10a presents an example of a heat map visualization of the seat pressure distribution at rest, which is the reference value for seat pressure. Figure 10b shows the results of differencing the seat pressure distribution values from one-arm to two-arm operation.

**Figure 10.** Different seat pressure distribution values for the nine segments: (**a**) an example of a heat map at rest, (**b**) the difference results for seat pressure distribution. **Figure 10.** Different seat pressure distribution values for the nine segments: (**a**) an example of a heat map at rest, (**b**) the difference results for seat pressure distribution.

#### **4. Discussion 4. Discussion**

#### *4.1. Muscle Activation Evaluation Using iEMG during One-Arm Propulsion 4.1. Muscle Activation Evaluation Using iEMG during One-Arm Propulsion*

For EMG data recorded during the first experiment, the EMG data for one stroke of the hand-rim operation were extracted. After full-wave rectification, the iEMG data were calculated. These iEMG data quantitatively represent the total workload of the electrical activity of the muscles, meaning that they represent the amount of muscle activity for one drive during wheelchair operation. The graph presented in Figure 5 depicts the transition per stroke for one-arm and two-arm driving up to eight strokes. It is apparent from this figure that the (a) flexor digitorum profundus, (b) triceps brachii, and (d) pectoralis major, showed a large increase in muscle activity at the first stroke and a monotonic decrease with each increase in the number of strokes. For one-arm driving as well, these muscles indicate that the greatest amount of muscle force is necessary at the start of driving. Subsequently, the load on these muscles during the push phase decreased concomitantly with increasing wheelchair speed. Consequently, muscle activation also decreased concomitantly with the increasing stroke rate. In fact, the (c) deltoid and (e) latissimus dorsi results indicate that the amount of muscle activity decreased at the first stroke and end strokes for both one-handed and two-handed driving. Results obtained for the (c) deltoid showed marked changes in muscle activity from the first stroke to the second stroke. Moreover, results found for the (e) latissimus dorsi showed a gradual decrease in muscle activity from the second stroke. These results also indicated a similar trend for both one-handed and two-handed driving, suggesting that these muscles are the main driving muscles playing a fundamentally important role in speed maintenance or acceleration movements For EMG data recorded during the first experiment, the EMG data for one stroke of the hand-rim operation were extracted. After full-wave rectification, the iEMG data were calculated. These iEMG data quantitatively represent the total workload of the electrical activity of the muscles, meaning that they represent the amount of muscle activity for one drive during wheelchair operation. The graph presented in Figure 5 depicts the transition per stroke for one-arm and two-arm driving up to eight strokes. It is apparent from this figure that the (a) flexor digitorum profundus, (b) triceps brachii, and (d) pectoralis major, showed a large increase in muscle activity at the first stroke and a monotonic decrease with each increase in the number of strokes. For one-arm driving as well, these muscles indicate that the greatest amount of muscle force is necessary at the start of driving. Subsequently, the load on these muscles during the push phase decreased concomitantly with increasing wheelchair speed. Consequently, muscle activation also decreased concomitantly with the increasing stroke rate. In fact, the (c) deltoid and (e) latissimus dorsi results indicate that the amount of muscle activity decreased at the first stroke and end strokes for both one-handed and two-handed driving. Results obtained for the (c) deltoid showed marked changes in muscle activity from the first stroke to the second stroke. Moreover, results found for the (e) latissimus dorsi showed a gradual decrease in muscle activity from the second stroke. These results also indicated a similar trend for both one-handed and twohanded driving, suggesting that these muscles are the main driving muscles playing a fundamentally important role in speed maintenance or acceleration movements even in one-handed driving.

even in one-handed driving. Next, we use statistical analyses for specific examination of muscle sites that are significantly different. In the (b) triceps brachii during one-arm operation, the muscle activation increased by about 14% compared to that during operation with two arms. In the (c) deltoid and (d) pectoralis major during one-arm operation, muscle activation increased by about 10% and 20% compared to that during operation by two arms. These muscle sites were found to play an important role during one-arm operation. The reason for this large role is that, when one correlates the results of the decrease in muscle activity from two to seven strokes with the increase in muscle activation of about 10% over one arm, it can be shown to have the strongest effect on speed maintenance and acceleration movements among the driving muscles that were measured. However, the amount of these muscle activations during one-handed driving was less than expected because the competitive wheelchair used for the experiment requires only one arm to transmit the propulsive force to the opposite drive wheels. A simple calculation indicates that one-arm operation requires twice as much muscle activation. In conclusion, the experimentally obtained results Next, we use statistical analyses for specific examination of muscle sites that are significantly different. In the (b) triceps brachii during one-arm operation, the muscle activation increased by about 14% compared to that during operation with two arms. In the (c) deltoid and (d) pectoralis major during one-arm operation, muscle activation increased by about 10% and 20% compared to that during operation by two arms. These muscle sites were found to play an important role during one-arm operation. The reason for this large role is that, when one correlates the results of the decrease in muscle activity from two to seven strokes with the increase in muscle activation of about 10% over one arm, it can be shown to have the strongest effect on speed maintenance and acceleration movements among the driving muscles that were measured. However, the amount of these muscle activations during one-handed driving was less than expected because the competitive wheelchair used for the experiment requires only one arm to transmit the propulsive force to the opposite drive wheels. A simple calculation indicates that one-arm operation requires twice as much muscle activation. In conclusion, the experimentally obtained results suggest that only one arm and other factors can maintain muscle activation producing the same propulsive force as that produced by two arms.

#### *4.2. Body Posture Estimation Using iEMG for Evaluation during One-Arm Propulsion* 4.2.1. Differences in Muscle Activation Attributable to Different Driving Patterns

The graph shown in Figure 7 presents iEMG findings up to eight strokes with one-arm and two-arm driving, divided into the left-side measurement site (a) and the right-side measurement site (b), centered on the body axis. The iEMG data of the erector spinae in the upper part of Figure 7 show similar trends for (a) the left side and (b) the right side. However, the trends of the external oblique abdominis shown in the middle part and the triceps brachii presented in the lower part are very different between the left and right sides. For the external oblique abdominis, a constant difference was found in the transition of each stroke between one arm and two arms on the left side. For the triceps brachii, a constant difference was found on the right side. The results confirmed that one-arm and two-arm operations produced differences in the external oblique abdominis and triceps brachii.

Next, we specifically examine categories for which significant differences were found based on one-arm operation according to the results of the statistical analyses for muscle activation. Muscle activation in the erector spinae was approximately 49% less on the left side (LO) than on the right side (RO) of the hand-rim operation. The left side (LO) was about 1.5 times more than the right side of two arms (RT). On the opposite side (RO) of the hand-rim operation, the number was approximately 1.7 times more than on the left side with two arms (LT). The erector spinae, long, large muscles located in the back, were found to be activated on the opposite side of the hand-rim operation. The erector spinae muscles are mainly involved in trunk extension movements. The role of the erector spinae is to extend (retroflex) the trunk during athletic movements. This role contributes to posture maintenance, stabilizing the upper body in all sports activities. The erector spinae are also involved in upper-body raising activities. The correct operating posture for straight running in two-arm operation is that the body axis does not tilt in the left–right direction. In addition, the upper limbs bend forward and backward repeatedly. Muscle activation of the erector spinae, which constitute the trunk of the user, is probably not biased to either side to any considerable degree. Therefore, one-arm operation during straight running might cause the user's movement body posture to be biased not only to forward bending and backward bending but also to the left or right, considering the effects of the decrease in muscle activation.

In the external oblique abdominis, left side (LO) muscle activation was about 2.7 times greater by than at other times. The external oblique abdominis were found to play an important role in the wheelchair propulsion force by one-arm operation. The external oblique abdominal muscles are the most superficial muscles on the lateral side of the abdomen. They are used for trunk rotation, and are also involved in other functions such as bending the trunk sideways. The role of the muscle is mainly to rotate, flex (forward bending), and bend the trunk laterally during locomotion. The muscles therefore contribute greatly to all motion behavior that twists the body. As described above, the bimanual drive-in straight-line running repeats only forward and backward bending of the upper limbs. Results indicate that the muscle activity of the left external oblique abdominis, which operates the hand-rim, increased nearly threefold during one-arm operation. The user's operating body posture includes lateral flexion and rotation of the upper limb in addition to flexion. This prediction can be clarified from results of the three-axis acceleration sensor attached to the upper limb and the change in the seat pressure distribution described below.

In the triceps brachii, right side (RO) muscle activation decreased 80% compared to that during the left side (LO). This is a reasonable result because the right arm, which does not operate the hand-rims, is placed on the user's lap; it is not moved. The triceps brachii extends the elbow joint. It is the main muscle used for the push phase of wheelchair pro-pulsion. The long head of the triceps, a so-called biarticular muscle, is also involved in shoulder joint internal rotation and extension. The muscle role is to throw, push, and lift objects during athletic activities. It contributes greatly to sports activities that involve pushing forward. As shown by the first experiment, one-arm operation exhibited the highest muscle activity near the beginning of the drive, which increased by more than 10% compared to two-handed driving. However, comparisons between one arm (LO) and two arms (LT) revealed no significant differences in muscle activation.

#### 4.2.2. Evaluation of Posture Estimation

This section presents discussion of the validity of posture estimation using iEMG based on the user's body axis and the amount of change in the seat pressure distribution. As shown in Figure 9, no characteristic changes were observed between cycle segments for the three axes. These results suggest that both one-arm and two-arm operations were cyclically repeated stably. In addition, the stable body posture during wheelchair propulsion suggests that we have also reduced the effect of postural changes on muscle activation. The amount of change in the transverse direction of the body axis (X-axis) increased approximately 1.5 times when averaged over all cycles during one-arm operation. However, the vertical direction of the body (Y-axis) increased by about 1.6 times during two-arm operation. The front–back direction (Z-axis) also increased by about 1.1 times in two arms. Additionally, it can be confirmed that the amount of change in the forward–backward direction is greatest for both one-arm and two-arm operations. The one-arm operation causes a forward tilt of the driving side (left side). The two-arm operation would tilt toward the direction of travel. The increase in the X-axis during one-arm operation suggests the influence of flexion and extension of the external oblique abdominis due to the trunk circumnutating motion. The explanation in Section 4.2.1 corroborates the results.

Figure 10b shows that the seat pressure distribution during one-arm operation showed an increase in pressure values on the opposite side (right side) of one-arm operation (left side) compared to two-arm operation. The maximum value of the difference was 196.4 (mmHg) at the rear of the right side. In the case of one-arm operation, the center of gravity of the seat pressure shifted to the opposite side of the hand-rim operation side to that found in the case of two arms. This result derives from movement of the center of gravity to the opposite side (right side) of the hand-rim when operating the hand rim with one arm, which reflects the effort to keep the body axis of the body centered in the operating body posture. To support this result, we have already explained, in Section 4.2.1, the muscle activation of the external oblique abdominis by one-arm operation, which are involved in torso rotation, flexion, and lateral bending, and increased muscle activation. Results also confirmed that the body position of the center of gravity on the seat surface moved to the right and rearward during movement to return the body axis to the center because of the influence of the increase in the amount of change in the body axis in the left oblique forward direction. In conclusion, we found that the body posture of one-arm operation is different from that of two-arm due to the effect of the activation site. These results suggest that an approximate motion posture can be estimated from observation of muscle activation by surface EMG.

#### **5. Conclusions**

This study was conducted to develop competitive wheelchairs that will allow participating in sports by facilitating straight-ahead movement with one arm. This wheelchair has the potential to be a new assistive technology for people participating in sports. However, the differences in muscle activation and body posture between one-arm and two-arm operations have not been clarified. Once these issues are resolved, it will be possible to determine the muscles needed to strengthen one-arm operation and the approximate range of adaptability to the site and degree of disability.

The first experiment, using iEMG results, was conducted to elucidate differences in muscle activation of a user's arm and shoulder joint muscles during propulsion using only one arm using two arms with a competitive wheelchair equipped with our dual hand-rim system. One-arm operation showed significant differences in activation of the triceps brachii and deltoid and pectoralis major, with a 10–20% increase in muscle activation compared to that when using two arms. Nevertheless, the amounts of muscle activation during one-handed driving were less than expected. The experimentally obtained results suggest that only one arm and other factors can provide the same propulsive force as that supplied by two-arm driving.

The second experiment was undertaken to use iEMG data to estimate the movement posture during wheelchair propulsion with one-arm operation. For the erector spinae, muscle activation was approximately half on the hand-rim manipulation side (left) compared to the right side. For the external oblique abdominis, the left-side muscle activation was about 2.7 times greater than at other times. For the triceps brachii, comparisons between the left side of one arm and the left side of two arms during hand-rim manipulation revealed no significant difference in muscle activation. The external oblique abdominis muscles played an important role in producing wheelchair propulsion during one-arm operation. Finally, the validity of using iEMG for posture estimation was verified by the amount of change found in the user's body axis and seat pressure distribution. In conclusion, results indicate that one-arm wheelchair operation activates different muscle sites and produces different body posture than wheelchair manipulation by two arms, thereby allowing movement and posture estimation using surface EMG.

**Author Contributions:** Conceptualization, methodology, S.O. and A.S.; experimental software and validation, S.O., A.S., K.H. and M.N.; investigation, S.O.; formal analysis, S.O., A.S. and K.H.; data curation, S.O. and A.S.; writing, original draft and preparation, S.O.; review and editing S.O., A.S., K.H., M.N. and H.U.; supervision, S.O.; funding acquisition, S.O. and A.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by JSPS KAKENHI, Grant-in-Aid for Scientific Research (C), Grant Number 18K11890. And Nagaoka University of Technology (NUT) grant for collaborative research with National Institute of Technology (NIT) 2019, Grant Number 02\_01.

**Institutional Review Board Statement:** The study was conducted in conformance to the guidelines of the Declaration of Helsinki and approved by the Ethics Committee of Nagaoka University of Technology (protocol code H30-1, H30-2 and date of approval is 27 June 2018).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study. Written informed consent has been obtained from the patients to publish this paper.

**Acknowledgments:** We gratefully acknowledge the work of past and present members of our laboratory.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Estimating the Emotional Information in Japanese Songs Using Search Engines**

**Jin Akaishi 1,\*, Masaki Sakata <sup>2</sup> , Jouichiro Yoshinaga <sup>3</sup> , Mitsutaka Nakano <sup>1</sup> , Kazuhiro Koshi <sup>1</sup> and Kimiyasu Kiyota <sup>1</sup>**


**Abstract:** Several studies have shown that music can reduce unpleasant emotions. Based on the results of this research, several systems have been proposed to suggest songs that match the emotions of the audience. As a part of the system, we aim to develop a method that can infer the emotional value of a song from its Japanese lyrics with higher accuracy, by applying the technology of inferring the emotions expressed in sentences. In addition to matching with a basic emotion dictionary, we use a Web search engine to evaluate the sentiment of words that are not included in the dictionary. As a further improvement, as a pre-processing of the input to the system, the system corrects the omissions of the following verbs or particles and inverted sentences, which are frequently used in Japanese lyrics, into normal sentences. We quantitatively evaluate the degree to which these processes improve the emotion estimation system. The results show that the preprocessing could improve the accuracy by about 4%. Japanese lyrics contain many informal sentences such as inversions. We pre-processed these sentences into formal sentences and investigated the effect of the pre-processing on the emotional inference of the lyrics. The results show that the preprocessing may improve the accuracy of emotion estimation.

**Keywords:** emotion tagging; mood management; Japanese song lyrics; search engine

#### **1. Introduction**

Recent studies have shown that music and songs have positive effects on a person's psychological state and physical health, such as improving concentration and reducing fatigue and stress [1–3]. It is becoming clear that negative emotions can be alleviated by listening to music and songs that remind people of a particular emotion. Matsumoto found that when a subject has very sad emotions and listens to sad music, the feeling of sadness decreases [4]. Similarly, a study by Sharman and Dingle revealed that listening to intense music when a subject is feeling angry can alleviate the anger [5]. Based on these research results, a system that proposes songs according to the mood of the listener has been proposed [6]. However, it has also been reported that offensive songs can have a negative impact on the audience [7,8]. In these systems, it is important to be able to avoid songs that may have a negative impact and suggest songs that have a positive impact. The provision of songs to people with serious mental conditions needs to be carefully proposed through the use of a white list to avoid negative effects.

One of the important functions in the song proposal system is to automatically estimate the emotions evoked by the song with higher accuracy. A system that automatically classifies the emotions in a song is indispensable for processing a certain request from a

**Citation:** Akaishi, J.; Sakata, M.; Yoshinaga, J.; Nakano, M.; Koshi, K.; Kiyota, K. Estimating the Emotional Information in Japanese Songs Using Search Engines. *Sensors* **2022**, *22*, 1800. https://doi.org/10.3390/s22051800

Academic Editor: Raffaele Gravina

Received: 24 December 2021 Accepted: 21 February 2022 Published: 24 February 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

huge number of songs. There have been many studies on methods for inferring emotions in songs [9]. Audio and lyrics can be considered as elements that evoke emotions contained in songs [10,11]. Several systems have been developed to estimate the emotion of a song using the lyrics as the target of analysis [12]. In addition to English, systems for estimating emotions have been developed for Chinese and Japanese lyrics [13,14]. To improve the accuracy of sentiment estimation, it is important to make adjustments that correspond to the unique features of each language.

Kaur and Saini pointed out that the accuracy of sentiment estimation decreases in sentences with an informal writing style [15]. It is important to propose a method to deal with this informal writing style to improve the accuracy of emotion estimation. In the case of Japanese lyrics, substantive stop and inversion are the most commonly used informal writing styles. Few studies have quantitatively evaluated the effects of informal writing styles on the accuracy of emotion estimation for Japanese lyrics.

Several methods have been put forward for extracting emotional information. Estimation methods using machine learning and contextual methods have been proposed [16,17]. In contrast, a classic and simple method is the analysis using an emotion dictionary. This method is easier to analyze than more complex methods and has the advantage that it is easier to understand the impact of adding new elements. One conventional method focuses on surface expression and processes emotions using a valence pattern using an emotion dictionary, ML-Ask, which was developed by Ptaszynski et al. [18]. An emotion evaluation system with more complex functions, such as consideration of negative sentences in Japanese, has also been used [18]. Furthermore, a method has been proposed by Shi [19] to search for related words in text using Web mining [19]. This method improves the accuracy of estimating the emotional value contained in the text. By extracting emotional words that co-occur from a large amount of text on the web for an input phrase and performing emotional estimation, even if there is no direct emotional expression in the song phrase, it is possible to infer the emotion.

There have been many studies aimed at extracting emotional information from sources such as news items, conversations, and song lyrics (e.g., Ptaszynski et al. [18]). There have also been many attempts at emotional estimation for lyrics. However, research on estimating the emotions in Japanese lyrics using a Web search engine is limited [20]. The previous study has shown that the accuracy of the estimation tends to be improved by using Web search for the estimation of emotions in Japanese lyrics. However, because punctuation could be omitted, it might also create subtle differences in meaning. In addition, it has been pointed out that the accuracy of sentiment estimation may be improved by revising sentences with substantive stop or ending a sentence with a noun or inversion to normal sentences. Japanese lyrics use many special grammatical features, such as substantive stops and inversions. By focusing on lyrics that contain many of these grammatical structures, we expect to obtain results that focus on the analysis of emotions in Japanese songs, which is different from research on ordinary literature. However, quantitative evaluation of the improvement in accuracy has not yet been conducted.

Quantifying the effect of modifying sentences with informal writing styles, which is peculiarly common in Japanese lyrics, to normal sentences on sentiment estimation using Web search will be useful from the technical point of view of improving the accuracy of sentiment estimation for Japanese sentences. It will also benefit from the perspective of improving the performance of systems that contribute to mental stability through music.

The purpose of this study is as follows. The object of analysis in this study is the lyrics, not the music. As part of the song proposal method, we use a Web search engine to build a system that can classify the emotions that a song evokes based on Japanese lyric information. We then compare the accuracy of this system with that of conventional methods. Next, we introduce a preprocessing method to correct substantive stops and inverted sentences, which are often found in Japanese lyrics, into a formal writing style and evaluate its effectiveness. In the future, it is envisioned that the system will be part of a system that helps people who are casually listening to music on portable music devices to ease their feelings when they feel somewhat naive or angry.

#### **2. Materials and Methods**

First, we selected recently popular Japanese songs and extracted the lyrics for testing. Next, the lyrics decomposed into morphemes were input into a system that infers the emotions that the lyrics recall to humans, and the results were output.

Seven emotion categories are dealt with in this study: joy, relief, sadness, anger, like, dislike, and excitement. There are 10 types of classifications widely used in Japanese sentiment analysis in the "Emotional Display Dictionary" [21]. However, when Japanese lyrics are presented to subjects, the emotions that the subjects feel from the lyrics are frequently five types: "joy, like, relief, sadness, and excitement." It has become clear that the five types of emotion, "dislike, fear, anger, shame, and surprise", rarely appear [22]. Based on the above results, to reduce the processing burden, this study deals with seven categories by adding two emotions that appeared relatively frequently. Among the five classifications with a low frequency of occurrence, two of them, unlike and anger, have a relatively high frequency of occurrence. We decided to handle seven sentiment categories, considering the balance between accuracy and processing load reduction.

The emotion dictionary used in this study was an extension of the standard emotion word dictionary in Japanese, "Emotional Display Dictionary". In addition to the 1097 words in the emotional expression dictionary, synonyms for emotional words have been added from the "Must-have Synonyms Practical Dictionary" published by Sanseido [23]. The total number of emotional words after the expansion is 1930.

We used the four systems described below as emotion estimation systems.


Table 1 shows the processes used in these methods. A checkmark indicates that the process was used. Of these four methods, we conducted Methods A through C as Experiment 1, and Method D as Experiment 2.

**Table 1.** Processing techniques used in each method.


#### *2.1. Experiment 1*

First, we selected 50 recently popular Japanese songs and extracted the lyrics for testing. Then, the lyrics of the 50 songs were shown to 10 subjects (eight males and two females, in their teens to 40 s, students, mean age: 22.4 years) on a website. Subjects were asked to choose from seven different emotions (joy, relief, sadness, anger, like, dislike, and excitement) that were evoked when they read the lyrics, and to vote by clicking on the corresponding emotion button on the homepage (multiple responses were allowed). The top three emotions were used as the majority of data for the human evaluation of the emotions.

#### 2.1.1. System for Classifying Lyrics Emotions

We describe in detail a method (Method C) that uses a Web search engine to infer the emotions evoked by the lyrics we propose this time. An outline of the system is shown in Figure 1. In the proposed method, emotions are first estimated using ML-Ask. Here, we use a method for inferring emotions from sentence-ending expressions and emotional expressions contained in the sentence itself using a superficial expression of the sentence. If the emotion expression cannot be extracted, an emotion estimation using a search engine, "emotion estimation by causal relation on the Web", is performed. Here, we used a method of acquiring sentences that have a causal relationship with sentences from the Web and recognizing emotions. We processed the lyrics in the following steps:

	- 2.1 Outputs the emotion value for the matching classification when the morpheme matches the emotional word in the dictionary of the system.
	- 2.2 If there is no emotional word in the sentence, analyze the sentence that holds the causal relationship that exists on the Web.
		- 2.2.1 Divide the sentences from the Web into morphemes with MeCab.
		- 2.2.2 Comparison with an emotion dictionary, and if they match, output the emotion values of each emotional classification.

The flow of this process is shown in the flowchart in Figure 1.

The following is an example of the process. For example, if a lyric contains an emotional word such as "happy", the word "joy" is judged. If the lyrics do not contain emotion words, such as "I played baseball", the system uses a search engine to retrieve sentences on the Web that are related to the lyrics. For example, if the sentence retrieved is "I had fun playing baseball", the co-occurrence of the emotion word "fun" is judged as "joy".

#### 2.1.2. Emotion Estimation Using Search Engine

This section describes the detailed procedure for emotion estimation using a search engine. When a lyric phrase does not contain a word with an emotion in the dictionary, the phrase in the lyric is extracted and a conjunction is added to the end of the sentence to create a search query. The search query is input and the resulting short sentences are saved. The search query is created based on the rules used by Shi [19], and some modifications are made according to the emotion estimation of the lyrics. The following is a description of these modifications:

	- ➢ If the following conditions are not met, lengthen it to 4-grams, 5-grams, etc.
	- ➢ Do not make phrases from particles or conjunctions

➢ Making the phrase grammatically correct when a conjunction is added

Next, to make it easier for emotional words to co-occur, the Japanese conjunctions "*-te, -to, -node*" are added to the end of the created phrase. A search query is a combination of these three conjunctions with a phrase. Shi revealed that by including some adjunct words in the search query, more search results including emotional words can be obtained [19]. Based on the results, we adopted the above three conjunctions, which had a high probability of acquiring emotional words. The reason for limiting the number of conjunctions to three is to limit the amount of searching. *Sensors* **2022**, *21*, x FOR PEER REVIEW 5 of 11

**Figure 1.** The flowchart of the system for classifying lyrics emotions. **Figure 1.** The flowchart of the system for classifying lyrics emotions.

The following is an example of the process. For example, if a lyric contains an emotional word such as "happy", the word "joy" is judged. If the lyrics do not contain emotion words, such as "I played baseball", the system uses a search engine to retrieve sentences on the Web that are related to the lyrics. For example, if the sentence retrieved is "I had fun playing baseball", the co-occurrence of the emotion word "fun" is judged as "joy". We obtained snippets from a search engine using a query with the conjunctions described above. In this study, we used Microsoft Bing as the search engine. Forty snippets obtained for each query were saved as text data. In other words, one phrase saves 120 snippets. From the acquired snippets, the sentences following the input phrase were saved for use in emotional estimation. However, if the sentence following the phrase contains a conjunction such as "but", the sentence was not saved. Table 2 shows an example of the above process.

#### 2.1.2. Emotion Estimation Using Search Engine 2.1.3. Analysis of Text Data

• Ignoring stop words

This section describes the detailed procedure for emotion estimation using a search engine. When a lyric phrase does not contain a word with an emotion in the dictionary, the phrase in the lyric is extracted and a conjunction is added to the end of the sentence to create a search query. The search query is input and the resulting short sentences are saved. The search query is created based on the rules used by Shi [19], and some modifications are made according to the emotion estimation of the lyrics. The following is a description of these modifications: • Creating phrases from 3-grams Morphological analysis was performed on the saved text data, and matching processing was carried out using a dictionary that is an extended "Emotional Expression Dictionary". Similar to Shi's research, the number of words that matched the headwords of each emotion category was used as the score for each emotion category. Each emotion category is sorted in descending order of score, and output by the system as the emotion that the song has. Multiple sentiment classifications can be output from the system. To simplify the processing, we decided not to output the emotion categories with low scores according to the following rules. Emotional categories that did not satisfy the following two

If the following conditions are not met, lengthen it to 4-grams, 5-grams, etc.

Making the phrase grammatically correct when a conjunction is added

Next, to make it easier for emotional words to co-occur, the Japanese conjunctions " *te, -to, -node*" are added to the end of the created phrase. A search query is a combination

• The end of the phrase is a verb or adjective

• The beginning of the phrase is not a particle or conjunction

• Changing the end of a sentence to a non-past terminal form

conditions were ignored and output was set to zero. (1) The score of the emotion category must be in the top 60% of all scores. (2) The score of the emotion category must account for 10% or more of the total score. For example, in the case of "Sad 51%, Joy 9%, and the other five emotions are 8% each", Joy is in the top 60%, but only sad emotions are output.

**Table 2.** Example of inputting Japanese lyrics to saving the sentence to be analyzed when estimating emotions by Web search.


#### 2.1.4. Aggregation of Emotional Value

The evaluations for each emotion category obtained by the emotion evaluation by ML-Ask and the emotion evaluation by the causal relationship on the Web are aggregated. The evaluation using ML-Ask, which is processed in the first stage, performs advanced processing, such as directly analyzing the lyrics and considering negative sentences. Therefore, it is considered to be more accurate than emotion estimation using a search engine. Therefore, in order to improve the accuracy, the output of the emotion category when emotional expression by ML-Ask is included is weighted. Five types of weights (1–5 times), were tried, and three times, which had a high F-score overall, were adopted. In other words, in this system, the evaluation by ML-Ask was three times heavier than the evaluation using the Web.

#### *2.2. Experiment 2*

The key point of the emotion prediction using Method D in Experiment 2 is the pre-processing before inputting the lyrics into the system used in Method C. In the preprocessing, the lyrics were divided into individual sentences to determine whether informal writing styles were used or not. If informal writing styles were used, the sentence was modified to a formal writing style by the process described below. We then input phrases into the system as in Method C of Experiment 1 and evaluated the accuracy of the emotion guessing. We describe the details of the preprocessing in Experiment 2 below.


In Experiment 2, 30 songs different from those in Experiment 1 were selected, and 15 people (11 males, 4 females; all students are in their twenties; mean age 20.0 years old) voted for the seven emotions (multiple responses were possible), and the top three were evaluated as the majority answer. Since the data were different from that of Experiment 1, emotion estimation was also conducted using Method A, which evaluates only the emotion dictionary, as a standard for comparison.

#### *2.3. Evaluation by F-Score*

We use the F-score as a measure of the prediction accuracy of the system. The process of obtaining the F-score is described below.

The seven emotion categories used in this study are denoted as set A:

$$\mathbf{A} \ := \{ \text{joy}, \text{relief}, \text{sadness}, \text{angel}, \text{like}, \text{dislike}, \text{excitemt} \}. \tag{1}$$

Using the lyrics of a song as input, the system outputs the probability of each emotion category. Next, the system eliminates the emotion categories with low probability according to the conditions described in Section 2.1.3. Finally, the system selects up to three emotional categories from among the ones with the highest probability, and outputs them as the final output. Let Es ⊂ A be the set of emotion categories predicted by the system, where Es is a subset of A. Similarly, let Eh ⊂ A be the set of emotion categories evaluated by a human after reading the lyrics of a song, where Eh is a subset of A.

The sets TP, FP, FN, and TN required for the calculation of the F-score can be obtained by the logical operation shown in the following equations:

$$\text{TP} = \text{Eh} \land \text{Es}, \text{ FP} = \neg \text{Eh} \land \text{Es}, \text{ FN} = \text{Eh} \land \neg \text{Es}, \text{ TN} = \neg \text{Eh} \land \neg \text{Es}.\tag{2}$$

Using the sets TP, FP, FN, and TN, Precision and Recall are obtained:

$$\text{Precision} = \frac{\text{n(TP)}}{\text{n(TP)} + \text{n(FP)}},\tag{3}$$

$$\text{Recall} = \frac{\text{n(TP)}}{\text{n(TP)} + \text{n(FN)}}.\tag{4}$$

Here, n(X) is a function to find the number of elements in the set X. Using this Precision and Recall, we calculated the F-score:

$$\text{Fscore} = \frac{\text{2Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}.\tag{5}$$

#### **3. Results**

The accuracy was compared by calculating the F-score (F1) for the emotion estimation results for the systems. The results of Experiment 1 are shown in Table 3.

**Table 3.** Results when the majority answer data are in the top three estimated emotions by Web search.


The results of Experiment 2 are shown in Table 4. In the results shown in Table 4, the F-score of Method A is 58% and that of Method D is 67%. Note that the results for Method A, here, evaluate a different dataset than in Experiment 1.


**Table 4.** Results when the majority answer data are in the top three estimated emotions by Web search.

#### **4. Discussion**

Comparing the F-values for Method C and Method B, Method C is 13% higher than Method B. Therefore, emotional estimation using causal relationships on the Web is effective. Comparing Method C and Method A, the recall rate improved by 8%, and the F-score improved by 7%. Therefore, the accuracy improved slightly.

The reason why Method A has a higher F score than Method B, even though the procedure is simpler, is considered to be because the emotion dictionary is larger than that for Method B. The dictionary used in Method A has 1930 emotion words, and Method B has 907 emotion words. The emotion dictionary used in Method C that evaluates sentences from the Web is the same as the dictionary in Method A.

There is a study by Ptaszynski using a Web search for emotion estimation, which was the basis of this study [18]. In their study, the F-score was 54% to 53% when the emotions of conversational sentences were estimated by a similar method. It is difficult to make a simple comparison with the results of the present study because the analysis target, the number of emotion categories, and the types of search engines are different. However, the emotion estimation of the lyrics in this study is about 7% better than the case for conversational sentences. This indicates that the use of a Web search engine may be more suitable than the analysis of conversational sentences in estimating emotions in lyrics.

The introduction of Method D improved the F-score by 7 compared to Method C. However, the datasets of Experiment 1 and Experiment 2 are different. Based on the difference between the results of Method A in each experiment, it can be inferred that the dataset in Experiment 2 tends to be easier to guess the emotion by 4 points. Even after subtracting these four points, the change to normal sentences tended to be effective.

There are several reasons why this change to normal text was effective, including the following. Table 5 shows an example of lyrics expressed using three types of methods, i.e., a normal sentence, a word stop, and an inversion method, and phrases generated from the sentence. Since the three original sentences in Table 5 do not contain emotion words, emotion inference based on causal relationships on the Web is applied. Therefore, phrase creation for the search query is performed. The phrases generated based on the phrase creation rules are shown on the right-hand side of Table 5.


**Table 5.** Effect of inversion and ending a sentence with a noun in generated phrases.

As can be seen in the generated phrases in Table 5, the word "meteor shower" is omitted from the phrases generated based on the sentences using the word-stopping and inversion methods. This is because the phrase creation rule "the end of the phrase is a verb or an adjective" is applied. By eliminating the word "meteor shower" in the search query, the resulting snippet becomes more abstract and the estimated emotion value can be different from that in the original lyrics. Therefore, the accuracy of the sentiment classification was improved by judging the sentences with substantive stop and inversion in advance and processing them back to plain text.

It was shown that Method C and D in this study improve the accuracy of emotion evaluation. However, this improvement in accuracy is limited when the cost of the network and the computational cost of using a search engine are taken into account.

This study was limited in several aspects due to the limitations of the research resources. The following is a discussion of these limitations.

The first limitation is the number of samples obtained as the target data. The sample size of this study is 10, and the age range is limited, so there is a limit to the generalization of the target data with the system. In the future, a more accurate comparison with the target data will be required by acquiring more data.

Next, there is a linguistic limitation. In this study, we used Japanese lyrics, so our findings are limited to the scope of Japanese.

The third limitation is the technical limitation. In the improved method of sentiment estimation using Web search engines used in this study, the search results are not stable because the search results of the engines are updated daily. In addition, it is necessary to balance the cost of using the network and creating search queries with the benefit of improving the accuracy obtained.

Furthermore, in the estimation of emotions evoked by songs, combining the analysis of lyrics and music is expected to achieve higher accuracy. Music is also an important factor in evoking emotions. The emotions evoked by lyrics can be diversified depending on various factors such as the background of the listener. In addition, music may evoke emotions opposite to those of lyrics [8]. Therefore, the accuracy of emotion estimation may be limited by analyzing only the lyrics. Due to the limitations of our expertise and research resources, we limited the scope of our study in this paper to Japanese lyrics. In the future, it is expected that the findings obtained in this study will contribute to the improvement of more accurate emotion analysis techniques by combining them with music analysis.

#### **5. Conclusions**

In this study, we constructed a system that can classify the emotions that a song evokes from Japanese lyric information using a Web search engine as part of a song proposal system that suits the mood of the listener. We also performed a quantitative evaluation using a method that combines emotion estimation using a superficial expression of sentences and emotion estimation using a search engine. The method using a search engine improved the accuracy by 4% compared to the method using a conventional dictionary.

In addition, we attempted to improve the accuracy of sentiment inference by introducing preprocessing techniques such as substantive stops and inversions to normal sentences. The results show that the accuracy could be improved by about 4%. However, it is difficult to make an exact comparison because the datasets used in the experiments are different.

As a future development, we will implement a system that can suggest songs suitable for the user's emotions using our results. If future research focuses on improving song recommendation logic, it should reflect findings related to music psychology and music therapy, taking into account that people with serious mental health conditions may use the system. There is a method to measure the effect of a certain function called ablation experiment, which is often used in deep learning research. In this method, the effect of a function is measured by pausing the function and checking the effect of the paused function. To clarify the effect of Web search, it is possible to conduct ablation experiments.

**Author Contributions:** Conceptualization, J.A., J.Y. and M.S.; methodology, J.A., J.Y. and M.S.; software, J.Y. and M.S.; validation, J.A., M.N., K.K. (Kazuhiro Koshi) and K.K. (Kimiyasu Kiyota ); formal analysis, J.A., J.Y. and M.S.; investigation, J.Y. and M.S.; resources, J.A.; data curation, J.Y. and M.S.; writing—original draft preparation, J.A., J.Y. and M.S.; writing—review and editing, M.N., K.K. (Kazuhiro Koshi) and K.K. (Kimiyasu Kiyota); visualization, J.A., J.Y. and M.S.; supervision, J.A.; project administration, J.A.; funding acquisition, J.A. and K.K. (Kimiyasu Kiyota). All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Acknowledgments:** We thank Akiyo Nadamoto of Konan University for permission to use her dictionary data in this study.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Human Standing Posture Motion Evaluation by the Visual Simulation of Multi-Directional Sea-Waves**

**Renon Doine <sup>1</sup> and Takanori Sakamaki 2,\***


**Abstract:** Crew fatigue from standing posture motion, caused by ship motion, can lead to marine accidents. Therefore, the mechanism of fatigue in crew members ought to be elucidated. The standing posture of humans is maintained by postural state detection through the visual, vestibular, and somatosensory systems. Humans can adjust their posture through corrective postural reactions (CPR) generated after anticipatory postural adjustments (APAs) by using information from these sensory systems. APAs refer to skills acquired by learning from past motions and perturbations and are prepared by the central nervous system based on visual information before the actual perturbation occurs. We hypothesized that APAs would decrease fatigue in crew members by stabilizing their standing posture motions. We aimed to clarify the human standing posture control influenced by APAs based on visual information. To this end, we presented wave images with different wave directions to the participants using a visual simulator and analyzed their standing posture motion. We found that the participants stabilized their standing posture based on the projected wave directions. This showed that the participants predicted ship motion from the wave images and controlled their center of pressure (COP) through APAs. Individual differences in standing postural motion may indicate the subjective variation of APAs based on individual experiences. This study was limited to males aged 20–23 years. To generalize this study, randomized controlled trials should be performed with participants of multiple age groups, including men and women.

**Keywords:** fatigue; visual simulator; human standing posture; center of pressure; anticipatory postural adjustments

#### **1. Introduction**

In 2019, the total annual trade volume of imports and exports of Japan exceeded 900 million tons, of which 99.6% constituted maritime trade. In 2020, 1954 marine accidents occurred, demonstrating the criticality of preventing them. Human error accounted for 74% of these accidents [1]. The fatigue of crew members is the primary cause of accidents. Therefore, the mechanism of crew member fatigue ought to be elucidated.

Crew members constantly control their posture to maintain their standing posture against ship motion, which causes physical fatigue [2]. On measuring the physical fatigue of crew members in terms of energy expenditure, pitch-and-roll motion, generated in a ship motion simulator, was identified as increasing fatigue [3]. The pitch-and-heeling motion of the ship was also found to affect the crew's energy expenditure [4]. Our previous studies focused on fatigue based on the postural control of crew members and analyzed their energy expenditure and standing posture motion [5–9]. The international standard ISO6954 evaluates the habitability of ships based on ship motion; however, it ignores physiological indicators [10].

The standing posture of humans is stabilized by corrective postural reactions (CPR), generated after anticipatory postural adjustments (APAs), using information from the

**Citation:** Doine, R.; Sakamaki, T. Human Standing Posture Motion Evaluation by the Visual Simulation of Multi-Directional Sea-Waves. *Sensors* **2022**, *22*, 5884. https:// doi.org/10.3390/s22155884

Academic Editor: Guillermo Villanueva

Received: 15 May 2022 Accepted: 4 August 2022 Published: 6 August 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

visual, vestibular, and somatosensory systems [11]. Specifically, the visual system detects postures based on visual information; the vestibular system senses the orientation of the head with respect to gravity, depending on the balance between the vestibule and the semicircular canal of the inner ear; the somatosensory system detects postural movements based on muscle and joint movements [11,12]. APAs refer to skills acquired by learning from past motions and perturbations and are prepared by the central nervous system based on visual information before the actual perturbation occurs [11]. CPR generates an adjustment after an actual perturbation [11]. Standing posture motion caused by APAs and CPR is observed as a change in the center of pressure (COP) [11].

The control system for the human standing posture depends strongly on visual information [11]. Thus, analyzing the effects of standing posture motion triggered by visual information on the crew members is crucial. In previous studies, we analyzed the energy expenditure of the participants and their standing posture motion triggered by the wave images presented using a ship-handling simulator. No significant differences were detected between the energy expenditure of participants exposed to images with and without waves [13–15]. The standing posture motion may have been caused by the APAs of the participants based on the visual information.

We hypothesized that APAs would decrease fatigue in crew members by stabilizing their standing posture. The current study aimed to clarify the human standing posture control resulting from APAs based on visual information. Thus, we presented wave images with different wave directions to the participants using a visual simulator and analyzed their standing posture motions.

#### **2. Methods**

#### *2.1. Participants*

For the present study, the inclusion criteria were an age between 20 and 23 years and male sex. Table 1 presents the details of the seven participants involved in the experiment. The procedures were explained to the participants before initiating the experiment, and they provided written informed consent. To prevent the effects of eating, exercising, and sleeping on energy expenditure, the participants were instructed to get sufficient sleep the previous night. They were also instructed to abstain from eating, drinking anything except water, and engaging in intense exercise within 4 h before the start of the experiment.


**Table 1.** Participants involved in the experiment.

Asterisk (\*) indicates that the participant has a marine license.

#### *2.2. Design*

This case study analyzed the relationship between the wave direction projected by a visual simulator and the direction of human standing posture motion. The intervention for the participants was based on visual information. Figure 1 shows a schematic of the study. Wave images were presented to seven participants using a visual simulator, and the COP was monitored to determine the human posture motion. Each participant balanced themselves on a Wii Balance Board (RVL-021; Nintendo, Kyoto, Japan).

the COP was monitored to determine the human posture motion. Each participant bal-

Four image patterns were presented to seven participants using a visual simulator: an image without waves (Pattern 1), an image with 180° waves moving from front to rear (Pattern 2), an image with 135° waves moving from front left to rear right (Pattern 3), and an image with 90° waves moving from left to right (Pattern 4). One set of experiments consisted of a 15 min break in the sitting posture with the image of Pattern 1 and a 15 min intervention in the standing posture with one of the four image patterns. In the first, second, third, and fourth sets of experiments, the images of Patterns 1, 3, 2, and 4 were presented to the participants during the intervention phase, respectively. We instructed the participants to stand naturally by placing their feet shoulder-width apart during the in-

anced themselves on a Wii Balance Board (RVL-021; Nintendo, Kyoto, Japan).

**Figure 1.** Schematic of the study. **Figure 1.** Schematic of the study.

tervention phase.

*2.3. Data Collection and Processing*  The standing posture motion caused by the APAs was observed as a change in the COP. In this study, COP was measured using a Wii Balance Board. The specifications of the Wii Balance Board are listed in Table 2. The Wii Balance Board has four strain-gaugebased load sensors capable of obtaining movement data in the COP and communicating wirelessly with a computer via Bluetooth. A systematic review indicated that the Wii Balance Board can provide data that are concurrently valid with typical commercial force platforms. In addition, the board has reliability characteristics similar to those of force Four image patterns were presented to seven participants using a visual simulator: an image without waves (Pattern 1), an image with 180◦ waves moving from front to rear (Pattern 2), an image with 135◦ waves moving from front left to rear right (Pattern 3), and an image with 90◦ waves moving from left to right (Pattern 4). One set of experiments consisted of a 15 min break in the sitting posture with the image of Pattern 1 and a 15 min intervention in the standing posture with one of the four image patterns. In the first, second, third, and fourth sets of experiments, the images of Patterns 1, 3, 2, and 4 were presented to the participants during the intervention phase, respectively. We instructed the participants to stand naturally by placing their feet shoulder-width apart during the intervention phase.

platforms for static standing [16]. The intraclass correlation coefficient (ICC) is a statistical

#### test of reliability. Four studies reported the excellent reliability of the Wii Balance Board *2.3. Data Collection and Processing*

(ICC = 0.76 to 0.94) [17–20]. **Table 2.** Specifications of the Wii Balance Board. **Description Specification**  Manufacturer Nintendo Product family Wii Type name RVL-021 Communications standard Bluetooth ver.1.2 Wireless frequency 2.4 GHz The standing posture motion caused by the APAs was observed as a change in the COP. In this study, COP was measured using a Wii Balance Board. The specifications of the Wii Balance Board are listed in Table 2. The Wii Balance Board has four strain-gauge-based load sensors capable of obtaining movement data in the COP and communicating wirelessly with a computer via Bluetooth. A systematic review indicated that the Wii Balance Board can provide data that are concurrently valid with typical commercial force platforms. In addition, the board has reliability characteristics similar to those of force platforms for static standing [16]. The intraclass correlation coefficient (ICC) is a statistical test of reliability. Four studies reported the excellent reliability of the Wii Balance Board (ICC = 0.76 to 0.94) [17–20].

Sampling interval 0.01 s

Weight limit 136 kg **Table 2.** Specifications of the Wii Balance Board.


The COP was measured at a sampling rate of 100 Hz. A low-pass filter with a cut-off frequency of 1 Hz was applied to the COP signals to eliminate noise. The characteristics of the COP signals were evaluated in terms of the total length of the COP, AP/ML, which is the ratio of the AP length to the ML length of the COP, and the relationship between the ML and AP lengths of the COP, using scatter diagrams and regression lines. The sample size for the total length of the COG and *AP*/*ML* was set to seventeen, respectively, to design an effect size of at least 1.0, and the level of statistical power was at least 0.8. The effect

size indicates Cohen's d. The statistical power was calculated using the significance level, sample size, and effect size. effect size indicates Cohen's d. The statistical power was calculated using the significance level, sample size, and effect size.

The COP was measured at a sampling rate of 100 Hz. A low-pass filter with a cut-off frequency of 1 Hz was applied to the COP signals to eliminate noise. The characteristics of the COP signals were evaluated in terms of the total length of the COP, AP/ML, which is the ratio of the AP length to the ML length of the COP, and the relationship between the ML and AP lengths of the COP, using scatter diagrams and regression lines. The sample size for the total length of the COG and / was set to seventeen, respectively, to design an effect size of at least 1.0, and the level of statistical power was at least 0.8. The

#### *2.4. Experimental Environment 2.4. Experimental Environment*

*Sensors* **2022**, *22*, 5884 4 of 14

#### 2.4.1. Visual Simulator 2.4.1. Visual Simulator

A ship-handling simulator was used as a visual simulator. A ship-handling simulator reproduces a vessel-maneuvering environment as close as possible to an actual wheelhouse by combining navigation equipment and computer graphics technology. Figure 2 shows the ship-handling simulator used in this study at the National Institute of Technology, Toba College, Mie, Japan. The simulator was based on the International Convention on Standards of Training, Certification, and Watchkeeping for Seafarers (STCW). The horizontal and vertical viewing angles, centered on a gyrocompass installed at the center of the wheelhouse, located 3 m from the screen, were 225◦ and 30◦ , respectively. A ship-handling simulator was used as a visual simulator. A ship-handling simulator reproduces a vessel-maneuvering environment as close as possible to an actual wheelhouse by combining navigation equipment and computer graphics technology. Figure 2 shows the ship-handling simulator used in this study at the National Institute of Technology, Toba College, Mie, Japan. The simulator was based on the International Convention on Standards of Training, Certification, and Watchkeeping for Seafarers (STCW). The horizontal and vertical viewing angles, centered on a gyrocompass installed at the center of the wheelhouse, located 3 m from the screen, were 225° and 30°, respectively.

**Figure 2.** Ship handling simulator: (**a**) wheelhouse; (**b**) setting room of the simulating condition. **Figure 2.** Ship handling simulator: (**a**) wheelhouse; (**b**) setting room of the simulating condition.

2.4.2. Specifications of the Simulated Ship and Wave Images 2.4.2. Specifications of the Simulated Ship and Wave Images

A high-speed boat (length: 39.8 m, breadth: 9.00 m) was used for the entire pattern. The course and speed were 0° and 15 kn, respectively. The sea wind speed was set to zero. Table 3 lists the four wave image patterns in different directions. Pattern 1 was defined as an image without waves. For Patterns 2, 3, and 4, the wave directions were set to 180°, 135°, and 90°, respectively. The wave height and wave period were set to 3 m and 8 s, respectively; the condition was confirmed to occur in the posture motion in previous research using the visual simulator [13]. Figure 3 illustrates the wave directions with respect to the participant's position. The participants were positioned behind the steering wheel at a distance of 2.2 m from the gyrocompass located at the center of the screen and instructed to look at the screen (bowside). Figure 4 depicts the experimental setup. The cables connecting the participant to the measurement instruments were fixed to a belt on the waist of the participant to avoid limiting their standing posture motion. Figure 5 shows the simulated wave images used for each experimental condition. A high-speed boat (length: 39.8 m, breadth: 9.00 m) was used for the entire pattern. The course and speed were 0◦ and 15 kn, respectively. The sea wind speed was set to zero. Table 3 lists the four wave image patterns in different directions. Pattern 1 was defined as an image without waves. For Patterns 2, 3, and 4, the wave directions were set to 180◦ , 135◦ , and 90◦ , respectively. The wave height and wave period were set to 3 m and 8 s, respectively; the condition was confirmed to occur in the posture motion in previous research using the visual simulator [13]. Figure 3 illustrates the wave directions with respect to the participant's position. The participants were positioned behind the steering wheel at a distance of 2.2 m from the gyrocompass located at the center of the screen and instructed to look at the screen (bowside). Figure 4 depicts the experimental setup. The cables connecting the participant to the measurement instruments were fixed to a belt on the waist of the participant to avoid limiting their standing posture motion. Figure 5 shows the simulated wave images used for each experimental condition.

**Table 3.** Wave image patterns.


motion about the *y*-axis) was the largest in Pattern 4.

motion about the *y*-axis) was the largest in Pattern 4.

*Sensors* **2022**, *22*, 5884 5 of 14

 **Pattern 1 Pattern 2 Pattern 3 Pattern 4** 

 **Pattern 1 Pattern 2 Pattern 3 Pattern 4** 

Table 4 lists the motion of the simulated high-speed boat. The pitch motion (rotational motion about the *x*-axis) was the largest in Pattern 2. The roll motion (rotational

Table 4 lists the motion of the simulated high-speed boat. The pitch motion (rotational motion about the *x*-axis) was the largest in Pattern 2. The roll motion (rotational

Direction 0° 180° 135° 90° Height 0 m 3 m 3 m 3 m Period 0 s 8 s 8 s 8 s

Direction 0° 180° 135° 90° Height 0 m 3 m 3 m 3 m Period 0 s 8 s 8 s 8 s

**Table 3.** Wave image patterns.

**Table 3.** Wave image patterns.

**Figure 3.** Wave directions with respect to the participant's position. **Figure 3.** Wave directions with respect to the participant's position. **Figure 3.** Wave directions with respect to the participant's position.

**Figure 4.** Experimental setup. **Figure 4.** Experimental setup. **Figure 4.** Experimental setup.

Table 4 lists the motion of the simulated high-speed boat. The pitch motion (rotational motion about the *x*-axis) was the largest in Pattern 2. The roll motion (rotational motion about the *y*-axis) was the largest in Pattern 4.

**Table 4.** Motion of the simulated high-speed boat.


*Sensors* **2022**, *22*, 5884 6 of 14

(**a**)

**Figure 5.** Simulated wave images: (**a**) Pattern 1 (images without waves); (**b**) Pattern 2 (wave direction: 180°); (**c**) Pattern 3 (wave direction: 135°); (**d**) Pattern 4 (wave direction: 90°). **Figure 5.** Simulated wave images: (**a**) Pattern 1 (images without waves); (**b**) Pattern 2 (wave direction: 180◦ ); (**c**) Pattern 3 (wave direction: 135◦ ); (**d**) Pattern 4 (wave direction: 90◦ ). Pitch 0.00° −10.78°–10.26° −8.77°–7.50° −5.93°–5.87° Yaw 0.00° −0.02°–0.02° −0.11°–0.11° −0.13°–0.13°

#### **Table 4.** Motion of the simulated high-speed boat. *2.5. Evaluation Indicators 2.5. Evaluation Indicators*

**Pattern 1 Pattern 2 Pattern 3 Pattern 4**  Figure 6 presents a conceptual diagram of the COP length. Figure 6 presents a conceptual diagram of the COP length.

**Figure 6.** Conceptual diagram of the center of pressure (COP) length. **Figure 6.** Conceptual diagram of the center of pressure (COP) length.

The coordinate value ( , ) indicates the th COP sample, and ିଵ(ିଵ,ିଵ) represents the ( − 1)th COP sample. denotes the distance from ିଵ to . The total length of the COP is denoted by ெ and is calculated as follows: The coordinate value *C<sup>m</sup>* (*xm*,*ym*) indicates the *m*th COP sample, and *Cm*−1(*xm*−1,*ym*−1) represents the (*m* − 1)th COP sample. *L<sup>m</sup>* denotes the distance from *Cm*−<sup>1</sup> to *Cm*. The total length of the COP is denoted by *L<sup>M</sup>* and is calculated as follows:

$$L\_M = \sum\_{m=1}^M L\_m = \sum\_{m=1}^M \sqrt{(\mathbf{x}\_m - \mathbf{x}\_{m-1})^2 + (y\_m - y\_{m-1})^2}. \tag{1}$$

The coordinate value ( , ) indicates the th COP sample, and ିଵ(ିଵ,ିଵ) represents the ( − 1)th COP sample. denotes the distance from *Lxm* denotes the distance from *xm*−<sup>1</sup> to *xm*. The ML lengths of the COP are denoted by *LxM*, which is calculated as

$$L\_{\rm xM} = \sum\_{m=1}^{M} L\_{\rm xm} = \sum\_{m=1}^{M} \left| \mathbf{x}\_{m} - \mathbf{x}\_{m-1} \right|. \tag{2}$$

*Lym* denotes the distance from *ym*−<sup>1</sup> to *ym*. The AP length of the COP is denoted by *LyM*, which is calculated as

$$L\_{yM} = \sum\_{m=1}^{M} L\_{ym} = \sum\_{m=1}^{M} |y\_m - y\_{m-1}|\,. \tag{3}$$

*AP*/*ML* is the ratio of the AP length to the ML length of the COP. The ratio was calculated as follows:

$$AP/ML = \frac{L\_{yM}}{L\_{xM}}.\tag{4}$$

*M* was set to 3200. Data for more than 30 s were necessary to investigate the standing posture [21]. This period included four periodic wave images.

Seventeen data points were calculated for each wave pattern for *L<sup>M</sup>* and *AP*/*ML*, which were common to all participants. *L<sup>M</sup>* or *AP*/*ML*, corresponding to the participants changing the position of their feet on the Wii Balance Board or shifting the position of their COP to one foot, was removed from the dataset of the 17 data points for each wave pattern. Statistical outliers were removed from the dataset of the 17 data points for each wave pattern.

The characteristics of the standing posture motion of each participant were analyzed by calculating the mean values and standard deviations of the 17 *L<sup>M</sup>* or *AP*/*ML* data points for each wave pattern. Additionally, the characteristics of the standing posture motion of all participants were analyzed by calculating the mean values and standard deviations of *L<sup>M</sup>* or *AP*/*ML* collected from all participants for each wave pattern and by using scatter diagrams and regression lines of *LxM* and *LyM*. Tukey's method [22], which is a parametric multiple comparison method, was employed to examine significant differences in the total length of the COP (*LM*) and ratio (*AP*/*ML*) between each wave image pattern. The significance level was set at *p* < 0.05.

#### **3. Results**

Figure 7 shows an example of the COP locus. The sampled data represented by the red lines indicate that the participant changed the position of their feet on the Wii Balance Board (Figure 7). *L<sup>M</sup>* and *AP*/*ML* included in the sampled data, shown by red lines, were removed from the dataset of the 17 *L<sup>M</sup>* or *AP*/*ML* data points for each wave pattern. The *AP* and *ML* motions of the COP differed from those presented in Figure 7b–d depending on the difference in the wave direction.

Figure 8a–g illustrate the mean values and standard deviations of *L<sup>M</sup>* for each participant. Figure 8h shows the mean values and standard deviations of *L<sup>M</sup>* for all participants. *N* represents the number of valid *L<sup>M</sup>* that were removed from the data of the participants changing or shifting their foot positions on the Wii Balance Board or the actual data of statistical outliers from the dataset of the 17 data points for each wave pattern. Moreover, significant differences were detected in the total COP length between Pattern 1 and the other patterns (*p* < 0.05) (Figure 8h). Significant differences were detected in the total length of the COP length of participants, as shown in Figure 8a,c,d–f, as well as between the same pairs of wave patterns, as shown in Figure 8h (*p* < 0.05). As shown in Figure 8b, no significant differences were detected in the total COP between all the pairs of wave patterns. The levels of the effect size of the multiple comparison method, which show significant differences, were 1.03 to 4.60. The levels of the statistical power of the multiple comparison method, which show significant differences, were 0.82 to 1.00.

sion line decreased as the wave direction changed from 180° to 90°.

**Figure 7.** Example of the center of pressure (COP) locus (participant D): (**a**) Pattern 1; (**b**) Pattern 2 (wave direction: 180°); (**c**) Pattern 3 (wave direction: 135°); (**d**) Pattern 4 (wave direction: 90°). **Figure 7.** Example of the center of pressure (COP) locus (participant D): (**a**) Pattern 1; (**b**) Pattern 2 (wave direction: 180◦ ); (**c**) Pattern 3 (wave direction: 135◦ ); (**d**) Pattern 4 (wave direction: 90◦ ).

Figure 9a–g present the mean values and standard deviations of / for each participant. Figure 9h shows the mean values and standard deviations of / for all participants. denotes the number of valid / that were removed from the data of the participants changing or shifting their feet position on the Wii Balance Board or the actual data of statistical outliers from the dataset of 17 data points for each wave pattern. Significant differences were detected in the ratios between Patterns 1 and 3, Patterns 1 and 4, Patterns 2 and 3, and Patterns 2 and 4 (*p* < 0.05) (Figure 9h). Significant differences were detected in the ratio of the AP to ML length of the COP of participants, as shown in Figure 9a,d,e, as well as between the same pairs of patterns, as shown in Figure 9h (*p* < 0.05). As shown in Figure 9b,g, no significant differences were detected in the ratio between all pairs of wave patterns. The levels of the effect size of the multiple comparison method, which show significant differences, were 1.08 to 5.82. The level of the statistical power of

the multiple comparison method, which shows significant differences, was 1.00.

Figure 10 shows the scatter diagrams and regression lines of the ML (௫ெ) and AP (௬ெ) lengths of the ratio presented in Figure 9h for each wave pattern. The slope of the regression line under Patterns 1 and 2 was larger than 1.0; therefore, the AP COP motion was dominant compared with the ML COP motion. In contrast, the slope of the regression line under Patterns 3 and 4 was smaller than 1.0; therefore, the ML COP motion was dominant compared with the AP COP motion. Figure 10b–d show that the slope of the regres-

Figure 9a–g present the mean values and standard deviations of *AP*/*ML* for each participant. Figure 9h shows the mean values and standard deviations of *AP*/*ML* for all participants. *N* denotes the number of valid *AP*/*ML* that were removed from the data of the participants changing or shifting their feet position on the Wii Balance Board or the actual data of statistical outliers from the dataset of 17 data points for each wave pattern. Significant differences were detected in the ratios between Patterns 1 and 3, Patterns 1 and 4, Patterns 2 and 3, and Patterns 2 and 4 (*p* < 0.05) (Figure 9h). Significant differences were detected in the ratio of the AP to ML length of the COP of participants, as shown in Figure 9a,d,e, as well as between the same pairs of patterns, as shown in Figure 9h (*p* < 0.05). As shown in Figure 9b,g, no significant differences were detected in the ratio between all pairs of wave patterns. The levels of the effect size of the multiple comparison method, which show significant differences, were 1.08 to 5.82. The level of the statistical power of the multiple comparison method, which shows significant differences, was 1.00.

*Sensors* **2022**, *22*, 5884 9 of 14

**Figure 8.** Total length of the center of pressure (COP) (ெ): (**a**) participant A; (**b**) participant B; (**c**) participant C; (**d**) participant D; (**e**) participant E; (**f**) participant F; (**g**) participant G; (**h**) all participants. is the number of valid ெ. The asterisk (\*) indicates a significant difference (\* *p* < 0.05). **Figure 8.** Total length of the center of pressure (COP) (*LM*): (**a**) participant A; (**b**) participant B; (**c**) participant C; (**d**) participant D; (**e**) participant E; (**f**) participant F; (**g**) participant G; (**h**) all participants. *N* is the number of valid *LM*. The asterisk (\*) indicates a significant difference (\* *p* < 0.05).

**Figure 9.** Ratio of the anteroposterior (AP) length to the mediolateral (ML) length of the center of pressure (COP) (/): (**a**) participant A; (**b**) participant B; (**c**) participant C; (**d**) participant D; (**e**) participant E; (**f**) participant F; (**g**) participant G; (**h**) all participants. is the number of valid . The asterisk (\*) indicates a significant difference (\* *p* < 0.05). **Figure 9.** Ratio of the anteroposterior (AP) length to the mediolateral (ML) length of the center of pressure (COP) (*AP*/*ML*): (**a**) participant A; (**b**) participant B; (**c**) participant C; (**d**) participant D; (**e**) participant E; (**f**) participant F; (**g**) participant G; (**h**) all participants. *N* is the number of valid *R*. The asterisk (\*) indicates a significant difference (\* *p* < 0.05).

Figure 10 shows the scatter diagrams and regression lines of the *ML* (*LxM*) and *AP* (*LyM*) lengths of the ratio presented in Figure 9h for each wave pattern. The slope of the regression line under Patterns 1 and 2 was larger than 1.0; therefore, the *AP* COP motion was dominant compared with the *ML* COP motion. In contrast, the slope of the regression line under Patterns 3 and 4 was smaller than 1.0; therefore, the *ML* COP motion was dominant compared with the *AP* COP motion. Figure 10b–d show that the slope of the regression line decreased as the wave direction changed from 180◦ to 90◦ . *Sensors* **2022**, *22*, 5884 11 of 14

**Figure 10.** Scatter diagram and regression lines of the mediolateral (ML) and anteroposterior (AP) lengths of the ratio shown in Figure 9h: (**a**) Pattern 1; (**b**) Pattern 2 (wave direction: 180°); (**c**) Pattern 3 (wave direction: 135°); (**d**) Pattern 4 (wave direction: 90°). **Figure 10.** Scatter diagram and regression lines of the mediolateral (ML) and anteroposterior (AP) lengths of the ratio shown in Figure 9h: (**a**) Pattern 1; (**b**) Pattern 2 (wave direction: 180◦ ); (**c**) Pattern 3 (wave direction: 135◦ ); (**d**) Pattern 4 (wave direction: 90◦ ).

#### **4. Discussion 4. Discussion**

The effects of unidirectional visual information on human standing posture motion have been previously studied [23–25]. These studies focused on the relationship between a unidirectional moving screen or moving object and the human standing posture motion while participants watched a moving object [13–15,23–25]. The effects of unidirectional visual information on human standing posture motion have been previously studied [23–25]. These studies focused on the relationship between a unidirectional moving screen or moving object and the human standing posture motion while participants watched a moving object [13–15,23–25].

In the current research, only sea-waves were shown to the participants using a visual simulator, and the experimental results reflected the effect of visual information on human standing posture. Figure 8h shows that the total length of the COP motion in the participants watching the wave motion (Patterns 2–4) was larger than that in those not watching the wave motion (Pattern 1). Figure 9h shows that a significant difference in the COP motion was observed depending on whether the sea-waves were incident on the In the current research, only sea-waves were shown to the participants using a visual simulator, and the experimental results reflected the effect of visual information on human standing posture. Figure 8h shows that the total length of the COP motion in the participants watching the wave motion (Patterns 2–4) was larger than that in those not watching the wave motion (Pattern 1). Figure 9h shows that a significant difference in the COP motion was observed depending on whether the sea-waves were incident on the images on the screen from the lateral direction, primarily because the COP motion

images on the screen from the lateral direction, primarily because the COP motion under Patterns 1 and 2 could be distinguished from that under Patterns 3 and 4. Figure 10 illus-

ticipants' posture motions was not an illusion of body movement but real body posture motions based on visual sensation. For sea-waves approaching head-on, the participants maintained their posture by moving back and forth, whereas they stood firmly on the deck for waves approaching laterally. In the quiet standing posture, the COG in the AP direction is controlled by the ankle dorsiflexors and plantar flexors, whereas the COG in the ML direction is controlled by the hip abductors and adductors [26–29]. Thus, humans

under Patterns 1 and 2 could be distinguished from that under Patterns 3 and 4. Figure 10 illustrates that the *ML* motion of the COP increased with the wave directions changing to 180◦ , 135◦ , and 90◦ . Most participants exhibited standing posture motion. The sense of the participants' posture motions was not an illusion of body movement but real body posture motions based on visual sensation. For sea-waves approaching head-on, the participants maintained their posture by moving back and forth, whereas they stood firmly on the deck for waves approaching laterally. In the quiet standing posture, the COG in the *AP* direction is controlled by the ankle dorsiflexors and plantar flexors, whereas the COG in the *ML* direction is controlled by the hip abductors and adductors [26–29]. Thus, humans maintain their posture based on visual information through *AP* and *ML* motions, owing to the structure of their lower limbs. In the current study, we found that the standing postures of the participants stabilized themselves according to the wave directions projected by a visual simulator. This showed that the participants predicted ship motion from the wave images and controlled their COP through the APAs.

In the experimental results (Figures 8 and 9), individual differences were observed in the standing postural motions based on visual information. These differences may indicate the APA ability of each participant. Figure 8b shows negligible differences in the participants' standing postural motion irrespective of the waves. Figure 9b,g also show no significant difference in the participants' standing postural motion between the directions of the waves. APAs are categorized as natural APAs and learned APAs [30]. When a participant experiences a situation in which they can control their standing posture using only CPR against a ship's motion, APAs may not occur for similar ship motions. The analysis of human standing postural motion based on the differences between the wave directions projected by a visual simulator using COP can be used to evaluate the physical aptitude of crews. Further experiments are required to confirm this hypothesis.

The assumed role of APAs is to minimize perturbations in standing posture, and APAs are prepared by the central nervous system before CPR occurs. Therefore, visual information is potentially effective for the predictive control of the standing posture. Most crew members, except navigators, cannot see ocean waves or the tilt of a ship, as they work in confined spaces. Consequently, they control their posture against ship motion by using vestibular and proprioceptive information, without relying on vision, which may increase fatigue. By setting a screen to project the sea-waves inboard, the crew members can develop visual APAs for standing posture adjustments, which may reduce fatigue. The evaluation approach of human standing posture motion using the multi-directional visual information presented in this study is a new method for evaluating APAs. Patients with Parkinson's disease have been shown to exhibit abnormalities in programming APAs, which contribute to their postural instability [31,32]. Thus, evaluating the APAs of patients with postural accommodation disorder using the evaluation method of this study may assist physical therapists in performing effective rehabilitation.

The limitation of this study is that the results cannot be applied to female populations and to age groups other than 20–23 years. To generalize this method, randomized controlled trials ought to be performed with participants of multiple age groups, including men and women.

#### **5. Conclusions**

The direction of the standing postural motion of the participants depended on the direction of sea-waves, which was projected by a visual simulator and was not an illusion of body movement. Individual differences in standing postural motion may indicate the ability of APAs for each participant based on their experience.

We would like to develop a method for evaluating the adaptability of crew members by conducting additional experiments to test this hypothesis. This method can be applied to the realization of a system for evaluating adaptability to crew training. The fatigue of crew members inboard may be reduced by APAs through the development of a wave motion presentation system.

**Author Contributions:** Conceptualization, T.S.; methodology, T.S.; software, R.D.; validation, R.D.; formal analysis, R.D. and T.S.; investigation, R.D.; resources, T.S.; data curation, R.D.; writing original draft preparation, R.D. and T.S.; writing—review and editing, R.D. and T.S.; visualization, R.D.; supervision, T.S.; project administration, T.S.; funding acquisition, R.D. and T.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was supported in part by the Japan Society for the Promotion of Science (JSPS), grant numbers 17H02056, 18K13951, and 21K04575.

**Institutional Review Board Statement:** The study was conducted in accordance with the guidelines of the Declaration of Helsinki and approved by the Ethics Committee of the National Institute of Technology, Toba College, Mie, Japan (No. 2401 on 19 December 2012).

**Informed Consent Statement:** Informed consent was obtained from all the subjects involved in the study. Written informed consent was obtained from the subjects for publication of this study.

**Data Availability Statement:** The data supporting the findings of this study are available from the corresponding author (S.T.) upon reasonable request.

**Acknowledgments:** The experiments described in this report were supported by the National Institute of Technology, Toba College, Japan.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Exploration of an Inflection Point of Ventilation Parameters with Anaerobic Threshold Using Strucchange**

**Takenori Aida <sup>1</sup> , Akira Shionoya 1,\*, Hirofumi Nonaka <sup>1</sup> , Kouji Hayami <sup>2</sup> , Hisashi Uchiyama <sup>1</sup> , Masahiro Nagamori <sup>1</sup> , Satoshi Ohhashi <sup>1</sup> , Mai Kobayashi <sup>3</sup> , Tsugumi Takayama <sup>4</sup> and Shinji Kimura <sup>4</sup>**


**Abstract:** (1) Background: When measuring anaerobic work threshold (AT), the conventional V-slope method includes the subjectivity of the examiner, which cannot be eliminated completely. Therefore, we implemented an engineering method using strucchange to objectively search for the inflection point of AT. (2) Methods: Seventeen subjects (15 men and 2 women) were included in the study. The subjects rode an ergometer and performed a ramp load test for 18 min and 30 s. (3) Results: In VE (Ventilation), 11 out of 12 subjects had the same results with 95% confidence intervals for the AT by the strucchange and respiratory metabolic apparatus. In VCO<sup>2</sup> (Carbon dioxide emissions), 9 out of 12 subjects had the same results with 95% confidence intervals for the AT with the strucchange and respiratory metabolic apparatus. In VE, 3 out of 12 subjects showed the same results for respiratory metabolic analysis and the AT by the V-slope method. In VCO<sup>2</sup> , 3 out of 12 subjects showed the same results for the respiratory metabolic analysis and AT by the V-slope method in VCO<sup>2</sup> . (4) Conclusions: Strucchange was more objective and significant in identifying the AT than the V-slope method.

**Keywords:** anaerobic threshold; inflection-point exploration; strucchange; ventilation expired gas volume; excess CO<sup>2</sup> production

#### **1. Introduction**

In recent years, the average life expectancy in Japan has been nearing 100 years. Therefore, many researchers are focusing on the relationship between long life expectancy and good health throughout one's life. In other words, studies on maintaining good health without compromising the quality of life (QOL) have been gaining popularity. However, good health and QOL are directly related to individual awareness. Although the current coronavirus pandemic is forcing people to restrict their daily activities, awareness regarding the importance of individual health is being raised by making people aware of the importance of wearing masks, hand sanitization, social distancing, vaccination, and other measures. The risks of lifestyle-related diseases, such as cancer, diabetes, cardiovascular diseases, as well as age-related decline in physical functions, are known to be reduced with appropriate physical activity, especially indoor exercises to relieve the stress caused by the lack of physical activity. Toshima et al. [1] reported that heart disease has been the second leading cause of death since 1985 due to dietary changes. In addition, ischemic heart disease (IHD) has become the leading cause of death worldwide [2]. Wannamethee et al. [3] reported that in addition to preventing cardiac disease, maintaining the quality of life (QOL) after cardiac disease leads

**Citation:** Aida, T.; Shionoya, A.; Nonaka, H.; Hayami, K.; Uchiyama, H.; Nagamori, M.; Ohhashi, S.; Kobayashi, M.; Takayama, T.; Kimura, S. Exploration of an Inflection Point of Ventilation Parameters with Anaerobic Threshold Using Strucchange. *Sensors* **2022**, *22*, 2682. https://doi.org/10.3390/s22072682

Academic Editor: Nunzio Cennamo

Received: 7 March 2022 Accepted: 29 March 2022 Published: 31 March 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

to an increased healthy life expectancy. Therefore, cardiac rehabilitation exercise therapy is essential for secondary prevention in patients with a history of heart disease [4]. Although excessive exercise load is contraindicated, an appropriate exercise index is used in rehabilitation exercise therapy. An optimal exercise load intensity index called the anaerobic threshold [5] (AT, also known as anaerobic work threshold) considers safety and efficiency. The AT reflects exercise intensity and O<sup>2</sup> uptake immediately prior to anaerobic metabolism occurring alongside aerobic metabolism during load-incremental exercise when the associated change in gas-exchange parameters occurs. Many studies on exercise therapy that use an appropriate exercise load index have been conducted on the elderly and cancer patients [6,7]. Tanebe et al. [8] reported that exercises that do not exceed the AT do not cause an excessive elevation of catecholamines in the blood and are safe exercises. There are two methods for determining the AT, namely the ventilatory threshold (the aforementioned ventilatory gas analysis) and lactate threshold (the lactate concentration in blood) methods. The former is non-invasive for the examinee and is the standard method used [9]. However, the ventilatory threshold is not suitable as an indicator of endurance training. The latter method is invasive, requires measurement techniques using sophisticated and expensive equipment, and relies on the experience of physicians and healthcare professionals [10]. In a previous study on AT determination, Tsubusadani et al. [11] described a method called cardiac work threshold, which determines the AT based on non-invasive data such as heart rate; however, this method requires improvement in measurement accuracy due to the effect of respiration on heart rate variability. Takemura et al. [12] proposed a technique to control respiratory rate during progressive load exercise using a bicycle ergometer, such as locomotor–respiratory coupling (LRC). Kimoto et al. [13] found that the maximum intensity spectrum of the frequency analyzed respiratory curve (corresponding to the respiratory rate) was significantly lower than that of the point at which the transition from low-frequency to high-frequency components was determined and used these intensity values as the exercise intensities before and after AT. The V-slope method [14,15], a conventional analog (Figure 1), is also used as an AT identification method. This method plots the change in carbon dioxide emissions (VCO2) against oxygen uptake (VO2) in exhaled air. However, the extraction of structural change points [16] includes subjectivity by the examiner and cannot be objectively identified. In other words, a method to objectively determine or identify the AT has not yet been established. Against this background, the purpose of this study was to develop a more objective method of searching for inflection points of the AT using strucchange [17] for the change points of each ventilation index in exhaled gas during exercise loading. In Wasserman's definition [18], the AT is defined based on each ventilation index (Figure 2) [19]. An explanation of the definition based on each ventilation index is as follows. (1) Nonlinear increase in ventilation (VE: L/min) with increasing exercise intensity; (2) nonlinear increase in VCO<sup>2</sup> (ml/kg/min); (3) nonlinear increase in respiratory quotient RQ; (4) increase in oxygen equivalent VE/VO<sup>2</sup> without a change in carbon dioxide equivalent VE/VCO2, and (5) increase in end-expiratory oxygen concentration (PETO2: mmHg) without a change in end-expiratory carbon dioxide concentration (PETCO2: mmHg).

#### **VCO2(ml/min)** 3000

**VCO2(ml/min)**

*Sensors* **2022**, *22*, x FOR PEER REVIEW 3 of 14

**Figure 1.** AT using the V-slope method. Eric et al. [14] presented a gas exchange plot from the submaximal exercise test of a representative participant to illustrate identification of the AT. With the V-slope method, the VCO2/VO2 plot is used to identify the point at which the VCO2 starts to increase more rapidly than VO2. The vertical dashed line represents the AT. Figure 1 was created based on subject data. **Figure 1.** AT using the V-slope method. Eric et al. [14] presented a gas exchange plot from the submaximal exercise test of a representative participant to illustrate identification of the AT. With the V-slope method, the VCO2/VO<sup>2</sup> plot is used to identify the point at which the VCO<sup>2</sup> starts to increase more rapidly than VO<sup>2</sup> . The vertical dashed line represents the AT. Figure 1 was created based on subject data. maximal exercise test of a representative participant to illustrate identification of the AT. With the V-slope method, the VCO2/VO2 plot is used to identify the point at which the VCO2 starts to increase more rapidly than VO2. The vertical dashed line represents the AT. Figure 1 was created based on subject data.

**Figure 1.** AT using the V-slope method. Eric et al. [14] presented a gas exchange plot from the sub-

**Figure 2.** The AT is defined based on each ventilation index. Figure 2 was created by partially reorganizing the subject data. **Figure 2.** The AT is defined based on each ventilation index. Figure 2 was created by partially reorganizing the subject data. **Figure 2.** The AT is defined based on each ventilation index. Figure 2 was created by partially reorganizing the subject data.

A total of 17 subjects (age 21 ± 1.4 years; BMI 21 ± 1.9), 15 healthy men and 2 healthy women, engaged in regular daily exercise were included in the study. Informed consent

A total of 17 subjects (age 21 ± 1.4 years; BMI 21 ± 1.9), 15 healthy men and 2 healthy women, engaged in regular daily exercise were included in the study. Informed consent

**2. Materials and Methods** 

**2. Materials and Methods** 

*2.1. Subjects* 

*2.1. Subjects* 

#### **2. Materials and Methods**

#### *2.1. Subjects*

A total of 17 subjects (age 21 ± 1.4 years; BMI 21 ± 1.9), 15 healthy men and 2 healthy women, engaged in regular daily exercise were included in the study. Informed consent was obtained from all the subjects involved in the study, which was conducted according to the guidelines of the Helsinki Declaration. This study was approved by the ethical review of the Nagaoka University of Technology (project identification code: H29-4, approval date: 12 May 2017). The experiments were conducted in the sports engineering laboratory of the Nagaoka University of Technology and the laboratory of the Seiryou Rehabilitation Institute. Table 1 lists the BMI and smoking history of the subjects.

**Table 1.** BMI and smoking history of the subjects.


#### *2.2. Experimental Protocol and Setup*

The experimental protocol and apparatus are presented in Figures 3 and 4, respectively. After adjusting the saddle height, the subject rode a bicycle ergometer (AEROBIKE EZ101, COMBIWELLNESS Corp., Konami Sports Co., Ltd., Tokyo, Japan). Both pedals were fixed, and they wore an expiratory gas mask. To stabilize the subject's heart rate, the subject was kept in a resting state (Rest) on the bicycle for 3 min and then warmed up for 3 min at a constant pedal rotation speed of 50 rpm on the ergometer (baseline load: 20 W). The load level (10 W) was increased every 30 s, and the pedal speed was maintained at 50 rpm. The duration of the exercise was 9 min and 30 s. After the exercise, there was a cool-down period of three minutes before the termination of the experiment. The total time from rest to the end of the experiment was 18 min and 30 s. The criterion for discontinuation of the experiment was when the heart rate became higher than the subject's maximum heart rate (220 − age (beats/min), where "age" is the age of the subject) [20], or when the subject complained of physical discomfort. The respiratory metabolic measurement device was Vmax 29 s, manufactured by Sensor Medics. The analysis was performed using the breath-by-breath method [21].

#### *2.3. Experimental Analysis*

Each ventilation index was recorded on a PC, converted to A/D, averaged for 10 s, and smoothed with a simple moving average (3 points). The smoothed data were saved on a USB drive and analyzed using the strucchange package in R language on another PC. The AT of each subject was calculated using the analysis function (V-slope method) of the

respiratory metabolic measurement device (Vmax 29 s) and reflected as a solid vertical line in the graph (time axis) of each ventilation index. Furthermore, the AT identification by the V-slope method, which is an analog method, was performed by two examiners: one of the authors (examiner A) and a clinical laboratory technician with more than 20 years of clinical experience (examiner B). Each examiner performed the V-slope method twice, and the mean value (converted to W) was used. Interclass correlation coefficients (ICCs) [22] were calculated using Excel 2016, assessing the reliability. *Sensors* **2022**, *22*, x FOR PEER REVIEW 5 of 14 *Sensors* **2022**, *22*, x FOR PEER REVIEW 5 of 14

stop

**Figure 3.** Experimental protocol. **Figure 3.** Experimental protocol. **Figure 3.** Experimental protocol.

**Figure 4.** Experimental setup. **Figure 4.** Experimental setup. **Figure 4.** Experimental setup.

#### **3. Results**

**3. Results** 

**3. Results** 

*2.3. Experimental Analysis 2.3. Experimental Analysis 3.1. Results of V-Slope Method and Strucchange and Respiratory Metabolic Analysis*

Each ventilation index was recorded on a PC, converted to A/D, averaged for 10 s, and smoothed with a simple moving average (3 points). The smoothed data were saved Each ventilation index was recorded on a PC, converted to A/D, averaged for 10 s, and smoothed with a simple moving average (3 points). The smoothed data were saved In Figure 5, the AT by the respiratory metabolic apparatus, strucchange, and V-slope method are shown as blue circles, red squares, and black circles, respectively.

on a USB drive and analyzed using the strucchange package in R language on another PC. The AT of each subject was calculated using the analysis function (V-slope method) of the respiratory metabolic measurement device (Vmax 29 s) and reflected as a solid vertical

one of the authors (examiner A) and a clinical laboratory technician with more than 20 years of clinical experience (examiner B). Each examiner performed the V-slope method twice, and the mean value (converted to W) was used. Interclass correlation coefficients

In Figure 5, the AT by the respiratory metabolic apparatus, strucchange, and V-slope

(ICCs) [22] were calculated using Excel 2016, assessing the reliability.

*3.1. Results of V-Slope Method and Strucchange and Respiratory Metabolic Analysis* 

method are shown as blue circles, red squares, and black circles, respectively.

on a USB drive and analyzed using the strucchange package in R language on another PC. The AT of each subject was calculated using the analysis function (V-slope method) of the respiratory metabolic measurement device (Vmax 29 s) and reflected as a solid vertical

one of the authors (examiner A) and a clinical laboratory technician with more than 20 years of clinical experience (examiner B). Each examiner performed the V-slope method twice, and the mean value (converted to W) was used. Interclass correlation coefficients

In Figure 5, the AT by the respiratory metabolic apparatus, strucchange, and V-slope

(ICCs) [22] were calculated using Excel 2016, assessing the reliability.

*3.1. Results of V-Slope Method and Strucchange and Respiratory Metabolic Analysis* 

method are shown as blue circles, red squares, and black circles, respectively.

3

**Figure 5.** Comparison of each AT for subject K by the V-slope method, strucchange, and respiratory metabolism analysis. **Figure 5.** Comparison of each AT for subject K by the V-slope method, strucchange, and respiratory metabolism analysis. **Figure 5.** Comparison of each AT for subject K by the V-slope method, strucchange, and respiratory metabolism analysis.

#### *3.2. Results from Strucchange and Respiratory Metabolic Analysis 3.2. Results from Strucchange and Respiratory Metabolic Analysis 3.2. Results from Strucchange and Respiratory Metabolic Analysis*

The strucchange results for the VE of subject K are shown in Figure 6. The horizontal axis shows the number of breakpoints, the left vertical axis shows the BIC analysis value, and the right vertical axis shows the RSS analysis value. The number of breakpoints is the minimum value for both BIC and RSS, indicating three breakpoints based on strucchange. The strucchange results for the VE of subject K are shown in Figure 6. The horizontal axis shows the number of breakpoints, the left vertical axis shows the BIC analysis value, and the right vertical axis shows the RSS analysis value. The number of breakpoints is the minimum value for both BIC and RSS, indicating three breakpoints based on strucchange. The strucchange results for the VE of subject K are shown in Figure 6. The horizontal axis shows the number of breakpoints, the left vertical axis shows the BIC analysis value, and the right vertical axis shows the RSS analysis value. The number of breakpoints is the minimum value for both BIC and RSS, indicating three breakpoints based on strucchange.

**Figure 6.** Number of breakpoints (VE). **Figure 6.** Number of breakpoints (VE).

be within 95%.

**Figure 6.** Number of breakpoints (VE).

The results of the strucchange results of VE (subject K) after smoothing are shown in Figure 7. The vertical solid line shows the AT by the respiratory metabolic apparatus. The vertical dashed line indicates the change point by strucchange, and the red H-shape indicates the confidence interval of strucchange. The confidence intervals were calculated to The results of the strucchange results of VE (subject K) after smoothing are shown in Figure 7. The vertical solid line shows the AT by the respiratory metabolic apparatus. The vertical dashed line indicates the change point by strucchange, and the red H-shape indicates the confidence interval of strucchange. The confidence intervals were calculated to be within 95%. The results of the strucchange results of VE (subject K) after smoothing are shown in Figure 7. The vertical solid line shows the AT by the respiratory metabolic apparatus. The vertical dashed line indicates the change point by strucchange, and the red H-shape indicates the confidence interval of strucchange. The confidence intervals were calculated to be within 95%.

**Figure 7.** The results of the strucchange results of VE (subject K) after smoothing. **Figure 7.** The results of the strucchange results of VE (subject K) after smoothing. **Figure 7.** The results of the strucchange results of VE (subject K) after smoothing.

The results of strucchange in the VCO2 of subject K are shown in Figure 8. Because the number of change points is minimal for both BIC and RSS analyses, there are four change points by strucchange. The results of strucchange in the VCO<sup>2</sup> of subject K are shown in Figure 8. Because the number of change points is minimal for both BIC and RSS analyses, there are four change points by strucchange. The results of strucchange in the VCO2 of subject K are shown in Figure 8. Because the number of change points is minimal for both BIC and RSS analyses, there are four change points by strucchange.

**Figure 8.** The number of breakpoints (VCO2). **Figure 8.** The number of breakpoints (VCO2). **Figure 8.** The number of breakpoints (VCO<sup>2</sup> ).

be within 95%.

The results of strucchange results of VCO2 (subject K) after smoothing are shown in Figure 9. The vertical solid line shows the AT by the respiratory metabolic apparatus. The vertical dashed line indicates the change point by strucchange, and the red H-shape indicates the confidence interval of strucchange. The confidence intervals were calculated to The results of strucchange results of VCO2 (subject K) after smoothing are shown in Figure 9. The vertical solid line shows the AT by the respiratory metabolic apparatus. The vertical dashed line indicates the change point by strucchange, and the red H-shape indicates the confidence interval of strucchange. The confidence intervals were calculated to be within 95%. The results of strucchange results of VCO<sup>2</sup> (subject K) after smoothing are shown in Figure 9. The vertical solid line shows the AT by the respiratory metabolic apparatus. The vertical dashed line indicates the change point by strucchange, and the red H-shape indicates the confidence interval of strucchange. The confidence intervals were calculated to be within 95%.

**Figure 9.** The results of strucchange results of VCO2 (subject K) after smoothing. **Figure 9.** The results of strucchange results of VCO<sup>2</sup> (subject K) after smoothing. **Figure 9.** The results of strucchange results of VCO2 (subject K) after smoothing.

The number of breakpoints (RQ). The results of strucchange in the VCO2 of subject K are shown in Figure 10. Because the number of change points is analyzed through both BIC and RSS analyses, there are three change points detected by strucchange. The number of breakpoints (RQ). The results of strucchange in the VCO<sup>2</sup> of subject K are shown in Figure 10. Because the number of change points is analyzed through both BIC and RSS analyses, there are three change points detected by strucchange. The number of breakpoints (RQ). The results of strucchange in the VCO2 of subject K are shown in Figure 10. Because the number of change points is analyzed through both BIC and RSS analyses, there are three change points detected by strucchange.

**Figure 10.** The number of breakpoints (RQ). **Figure 10.** The number of breakpoints (RQ). **Figure 10.** The number of breakpoints (RQ).

to be within 95%.

The results of strucchange results of RQ (subject K) after smoothing are shown in Figure 11. The vertical solid line shows the AT by the respiratory metabolic apparatus. The vertical dashed line indicates the change point by strucchange, and the red H-shape indicates the confidence interval of strucchange. The confidence intervals were calculated The results of strucchange results of RQ (subject K) after smoothing are shown in Figure 11. The vertical solid line shows the AT by the respiratory metabolic apparatus. The vertical dashed line indicates the change point by strucchange, and the red H-shape indicates the confidence interval of strucchange. The confidence intervals were calculated to be within 95%. The results of strucchange results of RQ (subject K) after smoothing are shown in Figure 11. The vertical solid line shows the AT by the respiratory metabolic apparatus. The vertical dashed line indicates the change point by strucchange, and the red H-shape indicates the confidence interval of strucchange. The confidence intervals were calculated to be within 95%.

**Figure 11.** The results of strucchange results of RQ (subject K) after smoothing. **Figure 11.** The results of strucchange results of RQ (subject K) after smoothing.

#### **4. Discussion 4. Discussion**

#### *4.1. Comparison between Strucchange and Respiratory Metabolic Analysis 4.1. Comparison between Strucchange and Respiratory Metabolic Analysis*

As shown in Figures 7, 9, and 11, strucchange was used to detect the points of change in the VE, VCO2,and RQ associated with a gradual increase in exercise load, and the points of change in all 17 subjects (A–Q) were detected. This is a result of the structural change extraction by strucchange. The background of the extraction of the structural change point is considered to be the process of the increase in CO2 storage in the tissue [16], starting from the buffering by HCO3− against the increase of CO2 in the blood during the gradual increase in exercise load. One hundred watt AT detection from Figure 7, and one hundred five watt AT detection from Figure 9. The AT detection by strucchange was close to each other from the two results. In other subjects, Table 2 shows that 11 out of 12 subjects (B– L) had the same results with 95% confidence intervals for the AT by the strucchange and respiratory metabolic apparatus (excluding those who could not be analyzed). Table 3 shows that 9 out of 12 subjects (B–H, J, and K) had the same results with 95% confidence intervals for the AT with strucchange and respiratory metabolic apparatus (excluding those who could not be analyzed). Table 4 shows that 2 out of the 17 subjects (B,H) had the same results, that is, 95% confidence intervals for the AT with strucchange and respiratory metabolic apparatus (excluding those who could not be analyzed). Compared with the aforementioned results for VE and VCO2, the results showed a significant decrease in the number of people. The respiratory quotient, RQ, is expressed as VCO2/VO2 and is defined as a measure of carbon dioxide emissions divided by oxygen uptake. As described previously, VCO2 or VO2 can be analyzed separately; however, when they are analyzed as a combined parameter, the results differ significantly due to the inclusion of errors from the analysis of each parameter. The cause of this difference has not yet been identified. Therefore, the RQ results require further investigation. There are two reasons why the respiratory metabolic analysis system could not detect the change points. One reason is that the subject could not continue the experiment and became exhausted. The other reason is the consideration of individual physiological effects on the subject. In the latter case, it is possible that the conventional V-slope method [16] could not detect this effect. As Nishijima et al. [23,24] pointed out, it is possible that VCO2 was shifted to the right in the As shown in Figures 7, 9 and 11, strucchange was used to detect the points of change in the VE, VCO2, and RQ associated with a gradual increase in exercise load, and the points of change in all 17 subjects (A–Q) were detected. This is a result of the structural change extraction by strucchange. The background of the extraction of the structural change point is considered to be the process of the increase in CO<sup>2</sup> storage in the tissue [16], starting from the buffering by HCO<sup>3</sup> − against the increase of CO<sup>2</sup> in the blood during the gradual increase in exercise load. One hundred watt AT detection from Figure 7, and one hundred five watt AT detection from Figure 9. The AT detection by strucchange was close to each other from the two results. In other subjects, Table 2 shows that 11 out of 12 subjects (B–L) had the same results with 95% confidence intervals for the AT by the strucchange and respiratory metabolic apparatus (excluding those who could not be analyzed). Table 3 shows that 9 out of 12 subjects (B–H, J, and K) had the same results with 95% confidence intervals for the AT with strucchange and respiratory metabolic apparatus (excluding those who could not be analyzed). Table 4 shows that 2 out of the 17 subjects (B,H) had the same results, that is, 95% confidence intervals for the AT with strucchange and respiratory metabolic apparatus (excluding those who could not be analyzed). Compared with the aforementioned results for VE and VCO2, the results showed a significant decrease in the number of people. The respiratory quotient, RQ, is expressed as VCO2/VO<sup>2</sup> and is defined as a measure of carbon dioxide emissions divided by oxygen uptake. As described previously, VCO<sup>2</sup> or VO<sup>2</sup> can be analyzed separately; however, when they are analyzed as a combined parameter, the results differ significantly due to the inclusion of errors from the analysis of each parameter. The cause of this difference has not yet been identified. Therefore, the RQ results require further investigation. There are two reasons why the respiratory metabolic analysis system could not detect the change points. One reason is that the subject could not continue the experiment and became exhausted. The other reason is the consideration of individual physiological effects on the subject. In the latter case, it is possible that the conventional V-slope method [16] could not detect this effect. As Nishijima et al. [23,24] pointed out, it is possible that VCO<sup>2</sup> was shifted to the right in the V-slope method during gradual exercise, and the AT could not be detected. This may be

V-slope method during gradual exercise, and the AT could not be detected. This may be

due to the state in which increased CO<sup>2</sup> in the blood is not sufficiently discharged from the alveoli during exercise load escalation; that is, the chemoreceptors are not functioning. In other words, CO<sup>2</sup> gas is stored in the alveoli and is not sufficiently expelled from the alveoli. Therefore, it is possible that the CO<sup>2</sup> that is no longer expelled is absorbed by the alveoli and circulated back into the bloodstream, causing a rightward shift (Figure 12) [23]. First, the conventional V-slope method is based on the complete decomposition of glucose C6H12O<sup>6</sup> + 6O<sup>2</sup> → 6CO<sup>2</sup> + 6H2O. It is difficult to say that the AT can be identified theoretically. Second, the detection algorithm used in the respiratory metabolic apparatus was based on the V-slope method. As long as the slope of the increase in VCO<sup>2</sup> relative to VO<sup>2</sup> is assumed to be greater than θ = 45◦ , it is challenging to detect AT, as shown in the experimental results. This result suggests that AT detection is insufficient in respiratory metabolic analysis based on the V-slope method, whereas that based on strucchange, which extracts structural change points, is significant. due to the state in which increased CO2 in the blood is not sufficiently discharged from the alveoli during exercise load escalation; that is, the chemoreceptors are not functioning. In other words, CO2 gas is stored in the alveoli and is not sufficiently expelled from the alveoli. Therefore, it is possible that the CO2 that is no longer expelled is absorbed by the alveoli and circulated back into the bloodstream, causing a rightward shift (Figure 12) [23]. First, the conventional V-slope method is based on the complete decomposition of glucose C6H12O6 + 6O2→6CO2 + 6H2O. It is difficult to say that the AT can be identified theoretically. Second, the detection algorithm used in the respiratory metabolic apparatus was based on the V-slope method. As long as the slope of the increase in VCO2 relative to VO2 is assumed to be greater than θ = 45°, it is challenging to detect AT, as shown in the experimental results. This result suggests that AT detection is insufficient in respiratory metabolic analysis based on the V-slope method, whereas that based on strucchange, which extracts structural change points, is significant.

*Sensors* **2022**, *22*, x FOR PEER REVIEW 10 of 14

**Figure 12.** Typical examples of a V-slope without ((**a**): left) and with ((**b**): right) Rtshift. The shift of V-slope is judged relative to the R = 1 diagonal line [23]. Figure 12 was created based on subject data. **Figure 12.** Typical examples of a V-slope without ((**a**): left) and with ((**b**): right) Rtshift. The shift of V-slope is judged relative to the R = 1 diagonal line [23]. Figure 12 was created based on subject data.

**Table 2.** AT by strucchange, V-slope methods, and respiratory metabolic analysis in VE (numbers in table converted to W). **Table 2.** AT by strucchange, V-slope methods, and respiratory metabolic analysis in VE (numbers in table converted to W).



**Table 3.** AT by strucchange, V-slope methods, and respiratory metabolic analysis in VCO<sup>2</sup> (numbers in table converted to W).

**Table 4.** AT by strucchange, V-slope methods, and respiratory metabolic analysis in RQ (numbers in table converted to W).


#### *4.2. Comparison of the V-Slope Method with Strucchange and Respiratory Metabolic Analysis*

Table 2 shows that 4 out of 17 subjects (A, B, F, J) had the same results with 95% confidence intervals for VE by the strucchange and V-slope methods. Similarly, 3 of 12 subjects (B, F, J) showed the same results for the AT by respiratory metabolic analysis (excluding those who could not be analyzed) and the V-slope method. Table 3 shows that 5 out of 17 subjects (A, B, F, J, Q) had the same results (95% confidence intervals) for VCO<sup>2</sup> by the strucchange and V-slope methods. Similarly, 3 out of 12 subjects (B, F, J) showed the same results in

respiratory metabolic analysis (excluding those who could not be analyzed) and the AT by the V-slope method. Table 4 shows that 1 out of 17 subjects (B) had the same results with 95% confidence intervals for RQ by the strucchange and V-slope methods. This result indicates a difference in the number of patients between the V-slope method and the statistical analysis methods (strucchange and respiratory metabolic analysis). Using the V-slope method, the intraclass correlation coefficient ICC (2, 2) between the two examiners was 0.4449. According to Landis et al.'s criteria, the confidence coefficient was moderate (0.41–0.60) [25]. With this result, the judgment using the V-slope method is acceptable. However, the ICC values are only point estimates, and the theoretical basis is unclear. However, Shrout et al. [22] recommended presenting a confidence interval (lower limit). The lower limit was 100% (1-α) (α: significance level). Here, if the significance level α = 0.1, the confidence interval (lower limit) is 90%, and the above ICC is not satisfied. The V-slope method is considered to have a limitation in AT identification because of individual errors and negligence errors caused by each examiner's subjectivity, compared with the analysis process described above in Section 4.1. To solve this problem in the V-slope method, it is necessary to improve the quality of judgment by increasing the number of examiners' tests, reviewing the judgment criteria, and increasing the number of examiners. As this study focuses on AT search, detailed estimation of sample size is not performed. The comparison of the V-slope method and the strucchange and respiratory metabolic apparatus showed that the V-slope method has low reproducibility in the statistical analysis of ICC values. It is necessary to consider systematic errors when making judgments, which suggests the superiority of statistical methods.

#### **5. Conclusions**

In this study, we performed a progressive loading test using a bicycle ergometer, measured and analyzed each ventilation parameter using a respiratory metabolic device, searched for the change point of the ventilation parameter by strucchange, and conducted an anaerobic threshold (AT) search. This study is summarized as follows.


However, the study also realized several methods and ways to improve the experimental procedure and the conclusion of the results. The first is to improve the quality of judgments by eliminating as many subjective factors as possible, such as the number of examiners in the V-slope method. The second is to make the AT detection system more versatile by considering past exercise habits and the presence or absence of personal medical history because of individual differences among subjects. Third, there is an error in analyzing RQ. It is necessary to fix the cause of the error and the analysis program. In the future, we would like to develop a medical-engineering collaboration system for diagnosis based on analysis using engineering methods and physiological ventilation indices.

**Author Contributions:** Conceptualization: T.A. and A.S.; methodology: H.N., T.A. and A.S.; software, K.H. and H.N.; formal analysis: H.N.; investigation: K.H. and T.A.; resources: M.K., T.T. and S.K.; data curation: K.H. and H.N.; writing—original draft preparation: T.A. and A.S.; writing review and editing: H.U., M.N., S.O. and A.S.; visualization: T.A. and A.S.; supervision: A.S.; project administration: T.A. and A.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted in conformance to the guidelines of the Declaration of Helsinki and approved by the Ethics Committee of Nagaoka University of Technology (Project identification code is H29-4, Date of approval is 12 May 2017).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Acknowledgments:** This study was conducted as a joint research project based on the comprehensive medical and industrial collaboration agreement between Niigata University Hospital and the Nagaoka University of Technology. We would like to express our gratitude to the concerned parties.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Development of a Remote Health Monitoring System to Prevent Frailty in Elderly Home-Care Patients with COPD**

**Chisato Ohashi <sup>1</sup> , Shunsuke Akiguchi 2,\* and Mineko Ohira <sup>3</sup>**


**Abstract:** Chronic obstructive pulmonary disease (COPD) is the general term used to describe respiratory diseases such as chronic bronchitis or emphysema of the lungs. COPD is known to cause the onset of frailty due to limitations of physical activity (PA) in daily life and undernutrition. Here, we report the development process of a remote health monitoring and support system employing a tablet computer (iPad), that was created to help prevent frailty in elderly home-care patients with COPD, and the results of its use by two elderly home-care COPD patients. There was a significant increase in PA duration in one participant after use of the system, compared to before use (15.2 min (8.9) vs. 24.2 min (7.4), *p* < 0.001). PA duration also increased in the other participant (39.7 (8.1) vs. 42.9 (12.9) min; 8.1%), although the difference was not statistically significant. The system enabled recognition of patients' behavior modifications to promote health. It is difficult to obtain quantitative data for health support, such as for respiratory rehabilitation in elderly COPD patients living at home. However, the present results suggest that virtually connecting patients with their support networks via information and communication technology (ICT) equipment provides support for the physical aspect of their care.

**Keywords:** COPD; frailty; home-care patients; health monitoring; physical activity

#### **1. Introduction**

Respiratory diseases are a common cause of physical limitations in daily life. They can lead to a vicious circle of decline in quality of life (QOL), which further promotes disease progression [1]. Chronic obstructive pulmonary disease (COPD) is the general term used to describe respiratory diseases, such as chronic bronchitis or emphysema of the lungs, resulting from a previous smoking habit [2]. COPD is known to cause a decline in QOL and the onset of frailty due to limitations of physical activity (PA) and undernutrition.

The Japanese Ministry of Health has stated the importance of countermeasures against COPD for extending healthy life expectancy. The government of Japan declared the "Healthy Japan 21" program in 2000 and set target values for various health indices. The second Healthy Japan 21 program, formulated in 2012, added COPD as a major lifestylerelated disease, along with cancer, cardiovascular diseases and diabetes. The government also set a target value for COPD awareness, raising it to >80% by 2022 [3]. However, the Global Initiative for Chronic Obstructive Lung Disease (GOLD) reported that COPD awareness in Japan remained at 28% as of December 2020 [4], which indicates that large numbers of Japanese people with subjective symptoms such as breathlessness, cough, or phlegm have not noticed that they have symptoms of COPD. Therefore, it is likely that initial diagnostic testing and subsequent treatment are delayed in Japanese people, which increases the risk of severe disease.

**Citation:** Ohashi, C.; Akiguchi, S.; Ohira, M. Development of a Remote Health Monitoring System to Prevent Frailty in Elderly Home-Care Patients with COPD. *Sensors* **2022**, *22*, 2670. https://doi.org/10.3390/s22072670

Academic Editors: Kimiyasu Kiyota, Akira Shionoya and Mitsuteru Inoue

Received: 18 February 2022 Accepted: 26 March 2022 Published: 30 March 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

The following targets have been identified in the management of COPD: (1) improvement of the patients' symptoms and QOL; (2) maintenance and improvement of exercise and PA capacity; (3) preventing acute exacerbations; (4) halting progression of COPD; (5) preserving and treating pulmonary complications; and (6) improving life prognosis. Breathing exercises, exercise therapy and nutritional care are the central non-pharmacotherapeutic interventions for patients with COPD. Additionally, home oxygen therapy (HOT) is used in cases with progressive hypoxemia due to chronic respiratory failure, to reduce symptoms, improve QOL and improve prognosis [5].

Once present, frailty leads to a weakening of mental and physical vitality, mobility and cognitive function, and is considered a comorbidity for other chronic diseases. Vital function deteriorates and fragility of the mind and body occur in the intermediate state between being healthy and requiring nursing care. However, it is possible to reverse frailty to some degree by adopting interventions and early treatment. Frailty has the following five standard symptoms: (1) weight loss; (2) susceptibility to fatigue; (3) reduction in walking speed; (4) grip strength weakness; and (5) reduction in physical activity. The presence of three or more of these symptoms confers a diagnosis of frailty, whereas the presence of one or two of these indicates pre-frailty [6].

A previous study reported very high prevalence rates of pre-frailty and frailty, of 56% and 20%, respectively, in elderly persons with COPD. One of the reasons for this situation is that dyspnea, a symptom of COPD, leads to a state of chronic undernutrition and a decline in PA. A combination of exercise and nutrition therapy has been reported as being effective in preventing frailty in patients with COPD [7]. Exercise therapy is known to reduce respiratory distress and depression, and to improve exercise tolerability, respiratory strength, activities of daily living (ADL) and QOL in patients with COPD [8]. The positive effects of home health care exercise as a part of respiratory rehabilitation have been verified in patients with COPD [9]. However, the rate of exercise therapy implementation among elderly patients is low [8]. One of the reasons for this is thought to be decreased motivation to participate in exercise therapy because of feelings of alienation from society or the community.

However, recent research has reported that public participation has a strong positive effect on preventing frailty in healthy elderly persons [10]. Another study found that inhome elderly persons who participate in nearby community activities maintain higher rates of mental and social health and have lower rates of decline in high-level vital functions [11]. Accordingly, we consider that the following three points are vitally important in health support that aims to prevent frailty in elderly patients with COPD: (1) exercise; (2) nutrition; and (3) public participation [10]. Studies on telemedicine for home-care patients with COPD are currently underway. Remote systems, such as monitoring of physiological parameters, including respiratory rate, blood pressure and blood oxygen saturation [12], or monitoring of the physical condition using interview items [13] etc., have already been developed. However, it is necessary to consider the issues of whether these devices are easily operable and inexpensive for patients, since most COPD patients are middle-aged and elderly. In addition, it is important to support the maintenance of exercise motivation, which is very effective in patients living at home with COPD. We developed a digital system to virtually support improvement and maintenance of the physical activity levels of elderly COPD patients. Our goals in the development of this health support system were to:


#### **2. Purpose**

Since the rate of frailty in elderly patients with COPD is very high [7], which in turn worsens the symptoms of COPD, it is very important to both prevent frailty and manage patients' symptoms and body condition. To prevent frailty and deterioration of symptoms among elderly patients with COPD on a long-term basis, the existing comprehensive health support system enables family doctors and visiting nurses to share a patient's data and to follow up with patients in a medical setting. In the case of an acute exacerbation of COPD, there can be a striking deterioration in symptoms, such as breathlessness, respiratory distress and cough. Therefore, it is desirable to monitor the condition of home-care patients with COPD on a daily basis, to enable early detection of any deterioration in symptoms and prevent an acute exacerbation.

For these reasons, we developed a remote health monitoring and support system that uses information and communication technology (ICT), with the aim of assessment and management of the physical condition and PA levels of home-care patients with COPD.

#### **3. Development of the Health Condition Monitoring and Support System**

We developed a remote system for monitoring the health condition and providing home health care support for elderly patients with COPD. The aim was to improve their PA habits and to prevent their condition from worsening, which can lead to other health problems, such as frailty and depression [11]. The system is outlined in Figure 1. We decided to utilize a tablet computer as the system device and, accordingly developed an application to handle input and transfer of the following data: six assessment items regarding extent of symptoms (cough, phlegm, breathing, sleep, appetite, vitality), daily step count, and energy expenditure. The six symptom assessment items were developed by a doctor and nurses involved in the treatment of home-care patients with COPD (Table 1). In the self-assessment, the application shows the patient's own data as a radar-chart or a time-series graph on the screen, and data for the six items are displayed on the screen along with the extent of symptoms. The results of an overall evaluation of the six items are presented as the 'Total Health Index', which is unique to this application, and was calculated using an original formula. The ratings for each of the six assessment items (from 0 to 5) were added and multiplied by 20, and then divided by the number of items, i.e., six, to obtain the value of the Total Health Index (Equation (1)). As the target users of the system were elderly patients with COPD, we designed the operation method to be as easy as possible, with a minimal amount of information displayed.

We also built a network server for the tablet application where the patients' data could be saved and accessed by support staff, such as doctors, nurses and nutritionists. In this study, we obtained the consent of the patients after explaining that only the three researchers in this study would operate the server and access the data stored in the server. Additionally, the researchers could share the patients' personal information between themselves and their doctors. Finally, we developed a tablet application that enabled doctors and other support staff to check and assess data sent remotely from each patient's tablet. These data included all items in the questionnaire regarding the extent of symptoms, as shown in Table 1. The patients were asked to complete the questionnaire every day, to provide a continuous record of their symptoms and thus enable rapid detection of any worsening of their health condition. Since the data could be shared between the doctor and patient, the doctor could select a suitable treatment method that matched the condition of the patient. The operating system used to develop the tablet application, which we named "Health Monitoring & Support Application", was iPad OS 14 Xcode 13.0, and the Swift 5.5 programming language was used for software development. The application has the following functions: (1) monitoring of health conditions, (2) enabling early response to acute exacerbations; and (3) providing health education. The complete system uses two applications (one for patients and another for support staff, such as doctors, nurses and nutritionists).

*Sensors* **2022**, *22*, x FOR PEER REVIEW 4 of 14

nurses and nutritionists).

**Figure 1.** Outline of the developed remote health monitoring and support system. **Figure 1.** Outline of the developed remote health monitoring and support system.



*x*; The level of the six assessment items (0 to 5)

*m*; Total health index

*x*; The level of the six assessment items (0 to 5)

tor and patient, the doctor could select a suitable treatment method that matched the condition of the patient. The operating system used to develop the tablet application, which we named "Health Monitoring & Support Application", was iPad OS 14 Xcode 13.0, and the Swift 5.5 programming language was used for software development. The application has the following functions: (1) monitoring of health conditions, (2) enabling early response to acute exacerbations; and (3) providing health education. The complete system uses two applications (one for patients and another for support staff, such as doctors,

*3.1. Input & Send Health Condition and Physical Activity Data 'APP for Patients' 3.1. Input & Send Health Condition and Physical Activity Data 'APP for Patients'*

3.1.1. Input & Send Data Functions 3.1.1. Input & Send Data Functions



4. Tap on the button to transmit all daily data to the data server (Figure 2).

**Figure 2.** Data input and send screen of the tablet (APP for patients). The data inputted include (1) body condition; (2) medication check; (3) number of steps/day and daily energy expenditure. When the patient taps the "send" button, all data for that day are transmitted to our server. **Figure 2.** Data input and send screen of the tablet (APP for patients). The data inputted include (1) body condition; (2) medication check; (3) number of steps/day and daily energy expenditure. When the patient taps the "send" button, all data for that day are transmitted to our server.

3.1.2. Data Visualization Function for Patients 1. Users tap on a date in the calendar to display goal attainment levels in the chart on 3.1.2. Data Visualization Function for Patients

	- 3. Numerical data for that date are displayed on a radar chart (Figure 3). 3. Numerical data for that date are displayed on a radar chart (Figure 3).

**Figure 3.** Patient application. Goal attainment levels are presented on a radar chart for the day selected (number of steps per day, duration of PA per day, numerical value of body condition). Patients can check their own data anywhere and anytime on the tablet device. **Figure 3.** Patient application. Goal attainment levels are presented on a radar chart for the day selected (number of steps per day, duration of PA per day, numerical value of body condition). Patients can check their own data anywhere and anytime on the tablet device.

*3.2. Remote Check and Assessment of Patient's Health Condition 'APP for Support Staff' Enter* 

2. Specify the period of data to be displayed. Data are displayed as a time-series graph

1. Select the numerical data item(s) for display.

in the center of the screen.

*the Patient's ID Number* 

(Figure 4).

*3.2. Remote Check and Assessment of Patient's Health Condition 'APP for Support Staff' Enter the Patient's ID Number*


**Figure 4.** Example screen of the support staff application. A time series graph is displayed on the tablet device. The user selects the period and data items for display. **Figure 4.** Example screen of the support staff application. A time series graph is displayed on the tablet device. The user selects the period and data items for display.

#### **4. Methods of Experimental Observation and Practical Assessment of the System 4. Methods of Experimental Observation and Practical Assessment of the System**

This study was conducted with the approval of the University of Toyama, National Institute of Technology, Toyama College, and National Hospital Organization Higashinagano Hospital. This study was conducted with the approval of the University of Toyama, National Institute of Technology, Toyama College, and National Hospital Organization Higashinagano Hospital.

The two participants in the study were a 74-year-old male with COPD (patient A) and a 65-year-old male with COPD (patient B). After an explanation of the study, the patients provided written consent to take part in the clinical evaluation for 8 months, from January 2021 to August 2021, in collaboration with two hospitals in Nagano Prefecture, Japan. Although patient A regularly visited the hospital for HOT, his condition was stable and did not interfere with his daily life. The condition of patient B was also stable when performing his daily life activities, and he did not require HOT. Tables 2 and 3 list patient data on the following physical characteristics: sex, age, height, weight, body mass index (BMI), results of the 6-min walking distance test, pulmonary function, and disease stage. The patients were required to use two devices in this study, a pedometer for daily The two participants in the study were a 74-year-old male with COPD (patient A) and a 65-year-old male with COPD (patient B). After an explanation of the study, the patients provided written consent to take part in the clinical evaluation for 8 months, from January 2021 to August 2021, in collaboration with two hospitals in Nagano Prefecture, Japan. Although patient A regularly visited the hospital for HOT, his condition was stable and did not interfere with his daily life. The condition of patient B was also stable when performing his daily life activities, and he did not require HOT. Tables 2 and 3 list patient data on the following physical characteristics: sex, age, height, weight, body mass index (BMI), results of the 6-min walking distance test, pulmonary function, and disease stage.

metabolic equivalents (METs) as a ratio of the metabolic rate, in which light-intensity ac-

monitoring of their step count and PA level, and an iPad with the monitoring system APP installed, for evaluation of their data and for sharing of their data with the support staff **Table 2.** Participant characteristics.


(PA levels 7–9). PA levels of 0 and micromovement indicate inactivity. According to the system of Kumahara et al., the energy cost of PA is expressed physiologically in terms of <sup>1</sup> An exercise tolerability index. <sup>2</sup> HOT, home oxygen therapy.

The patients were required to use two devices in this study, a pedometer for daily monitoring of their step count and PA level, and an iPad with the monitoring system APP installed, for evaluation of their data and for sharing of their data with the support staff and their medical teams. The LC pedometer (Suzuken, Nagoya, Japan) is a waist-fitted pedometer (Figure 5). The built-in accelerometer of the LC classifies the intensity of physical activity into 11 stages (0–9 and micromovement) using a unique algorithm based on the data acquired from the vertical vibration and its frequency that occurs with motion of the body. The data are recorded in the LC every 4 s. The 11 PA intensity levels according to the LC were classified as light (PA levels 1–3), moderate (PA levels 4–6), and vigorous (PA levels 7–9). PA levels of 0 and micromovement indicate inactivity. According to the system of Kumahara et al., the energy cost of PA is expressed physiologically in terms of metabolic equivalents (METs) as a ratio of the metabolic rate, in which light-intensity activity is defined as <3 METs, moderate-intensity activity as 3–6 METs, and vigorousintensity activity is >6 METs [14]. In the present study, PA was calculated as the total duration of PA per day at light, moderate, and vigorous intensity levels. Based on the age, sex, height and weight set in the LC, the number of steps and duration of PA by intensity (light, moderate, vigorous), walking distance and energy expenditure were calculated on a daily basis. LC has been used as the PA measurement sensor in many health-related studies. In addition, since patients are least resistant to wearing an LC compared to other sensors, the LC was adopted as the physical activity measurement sensor in this study [15]. The iPad used in the current study was an iPad Air (Apple, Cupertino, CA, USA) with cellular capability, to overcome any limitations of the Wi-Fi network environment at the patients' houses. To ensure that the tablet was user-friendly, the number of icons on the display was kept to a bare minimum. *Sensors* **2022**, *22*, x FOR PEER REVIEW 9 of 14

**Figure 5.** Images of the Lifecorder (LC). The image on the left shows how the LC is worn on the waist, and the image on the right shows the display of the LC. **Figure 5.** Images of the Lifecorder (LC). The image on the left shows how the LC is worn on the waist, and the image on the right shows the display of the LC.

**Table 2.** Participant characteristics. **Subject Sex Age (y) Height (cm) Weight (kg) BMI 6 Minute Walking Distance 1 (m) HOT 2** Patient A Male 74 172.9 55.8 18.7 217 Yes Patient B Male 65 167.5 41.3 14.7 336 No 1 An exercise tolerability index. 2 HOT, home oxygen therapy. **Table 3.** Pulmonary function and disease stage. Patients were asked to wear the LC every day to measure their amount of daily PA for 8 months, from January to August 2021. The study was divided into the following three periods: a 2 month pre-remote monitoring and support period (pre-support period) from January to February 2021; a remote monitoring and support period (support period) from March to April 2021; and a further 4 month implementation period (from May-August 2021). In the first 2 month pre-support period (January to February 2021), we only measured the patients' PA using the LC pedometer. During the second 2 month period, the support period (March to April 2021), in addition to continuing monitoring of their PA using the LC pedometer, the patients were asked to input their PA data recorded on the pedometer and their health data on the iPad APP, and to transmit the data to the support team. In the third

**(%) Stage 2 MRC Grade 3**

**%FEV1** 

1 FVC, forced vital capacity; FEV1, forced expiratory volume in one second; FEV1%, percent of forced expiratory volume in one second; %FEV1, percent predicted forced expiratory volume in one second. FVC is the total volume of air that can be exhaled with a maximum and slow exhalation. FEV is the volume of air that is exhaled with forceful expiration after maximum inhalation. Patients with COPD are unable to fully exhale, and their FEV is significantly lower than their FVC. The standard value of FEV is 3.00 to 4.00 L for men and 2.00 to 3.00 L for women. 2 According to the Global Initiative for Chronic Obstructive Lung Disease (GOLD), at Stage III, predicted FEV1 is 30– 50%; Stage IV, predicted FEV1 is < 30% [2]. 3 Six-point MRC dyspnea scale, Japanese Respiratory Society (2003) [16]. Grade 3 dyspnea: walks slower than people of the same age on level ground or stops to catch breath while walking at own pace on level ground; Grade 4 dyspnea: stops for breath

In patient A, the average number of steps per day and duration of PA per day were significantly higher during the support period than the pre-support period (1954 steps (591) vs. 1243 steps (680); 24.2 min (7.4) vs. 15.2 min (8.9), respectively; both *p* < 0.001) (Table 4, Figure 6). In patient B, although the average number of steps per day and total duration of PA per day improved after using the system, the increase was not significant

after walking approximately 100 yards or after a few minutes on level ground.

(3253 steps (640) vs. 3474 steps (1003) steps; 39.7 min (8.1) vs. 42.9 min (12.9)).

Patient B 2.67 1.19 44.57 39.0 III (Severe) 3

**FEV1% (%)** 

 **Pulmonary Function 1** 

**5. Results** 

**FEV1 (L)** 

**(L)** 

**Subject FVC** 

period, the implementation period, from May to August 2021, the patients continued using the LC pedometer and iPad APP, as in the support period, to evaluate their utilization rate of our system over the 4 month implementation period.


**Table 3.** Pulmonary function and disease stage.

<sup>1</sup> FVC, forced vital capacity; FEV1, forced expiratory volume in one second; FEV1%, percent of forced expiratory volume in one second; %FEV1, percent predicted forced expiratory volume in one second. FVC is the total volume of air that can be exhaled with a maximum and slow exhalation. FEV is the volume of air that is exhaled with forceful expiration after maximum inhalation. Patients with COPD are unable to fully exhale, and their FEV is significantly lower than their FVC. The standard value of FEV is 3.00 to 4.00 L for men and 2.00 to 3.00 L for women. <sup>2</sup> According to the Global Initiative for Chronic Obstructive Lung Disease (GOLD), at Stage III, predicted FEV1 is 30–50%; Stage IV, predicted FEV1 is <30% [2]. <sup>3</sup> Six-point MRC dyspnea scale, Japanese Respiratory Society (2003) [16]. Grade 3 dyspnea: walks slower than people of the same age on level ground or stops to catch breath while walking at own pace on level ground; Grade 4 dyspnea: stops for breath after walking approximately 100 yards or after a few minutes on level ground.

Based on the data transmitted to the support team, the number of steps per day and the duration of PA per day (total duration of light-, moderate- and vigorous-intensity activity) were estimated in the pre-support and support periods, and the data from the two periods were compared to evaluate changes in the PA behavior of each participant due to use of the system. The changes in patient PA behavior between the first and second periods were also used to evaluate the utility of our system.

Furthermore, we held face-to-face meetings with each participant at the two participating hospitals in Nagano Prefecture and downloaded their detailed LC data to estimate their PA at a monthly medical examination for 8 months. We also met with their family doctors, who were provided with monthly reports of their health status. Additionally, we conducted interviews with the patients once a month and provided lifestyle guidance based on the data stored in the system for the period from March to August 2021.

A parametric test was utilised to compare the amounts of PA, using SPSS software (SPSS 16.0 Family; IBM Co., Chicago, IL, USA). We first performed a Levene test and confirmed that the two patients were homoscedastic. Next, we conducted a *t*-test with the significance level set to less than 5%.

#### **5. Results**

In patient A, the average number of steps per day and duration of PA per day were significantly higher during the support period than the pre-support period (1954 steps (591) vs. 1243 steps (680); 24.2 min (7.4) vs. 15.2 min (8.9), respectively; both *p* < 0.001) (Table 4, Figure 6). In patient B, although the average number of steps per day and total duration of PA per day improved after using the system, the increase was not significant (3253 steps (640) vs. 3474 steps (1003) steps; 39.7 min (8.1) vs. 42.9 min (12.9)).


**Table 4.** Comparison of daily step count and duration of PA per day between the pre-support and support periods. presented below. **Table 4.** Comparison of daily step count and duration of PA per day between the pre-support and

The change in the amount of physical activity following use of the application, as well as the outcomes of guidance for living from meetings held at the two hospitals are

*Sensors* **2022**, *22*, x FOR PEER REVIEW 10 of 14

Data are presented as the average (SD), \*; *p* < 0.001, <sup>1</sup> Data for days on which the LC was worn for less than 8 h were excluded. Data are presented as the average (SD), \*; *p* < 0.001, 1 Data for days on which the LC was worn for less than 8 h were excluded.

**Figure 6.** Box-and-whisker plots of the daily step counts in the pre-support and support periods in the two patients. **Figure 6.** Box-and-whisker plots of the daily step counts in the pre-support and support periods in the two patients.

*5.1. Patient A*  Figure 7 shows changes in the levels of selected health indices per week in patient A The change in the amount of physical activity following use of the application, as well as the outcomes of guidance for living from meetings held at the two hospitals are presented below.

#### from May to August 2021. The graph shows that patient A performed less PA when symp-*5.1. Patient A*

season."

toms such as coughing and breathlessness were severe, and when he had insufficient sleep. PA improved along with improvements in his physical condition. In the hospital interviews with patient A, he said that he tried to limit his activity when he was not feeling well; therefore, we considered that he had appropriate self-management ability in relation to coexistence with COPD. In addition, collection of patient A's previous PA data that had been measured over the course of his treatment for two years showed that the number of steps tended to decrease in the coldest part of winter and in the hottest part of summer. This kind of seasonal data will enable doctors and other support staff to consider personalized support methods, in anticipation of worsening COPD symptoms at these times. Changes in the advice to patient A were as follows: Figure 7 shows changes in the levels of selected health indices per week in patient A from May to August 2021. The graph shows that patient A performed less PA when symptoms such as coughing and breathlessness were severe, and when he had insufficient sleep. PA improved along with improvements in his physical condition. In the hospital interviews with patient A, he said that he tried to limit his activity when he was not feeling well; therefore, we considered that he had appropriate self-management ability in relation to coexistence with COPD. In addition, collection of patient A's previous PA data that had been measured over the course of his treatment for two years showed that the number of steps tended to decrease in the coldest part of winter and in the hottest part of summer. This kind of seasonal data will enable doctors and other support staff to consider personalized support methods, in anticipation of worsening COPD symptoms at these times.

Pre-support period-Lifestyle guidance based on evaluation of only PA data

"Your daily step count has decreased, are you feeling sick?"

"Your PA duration is a little low, so do your best to walk and move in daily life" Support period- Lifestyle guidance using health monitoring system and PA data

"Do not overdo it as you may have coughing and other symptoms at the turn of the

**Figure 7.** Average steps per week and levels of selected health items per week in patient A. **Figure 7.** Average steps per week and levels of selected health items per week in patient A.

Before using the system, we asked patient A to increase his PA as part of exercise rehabilitation to prevent a decrease in exercise tolerability. The system enabled us to comprehensively monitor the patient's PA and physical condition data and provide individualized lifestyle guidance. In this way, it might be possible to prevent acute exacerbations

of COPD, and enable the early detection and treatment of acute exacerbations.

*5.2. Patient B*  Changes in the advice to patient A were as follows:


supporting people to get in and out of wheelchairs, his intensity of PA might increase. He is usually mindful to spend his time at home slowly, but when it comes to work, he some-"Do not overdo it as you may have coughing and other symptoms at the turn of the season."

times feels unwell due to moderate or vigorous intensity PA, as is evident in Figure 8. These data led us to conduct interviews with patient B in which we discussed how he could successfully balance work with management of his physical condition. Changes in the advice to patient B were as follows: Before using the system, we asked patient A to increase his PA as part of exercise rehabilitation to prevent a decrease in exercise tolerability. The system enabled us to comprehensively monitor the patient's PA and physical condition data and provide individualized lifestyle guidance. In this way, it might be possible to prevent acute exacerbations of COPD, and enable the early detection and treatment of acute exacerbations.

#### Pre-support period-Lifestyle guidance based on evaluation of only PA data *5.2. Patient B*

"You have been walking a lot, so it's going well." "Try and maintain this condition." Support period- Lifestyle guidance using health monitoring system and PA data "Have you been overdoing it at work lately?" "Please slow down and take it easy if you feel pain or have difficulty breathing." Before using the system, we had praised Patient B for increasing his PA but we had not noticed that the increased activity caused him knee pain. Using the system enabled us to advise him about developing the ability of self-management to control his PA level, in Figure 8 shows changes in levels of selected health indices per week in patient B. The graph shows that patient B's health index decreased on days of high PA levels, and that his health index recovered when his PA level was lower. Patient B performs regular work for a welfare bus transfer service. At that time, depending on the type of work, such as supporting people to get in and out of wheelchairs, his intensity of PA might increase. He is usually mindful to spend his time at home slowly, but when it comes to work, he sometimes feels unwell due to moderate or vigorous intensity PA, as is evident in Figure 8. These data led us to conduct interviews with patient B in which we discussed how he could successfully balance work with management of his physical condition.

order to avoid worsening of his condition.

**Figure 8.** Average steps per week and levels of selected health items per week in patient B. **Figure 8.** Average steps per week and levels of selected health items per week in patient B.

Changes in the advice to patient B were as follows:


We developed a remote health support system for home-care patients with COPD "Have you been overdoing it at work lately?"

that contributes to frailty prevention and tested its applicability and utility in two patients. "Please slow down and take it easy if you feel pain or have difficulty breathing."

In developing the application, we kept in mind that the users are elderly. Therefore, we made operation of the application as easy as possible, and deliberately presented only the essential information on the display. Previous researchers have developed a remote health monitoring system with multiple functions, but their system was hindered by the com-Before using the system, we had praised Patient B for increasing his PA but we had not noticed that the increased activity caused him knee pain. Using the system enabled us to advise him about developing the ability of self-management to control his PA level, in order to avoid worsening of his condition.

#### plexity of operation for elderly patients and their support teams [17]. In fact, many elderly **6. Discussion**

patients refused to participate in the present study due to concerns about the operability of the system, which highlights the extreme importance of simplifying the system operation. In the implementation experiment, the two participants utilized the application for 4 The three pillars of nutrition, exercise and public participation are important in frailty prevention. Since frailty is highly prevalent in COPD patients, its prevention is an important issue for maintaining their physical, psychological and social QOL.

months, from May to August 2021, at rates of 94.0% (110/117 days, patient A) and 94.9% (111/117 days, patient B). We consider that the convenience of the system enhanced its effectiveness in inducing a change in awareness in these two elderly patients with COPD, and that it could thus contribute to long-term frailty prevention. Other advantages of the system are that it allows remote management of the physical condition and PA of homecare patients, and a quick response to changes in medical conditions such as acute exacerbations. In addition, we attempted to incorporate patient education into the system, to foster patients' self-management by taking an interest in their own physical condition. Accordingly, we tried to realize "data visualization" for the patient by taking advantage of the features of the tablet computer (iPad), and aimed to improve the patients' own health management ability via this function. In the implementation experiment using the developed system, one of the two COPD patients had a statistically significant increase in PA after using the application. The other patient had no statistically significant difference in PA after using the application, although his step count increased by 6.8% and his duration of PA increased by 8.1%. These We developed a remote health support system for home-care patients with COPD that contributes to frailty prevention and tested its applicability and utility in two patients. In developing the application, we kept in mind that the users are elderly. Therefore, we made operation of the application as easy as possible, and deliberately presented only the essential information on the display. Previous researchers have developed a remote health monitoring system with multiple functions, but their system was hindered by the complexity of operation for elderly patients and their support teams [17]. In fact, many elderly patients refused to participate in the present study due to concerns about the operability of the system, which highlights the extreme importance of simplifying the system operation. In the implementation experiment, the two participants utilized the application for 4 months, from May to August 2021, at rates of 94.0% (110/117 days, patient A) and 94.9% (111/117 days, patient B). We consider that the convenience of the system enhanced its effectiveness in inducing a change in awareness in these two elderly patients with COPD, and that it could thus contribute to long-term frailty prevention. Other advantages of the system are that it allows remote management of the physical condition and PA of home-care patients, and a quick response to changes in medical conditions such

data suggest that the patients modified their behavior for health promotion. Since step count is related to QOL and frailty [18], this behavior modification is considered to conas acute exacerbations. In addition, we attempted to incorporate patient education into the system, to foster patients' self-management by taking an interest in their own physical condition. Accordingly, we tried to realize "data visualization" for the patient by taking advantage of the features of the tablet computer (iPad), and aimed to improve the patients' own health management ability via this function.

In the implementation experiment using the developed system, one of the two COPD patients had a statistically significant increase in PA after using the application. The other patient had no statistically significant difference in PA after using the application, although his step count increased by 6.8% and his duration of PA increased by 8.1%. These data suggest that the patients modified their behavior for health promotion. Since step count is related to QOL and frailty [18], this behavior modification is considered to contribute, not only to preventing deterioration of the patient's medical condition, but also to retention and improvement of QOL and prevention of frailty. Additionally, our developed system uses a pedometer to measure the amount of PA, since elderly patients are less resistant to using such devices. However, the system requires users to input their PA into the tablet manually, which could lead to input errors or forgetting to input the data.

A previous study reported that the rate of home exercise therapy in elderly patients with COPD is very low. Even understanding the need for exercise or PA is not always sufficient motivation for patients to increase their activity level [19]. Supervised highintensity exercise therapy during hospitalization has a certain training effect, and while patients exercising on their own after discharge could have benefits, it is also associated with certain risks. On the other hand, it has been reported that, after discharge, low-intensity exercise, such as walking, is easier for elderly patients with COPD to continue at home [8]. Our system does not force patients to perform exercise, but expects patients to develop a self-management ability by recording data about their physical condition and PA in the application. From this perspective, the study results suggest the possibility of a beneficial effect, even though the number of subjects was small. Moreover, remote monitoring of the patients' daily physical condition and PA enabled us to tailor lifestyle guidance to their own lifestyles and medical conditions. As the symptoms of COPD vary among individuals, the system shows future promise for providing personalized support in telemedicine. We believe that our developed remote health monitoring system will be useful in the medical and welfare fields.

In the implementation experiment, we approached remote health support from the following three perspectives: (1) prevention of physical frailty; (2) early response to acute exacerbations; and (3) health education for patients. It is difficult to obtain quantitative results for health support, including respiratory rehabilitation, for home-care elderly COPD patients. However, by connecting home-care elderly patients and their support team via ICT equipment, we found that it was possible to provide support in terms of physical as well as psychological and social aspects. In the future, we will continue to assist patients with COPD and their support teams by further developing the system to enable continuous support over a longer period of time.

#### **7. Conclusions**

An ICT-based support system, developed to provide comprehensive health support to elderly patients with COPD, was evaluated in a clinical setting. The amount of PA was significantly increased in one of the patients that used the system. Although the second patient showed no significant improvement in his PA levels, he did show a tendency for increased PA. Health and PA data submitted by the two patients with COPD enabled the provision of comprehensive and personalized health support for their individual lifestyles and medical conditions. In future development of the application, we will consider other devices that could make the system even more convenient to use. We will also add functions that support the aspect of social participation, which is important in frailty prevention. Full development of the system to encompass PA, nutrition and social participation will enable appropriate support to be provided to patients with COPD, and realization of its wider application in a variety of areas, including the medical and welfare fields.

**Author Contributions:** Conceptualization, C.O., M.O.; methodology, C.O.; software, S.A.; validation, C.O., M.O. and S.A.; formal analysis, C.O.; investigation, C.O.; data curation, C.O.; writing, C.O.; supervision, C.O.; project administration, C.O.; funding acquisition, C.O., S.A. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by JSPS KAKENHI, grant-in-Aid for scientific Research (C), grant number JP19K11316 (April 2019–March 2022).

**Institutional Review Board Statement:** The study was conducted according to the principles of the Declaration of Helsinki, and approved by the Institutional Review Board of the University of Toyama (protocol code: R2017148, date: 23 April 2018).

**Informed Consent Statement:** Informed consent was obtained from both subjects, who were judged by Mineko Ohira as being able to participate in the study.

**Acknowledgments:** We would like to extend our thanks to the staff of National Hospital Organization East Nagano Hospital and the staff of Iizuna Municipal Hospital for their support and cooperation in carrying out this research.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Deep Learning-Based Myoelectric Potential Estimation Method for Wheelchair Operation**

**Shimpei Aihara 1,2,\* , Ryusei Shibata 3,\*, Ryosuke Mizukami <sup>3</sup> , Takara Sakai <sup>3</sup> and Akira Shionoya <sup>3</sup>**


**Abstract:** Wheelchair sports are recognized as an international sport, and research and support are being promoted to increase the competitiveness of wheelchair sports. For example, an electromyogram can observe muscle activity. However, it is generally used under controlled conditions due to the complexity of preparing the measurement equipment and the movement restrictions imposed by cables and measurement equipment. It is difficult to perform measurements in actual competition environments. Therefore, in this study, we developed a method to estimate myoelectric potential that can be used in competitive environments and does not limit physical movement. We developed a deep learning model that outputs surface myoelectric potentials by inputting camera images of wheelchair movements and the measured values of inertial sensors installed on wheelchairs. For seven subjects, we estimated the myoelectric potential during chair work, which is important in wheelchair sports. As a result of creating an in-subject model and comparing the estimated myoelectric potential with the myoelectric potential measured by an electromyogram, we confirmed a correlation (correlation coefficient 0.5 or greater at a significance level of 0.1%). Since this method can estimate the myoelectric potential without limiting the movement of the body, it is considered that it can be applied to the performance evaluation of wheelchair sports.

**Keywords:** myoelectric potential; deep learning; camera; inertia sensor

#### **1. Introduction**

Sports promote mental and physical development, enrich humanity, and play an important role in living a healthy life. For persons with disabilities, sports can be an important component of medical rehabilitation. In addition, sports can provide lifelong recreation and can be played at various levels, e.g., the Paralympics. In Japan, especially in competitive sports, interest in sports for the disabled increased with the opening of the Olympic and Paralympic Tokyo 2020 Games. In particular, wheelchair sports account for 50% of the competitions held at the Paralympic Games, e.g., tennis, basketball, athletics marathons, short/medium/long-distance running and relays, badminton, rugby, and table tennis. It is recognized as an international sport. In recent years, the rules for wheelchair sports, which have been disparate in each country, have been established as international rules, and global competitiveness has advanced significantly. In wheelchair sports, a person and their wheelchair play together; thus, the chair work that controls the wheelchair similar to part of the body greatly affects the competitiveness of wheelchair athletes. The relationship between chair work and performance has been reported previously with a focus on wheelchair basketball, wheelchair tennis, and wheelchair rugby [1–3].

**Citation:** Aihara, S.; Shibata, R.; Mizukami, R.; Sakai, T.; Shionoya, A. Deep Learning-Based Myoelectric Potential Estimation Method for Wheelchair Operation. *Sensors* **2022**, *22*, 1615. https://doi.org/10.3390/ s22041615

Academic Editor: Mario Munoz-Organero

Received: 30 November 2021 Accepted: 15 February 2022 Published: 18 February 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

There is a kinetic approach that focuses on the changes and shapes of body parts [4]. In this approach, motion capture is a typical tool. Optical motion capture is widely considered the gold standard for motion capture because it is the most accurate method to track the kinematics of human movement [5]. For example, Franchin et al. reported that it was possible to identify the technology by measuring the trajectory of the wrist and elbow during a push motion using optical motion capture for wheelchair rugby [6]. However, optical motion capture can only be used in a narrow observation area, and the calibration procedure requires time and skill; thus, it is primarily used in laboratory settings. Therefore, using computer vision to analyze video images recorded at competition practice sites is increasing because there are few restrictions on the measurement location, the measurement process is easy, and the cost is low [7,8]. For the same reason, the use of wearable inertial sensors is also increasing in the field of competition [9,10].

There is also a kinematic approach that focuses on the forces acting on or inside the body. A kinematic approach that focuses on forces that cause movement is important to obtain knowledge that can be linked to actual training. This includes measuring the ground reaction force using a force plate and muscle contraction force using an electromyogram (EMG). Surface EMG data can noninvasively capture individual muscle activity during physical exercise [11]. Analysis and evaluation using EMG have been reported for the push movement of the hand rim, which is a basic element of wheelchair control. The push motion is performed by activating various muscles in a complex manner, and it is possible to evaluate the magnitude of the force exerted by observing the magnitude of the action potential of each muscle using surface myoelectricity. In addition, it is possible to evaluate the adjustment ability and smoothness of the push motion by looking at the target pattern [12–15].

However, EMGs are generally used under conditional control due to the limitation of body movements caused by attaching electrodes, devices, and cables to the body [16]. In addition, electrodes can shift position due to intense movements, which increases noise [17]. Thus, the development of a small wireless electromyogram [18] and an electrode [17] that does not come off easily using conductive gel has been reported. However, these EMGs are very expensive; thus, their accessibility is limited [19].

As a result, a method that can easily measure myoelectric potential at a low cost is expected. With the development of machine learning, markerless pose estimation using cameras has become possible [20,21]. This has eased the distance limitation from the camera to the subject, making it possible to estimate the posture of multiple people at relatively medium to long distances and to analyze movements during a game, where it is difficult to attach markers or sensors to the body. In the field of sports, markerless posture estimation using a camera has been applied to motion analysis and skill evaluation [22]. However, there are few previous studies that have used cameras for estimating muscle potential. The purpose of this study is to estimate muscle potential, which is kinematic information, using a video camera and an inertial sensor, both of which are increasingly used in the field of athletics as a kinematic approach. Once a training model is created, subsequent EMG potentials of the same person can be measured easily and inexpensively. Video cameras are unconstrained, and inertial sensors are small and lightweight; thus, such devices would not interfere with body movements. In addition, kinematics and kinetics information can be measured simultaneously; thus, this would be an extremely efficient measurement method. If we can establish such a data collection method, we can contribute to the development of new coaching and training concepts.

#### **2. Related Work**

In the sports field, many studies have implemented machine learning methods using inertial sensors and computer vision data as inputs [23]. For example, the use of support vector machines [24], logistic models [25], and hidden Markov models [26] has been reported. However, inertial sensor and computer vision data for sports movements are

high-dimensional and contain noise; thus, learning is ineffective in many cases with such machine learning methods [27]. high-dimensional and contain noise; thus, learning is ineffective in many cases with such machine learning methods [27]. high-dimensional and contain noise; thus, learning is ineffective in many cases with such machine learning methods [27]. Thus, deep learning methods have been reported in recent years [28,29]. Deep learn-

Thus, deep learning methods have been reported in recent years [28,29]. Deep learning is based on a neural network, which is a system that imitates the mechanism of human nerve cells [30]. Using a multilayer neural network, features contained in the data can be learned step by step; thus, it is possible to automatically extract features from the data acquired by inertial sensors [31]. This allows us to avoid some data preprocessing and feature extraction processes that are required by nondeep machine learning techniques. However, to the best of our knowledge, few previous studies have reported myoelectric potential estimation methods for wheelchair operation using video cameras and inertial sensors. Therefore, we have developed a method to estimate myoelectric potential by applying a deep learning method to inertial sensor and computer vision data. Figure 1 shows an outline of the proposed method. Thus, deep learning methods have been reported in recent years [28,29]. Deep learning is based on a neural network, which is a system that imitates the mechanism of human nerve cells [30]. Using a multilayer neural network, features contained in the data can be learned step by step; thus, it is possible to automatically extract features from the data acquired by inertial sensors [31]. This allows us to avoid some data preprocessing and feature extraction processes that are required by nondeep machine learning techniques. However, to the best of our knowledge, few previous studies have reported myoelectric potential estimation methods for wheelchair operation using video cameras and inertial sensors. Therefore, we have developed a method to estimate myoelectric potential by applying a deep learning method to inertial sensor and computer vision data. Figure 1 shows an outline of the proposed method. ing is based on a neural network, which is a system that imitates the mechanism of human nerve cells [30]. Using a multilayer neural network, features contained in the data can be learned step by step; thus, it is possible to automatically extract features from the data acquired by inertial sensors [31]. This allows us to avoid some data preprocessing and feature extraction processes that are required by nondeep machine learning techniques. However, to the best of our knowledge, few previous studies have reported myoelectric potential estimation methods for wheelchair operation using video cameras and inertial sensors. Therefore, we have developed a method to estimate myoelectric potential by applying a deep learning method to inertial sensor and computer vision data. Figure 1 shows an outline of the proposed method.

**Figure 1.** Outline of the proposed method. **Figure 1.** Outline of the proposed method. **Figure 1.** Outline of the proposed method.

#### **3. Proposed Method 3. Proposed Method 3. Proposed Method**

In this section, we first describe our data collection, data preprocessing, and dataset formation processes. Then, the proposed myoelectric potential estimation model is described. Finally, we describe model learning and present an evaluation. Figure 2 shows the procedure used to develop a myoelectric potential estimation model. In this section, we first describe our data collection, data preprocessing, and dataset formation processes. Then, the proposed myoelectric potential estimation model is described. Finally, we describe model learning and present an evaluation. Figure 2 shows the procedure used to develop a myoelectric potential estimation model. In this section, we first describe our data collection, data preprocessing, and dataset formation processes. Then, the proposed myoelectric potential estimation model is described. Finally, we describe model learning and present an evaluation. Figure 2 shows the procedure used to develop a myoelectric potential estimation model.

**Figure 2.** Steps to develop the myopotential estimation model. **Figure 2. Figure 2.** Steps to develop the myopotential estimation model. Steps to develop the myopotential estimation model.

#### *3.1. Dataset Creation 3.1. Dataset Creation 3.1. Dataset Creation*

#### 3.1.1. Data Collection 3.1.1. Data Collection 3.1.1. Data Collection

The data collection process involved synchronous measurement of myoelectric potential during wheelchair operation and capturing camera images and inertial sensor data. The subjects included seven healthy male subjects. (age: 22.9 ± 0.6 years; height: 171.7 ± 5.3 cm; weight: 64.2 ± 6.8 kg). Note that all subjects were right-handed. In addition, the subjects did not use a wheelchair on a daily basis but participated in the test after sufficient training. The measurement location was a gymnasium with a wooden floor. The data collection process involved synchronous measurement of myoelectric potential during wheelchair operation and capturing camera images and inertial sensor data. The subjects included seven healthy male subjects. (age: 22.9 ± 0.6 years; height: 171.7 ± 5.3 cm; weight: 64.2 ± 6.8 kg). Note that all subjects were right-handed. In addition, the subjects did not use a wheelchair on a daily basis but participated in the test after sufficient training. The measurement location was a gymnasium with a wooden floor. The data collection process involved synchronous measurement of myoelectric potential during wheelchair operation and capturing camera images and inertial sensor data. The subjects included seven healthy male subjects. (age: 22.9 ± 0.6 years; height: 171.7 ± 5.3 cm; weight: 64.2 ± 6.8 kg). Note that all subjects were right-handed. In addition, the subjects did not use a wheelchair on a daily basis but participated in the test after sufficient training. The measurement location was a gymnasium with a wooden floor.

*Sensors* **2022**, *22*, x FOR PEER REVIEW 4 of 15

The measurement instruments included cameras (Pocket Cinema Camera 4K by Blackmagic Design Pty. Ltd., Port Melbourne, Australia), inertial sensors (IMS-SD, Tec Gihan Co., Ltd., Kyoto, Japan), and electromyograms (Polymate Pro MP6000, Miyuki Giken Co., Ltd., Tokyo, Japan) Met. Here, three cameras were used, and the cameras were installed such that the entire measurement field (length 10 m, width 10 m) could be seen. The sampling frequency was 60 Hz, and the image resolution was 4K. The three cameras input external triggers and measured them synchronously. Figure 3 shows images captured simultaneously by the three cameras. Figure 4 shows the installation position of the inertial sensor. The inertial sensor was equipped with a three-axis accelerometer, a three-axis angular velocity sensor, and a three-axis geomagnetic sensor. The coordinate system of the sensor was a right-handed system. Sensor A was installed at the center of the axle. The front direction of the wheelchair was the X-axis positive direction, and the vertical upward direction was the Z-axis positive direction. Sensor B was installed at the center of the right wheel, and sensor C was installed at the center of the left wheel. The vertical upward direction relative to the surface of the wheel was defined as the Z-axis positive direction. It was recorded at a sampling frequency of 1000 Hz. The inertial sensor has a resolution of 12 bits and a dynamic range of ±16 G, and the angular velocity sensor has a resolution of 16 bits and a dynamic range of ±2000 degree per second (dps). The measurement instruments included cameras (Pocket Cinema Camera 4K by Blackmagic Design Pty. Ltd., Port Melbourne, Australia), inertial sensors (IMS-SD, Tec Gihan Co., Ltd., Kyoto, Japan), and electromyograms (Polymate Pro MP6000, Miyuki Giken Co., Ltd., Tokyo, Japan) Met. Here, three cameras were used, and the cameras were installed such that the entire measurement field (length 10 m, width 10 m) could be seen. The sampling frequency was 60 Hz, and the image resolution was 4K. The three cameras input external triggers and measured them synchronously. Figure 3 shows images captured simultaneously by the three cameras. Figure 4 shows the installation position of the inertial sensor. The inertial sensor was equipped with a three-axis accelerometer, a threeaxis angular velocity sensor, and a three-axis geomagnetic sensor. The coordinate system of the sensor was a right-handed system. Sensor A was installed at the center of the axle. The front direction of the wheelchair was the X-axis positive direction, and the vertical upward direction was the Z-axis positive direction. Sensor B was installed at the center of the right wheel, and sensor C was installed at the center of the left wheel. The vertical upward direction relative to the surface of the wheel was defined as the Z-axis positive direction. It was recorded at a sampling frequency of 1000 Hz. The inertial sensor has a resolution of 12 bits and a dynamic range of ±16 G, and the angular velocity sensor has a resolution of 16 bits and a dynamic range of ±2000 degree per second (dps). The measurement instruments included cameras (Pocket Cinema Camera 4K by Blackmagic Design Pty. Ltd., Port Melbourne, Australia), inertial sensors (IMS-SD, Tec Gihan Co., Ltd., Kyoto, Japan), and electromyograms (Polymate Pro MP6000, Miyuki Giken Co., Ltd., Tokyo, Japan) Met. Here, three cameras were used, and the cameras were installed such that the entire measurement field (length 10 m, width 10 m) could be seen. The sampling frequency was 60 Hz, and the image resolution was 4K. The three cameras input external triggers and measured them synchronously. Figure 3 shows images captured simultaneously by the three cameras. Figure 4 shows the installation position of the inertial sensor. The inertial sensor was equipped with a three-axis accelerometer, a threeaxis angular velocity sensor, and a three-axis geomagnetic sensor. The coordinate system of the sensor was a right-handed system. Sensor A was installed at the center of the axle. The front direction of the wheelchair was the X-axis positive direction, and the vertical upward direction was the Z-axis positive direction. Sensor B was installed at the center of the right wheel, and sensor C was installed at the center of the left wheel. The vertical upward direction relative to the surface of the wheel was defined as the Z-axis positive direction. It was recorded at a sampling frequency of 1000 Hz. The inertial sensor has a resolution of 12 bits and a dynamic range of ±16 G, and the angular velocity sensor has a resolution of 16 bits and a dynamic range of ±2000 degree per second (dps).

**Figure 3.** Example of measured camera images. **Figure 3.** Example of measured camera images. **Figure 3.** Example of measured camera images.

**Figure 4.** Example of measured camera images. **Figure 4. Figure 4.** Example of measured camera images. Example of measured camera images.

Figure 5 shows where the myoelectric potential is attached. The measurement points were the flexor digitorum profundus muscle, the biceps brachii muscle, the triceps brachii muscle, the posterior deltoid muscle, and the right pectoralis major muscle. Measurements were performed using active electrodes with a center distance of 2 cm between the electrodes. Measurements were recorded at a sampling frequency of 1000 Hz. Based on interviews with experts familiar with wheelchair sports, the target movements considered in this study included three-intensity straight running, zigzag running, 90-degree turns, 180-degree turns, and 360-degree turns, covering important chair work movements in wheelchair sports. Here, each subject performed a wheelchair movement for approximately five to eight minutes by mixing these movements. The wheelchair used in this study was a tennis wheelchair (BWZ, OX Engineering Co., Ltd., Chiba, Japan). Figure 5 shows where the myoelectric potential is attached. The measurement points were the flexor digitorum profundus muscle, the biceps brachii muscle, the triceps brachii muscle, the posterior deltoid muscle, and the right pectoralis major muscle. Measurements were performed using active electrodes with a center distance of 2 cm between the electrodes. Measurements were recorded at a sampling frequency of 1000 Hz. Based on interviews with experts familiar with wheelchair sports, the target movements considered in this study included three-intensity straight running, zigzag running, 90-degree turns, 180-degree turns, and 360-degree turns, covering important chair work movements in wheelchair sports. Here, each subject performed a wheelchair movement for approximately five to eight minutes by mixing these movements. The wheelchair used in this study was a tennis wheelchair (BWZ, OX Engineering Co., Ltd., Chiba, Japan).

**Figure 5.** Attachment position for myopotential. **Figure 5.** Attachment position for myopotential.

#### 3.1.2. Data Preprocessing 3.1.2. Data Preprocessing

Here, we describe data preprocessing and dataset construction. Figure 6 shows the procedure used to create a dataset. Figure 7 shows example data used in data preprocessing and dataset construction. First, the data of each measurement instrument were time-synchronized by the external trigger signal input at the time of data collection. The images acquired by the cameras were processed as follows. First, the two-dimensional coordinates of the body joints of the person in the image were calculated from the camera image using Open Pose [32]. Of the three cameras, the camera image in which the subject was most prominent in the same time frame was used. In this study, we calculated the key point coordinates of the upper body, which are expected to affect the wheelchair rowing motion. Figure 8 shows the joint sites considered in this study. Next, the 3D coordi-Here, we describe data preprocessing and dataset construction. Figure 6 shows the procedure used to create a dataset. Figure 7 shows example data used in data preprocessing and dataset construction. First, the data of each measurement instrument were timesynchronized by the external trigger signal input at the time of data collection. The images acquired by the cameras were processed as follows. First, the two-dimensional coordinates of the body joints of the person in the image were calculated from the camera image using Open Pose [32]. Of the three cameras, the camera image in which the subject was most prominent in the same time frame was used. In this study, we calculated the key point coordinates of the upper body, which are expected to affect the wheelchair rowing motion. Figure 8 shows the joint sites considered in this study. Next, the 3D coordinates were calculated from the calculated 2D coordinates of the body joints using a pose-baseline [33].

nates were calculated from the calculated 2D coordinates of the body joints using a posebaseline [33]. Finally, the calculated 3D coordinates of the body joints were normalized by converting them to a local coordinate system with the key point of the pelvis as the origin. Here, the midpoint between the right and left hips was the origin, and the vertical upward direction to the ground was the Z-axis positive direction. The plane that passes through the origin and is horizontal to the ground was defined as the XY plane. The line connecting the right and left hips was projected onto the XY plane as the Y-axis, and the direction from the right hip to the left hip was the positive direction. Note that this was a righthanded coordinate system. In addition, this process was repeated for each frame. The data acquired by the inertial sensor were processed as follows. The three-axis acceleration data of inertial sensor A were used to express the moving acceleration of the wheelchair, and the three-axis angular velocity data of inertial sensors B and C were used to express the Finally, the calculated 3D coordinates of the body joints were normalized by converting them to a local coordinate system with the key point of the pelvis as the origin. Here, the midpoint between the right and left hips was the origin, and the vertical upward direction to the ground was the Z-axis positive direction. The plane that passes through the origin and is horizontal to the ground was defined as the XY plane. The line connecting the right and left hips was projected onto the XY plane as the Y-axis, and the direction from the right hip to the left hip was the positive direction. Note that this was a right-handed coordinate system. In addition, this process was repeated for each frame. The data acquired by the inertial sensor were processed as follows. The three-axis acceleration data of inertial sensor A were used to express the moving acceleration of the wheelchair, and the three-axis angular velocity data of inertial sensors B and C were used to express the rotation of the wheel. A high-pass filter with a cutoff frequency of 60 Hz was applied to the acceleration and angular velocity data to reduce high-frequency component noise. Then, to unify the

*Sensors* **2022**, *22*, x FOR PEER REVIEW 6 of 15

down-sampled it to 60 Hz.

3.1.3. Dataset Construction

it to 60 Hz.

sampling frequency of the data of each measurement instrument, we down-sampled it to 60 Hz. In this study, the window width w and slide width were set to six. The constructed dataset included approximately 3800 data points for each subject.

rotation of the wheel. A high-pass filter with a cutoff frequency of 60 Hz was applied to the acceleration and angular velocity data to reduce high-frequency component noise. Then, to unify the sampling frequency of the data of each measurement instrument, we

The data recorded by the electromyogram were processed as follows. As a full-wave rectification smoothing process, the absolute value of the amplitude was obtained, and a low-pass filter with a cutoff frequency of 2.6 Hz was applied. To identify an appropriate cutoff frequency, we referred to the report by Yoshida and Terao [34]. Next, the percent maximum voluntary contraction (% MVC) was derived by normalizing the maximum myoelectric potential of the myoelectric potential data for each subject. Finally, to unify the sampling frequency of the data of each measurement instrument, we down-sampled

Here, we describe the dataset construction process. The input data that served as explanatory variables were 3D coordinate data for 16 body joints (48 dimensions in total), 3D acceleration data for wheelchairs, and the three-axis angular velocity data for the left and right wheels (six dimensions in total), for a total of 57 dimensions. The output value (i.e., the objective variable) was each myoelectric potential data point. Here, the data structure of time point t was the data of the input value obtained by cutting out the data of window width w points before time point t and the data of the output value obtained by cutting out the data of time point t. Next, time t was slid by slide width p, and the input and output data were cut out again. A dataset was constructed by repeating this process.

**Figure 6.** Procedure for preprocessing and formation of datasets. **Figure 6.** Procedure for preprocessing and formation of datasets.

**Figure 7.** Sample data of the preprocessing and formation of datasets. **Figure 7.** Sample data of the preprocessing and formation of datasets.

**Figure 8.** Body joints calculated from the camera image.

A neural network model designed for myoelectric potential estimation is described in the following. Long Short-Term Memory (LSTM) [35] is generally used to extract features from time-series data. However, when attempting to recognize time-series data derived from muscle activity represented by an electrocardiogram, the one-dimensional

*3.2. Neural Network Model Design*

**Figure 7.** Sample data of the preprocessing and formation of datasets.

**Figure 8.** Body joints calculated from the camera image. **Figure 8.** Body joints calculated from the camera image.

*3.2. Neural Network Model Design* A neural network model designed for myoelectric potential estimation is described in the following. Long Short-Term Memory (LSTM) [35] is generally used to extract features from time-series data. However, when attempting to recognize time-series data derived from muscle activity represented by an electrocardiogram, the one-dimensional The data recorded by the electromyogram were processed as follows. As a full-wave rectification smoothing process, the absolute value of the amplitude was obtained, and a low-pass filter with a cutoff frequency of 2.6 Hz was applied. To identify an appropriate cutoff frequency, we referred to the report by Yoshida and Terao [34]. Next, the percent maximum voluntary contraction (% MVC) was derived by normalizing the maximum myoelectric potential of the myoelectric potential data for each subject. Finally, to unify the sampling frequency of the data of each measurement instrument, we down-sampled it to 60 Hz.

#### 3.1.3. Dataset Construction

Here, we describe the dataset construction process. The input data that served as explanatory variables were 3D coordinate data for 16 body joints (48 dimensions in total), 3D acceleration data for wheelchairs, and the three-axis angular velocity data for the left and right wheels (six dimensions in total), for a total of 57 dimensions. The output value (i.e., the objective variable) was each myoelectric potential data point. Here, the data structure of time point t was the data of the input value obtained by cutting out the data of window width w points before time point t and the data of the output value obtained by cutting out the data of time point t. Next, time t was slid by slide width p, and the input and output data were cut out again. A dataset was constructed by repeating this process. In this study, the window width w and slide width were set to six. The constructed dataset included approximately 3800 data points for each subject.

#### *3.2. Neural Network Model Design*

A neural network model designed for myoelectric potential estimation is described in the following. Long Short-Term Memory (LSTM) [35] is generally used to extract features from time-series data. However, when attempting to recognize time-series data derived from muscle activity represented by an electrocardiogram, the one-dimensional convolutional neural network (1D-CNN) [36], which exhibits high performance in terms of feature extraction and learning short-time data, is better. Valid cases have been reported previously [37,38]. Therefore, a 1D-CNN was adopted in this study.

Figure 9 shows the architecture of the proposed model. The input to this model was a 6 × 57 size matrix. Next, it was connected to the convolutional layer for feature extraction. Here, the Relu function was adopted as the activation function. Next, we connected it to

the max. pooling layer, which takes the maximum value in the kernel. Then, to avoid overfitting, we included a dropout layer [39] with a dropout rate of 0.5. This process was repeated twice. Note that the kernel sizes were set to 1 × 2 and 1 × 3, and the number of channels was changed to 128 and 256. As a result, the data of each dimension were convolved in the time direction, and feature quantity extraction was in progress. The data of each dimension extracted by the 1D-CNN were reshaped and then connected to the fully-connected layer. Here, the Relu function was employed as the activation function. Note that the fully-connected layer was repeated three times. Finally, the one-dimensional value was output, i.e., the myoelectric potential of a single target site. the max. pooling layer, which takes the maximum value in the kernel. Then, to avoid overfitting, we included a dropout layer [39] with a dropout rate of 0.5. This process was repeated twice. Note that the kernel sizes were set to 1 × 2 and 1 × 3, and the number of channels was changed to 128 and 256. As a result, the data of each dimension were convolved in the time direction, and feature quantity extraction was in progress. The data of each dimension extracted by the 1D-CNN were reshaped and then connected to the fullyconnected layer. Here, the Relu function was employed as the activation function. Note that the fully-connected layer was repeated three times. Finally, the one-dimensional value was output, i.e., the myoelectric potential of a single target site.

convolutional neural network (1D-CNN) [36], which exhibits high performance in terms of feature extraction and learning short-time data, is better. Valid cases have been reported

Figure 9 shows the architecture of the proposed model. The input to this model was a 6 × 57 size matrix. Next, it was connected to the convolutional layer for feature extraction. Here, the Relu function was adopted as the activation function. Next, we connected it to

*Sensors* **2022**, *22*, x FOR PEER REVIEW 8 of 15

previously [37,38]. Therefore, a 1D-CNN was adopted in this study.

**Figure 9.** The architecture of the designed model (not to scale). **Figure 9.** The architecture of the designed model (not to scale).

#### *3.3. Model Learning*

*3.3. Model Learning* The deep learning model described in Section 3.2 was implemented using Python and TensorFlow. Note that this was an in-subject model. The data for each subject were randomly divided into learning and evaluation data at a ratio of 8:2. In addition, 20% of the learning data were used as the validation data to avoid overfitting. The number of epochs was set to a maximum of 30,000, and the model with the parameters that had the smallest MSE (Mean Squared Error) in the validation data was adopted. Adam [40] was used as the optimal function, and the initial learning rate was set to 0.01. The batch size was set to 32, referring to a previous report [41]. The model was learned for each myoe-The deep learning model described in Section 3.2 was implemented using Python and TensorFlow. Note that this was an in-subject model. The data for each subject were randomly divided into learning and evaluation data at a ratio of 8:2. In addition, 20% of the learning data were used as the validation data to avoid overfitting. The number of epochs was set to a maximum of 30,000, and the model with the parameters that had the smallest MSE (Mean Squared Error) in the validation data was adopted. Adam [40] was used as the optimal function, and the initial learning rate was set to 0.01. The batch size was set to 32, referring to a previous report [41]. The model was learned for each myoelectric potential site of each subject. For learning, we used an NVIDIA Tesla V100 on Google Colaboratory.

lectric potential site of each subject. For learning, we used an NVIDIA Tesla V100 on

#### Google Colaboratory. **4. Results**

**4. Results** Each trained model was applied to the evaluation data to evaluate the accuracy of myoelectric potential estimation. Here, the Spearman correlation coefficient between the measured EMG value and the value estimated by the proposed was obtained for each myoelectric site for each subject, and the significance of the correlation coefficient was tested. Table 1 shows the correlation coefficient between the measured and estimated values for each muscle site for each subject. As can be seen, all correlation coefficients were significant at the 0.1% level. The results of the estimation model for each subject and each myoelectric potential site exhibited a correlation coefficient of 0.5 or greater (*p* < 0.001). Thus, we confirmed that the proposed model can estimate a value with a positive correlation with the measured EMG value. By comparing the average values of the correlation coefficients for each subject, we found that the minimum and maximum values were 0.52 Each trained model was applied to the evaluation data to evaluate the accuracy of myoelectric potential estimation. Here, the Spearman correlation coefficient between the measured EMG value and the value estimated by the proposed was obtained for each myoelectric site for each subject, and the significance of the correlation coefficient was tested. Table 1 shows the correlation coefficient between the measured and estimated values for each muscle site for each subject. As can be seen, all correlation coefficients were significant at the 0.1% level. The results of the estimation model for each subject and each myoelectric potential site exhibited a correlation coefficient of 0.5 or greater (*p* < 0.001). Thus, we confirmed that the proposed model can estimate a value with a positive correlation with the measured EMG value. By comparing the average values of the correlation coefficients for each subject, we found that the minimum and maximum values were 0.52 and 0.75, respectively. Although there were some differences depending on the subjects, all subjects had a correlation coefficient of 0.5 or greater. Similarly, by comparing the average values of the correlation coefficients for each myoelectric site, we found that the minimum and maximum values were 0.51 and 0.75, respectively. Here, although some differences were observed depending on the target myoelectric site, the correlation coefficient was 0.5 or greater in all cases.


**Table 1.** The correlation coefficient between measured EMG values and proposed model estimated values. All correlation coefficients in the table are significant at a 0.1% level.

The average and standard deviation of the absolute errors of the estimates obtained by the proposed model when the measured EMG values were taken as true values were calculated. Table 2 shows the average and standard deviation of the absolute error for each muscle site for each subject. As can be seen, the mean absolute error was 0.96 at the minimum and 7.78 at the maximum. Here, the range of data that can be taken by the true value was 0% to 100% MVC; thus, the maximum prediction error was 7.78%.

**Table 2.** Mean and standard deviation of the absolute error between measured EMG values and proposed model estimated values. The unit is %MVC.


Figure 10 shows the comparison of the correlation coefficients between the proposed method and other time-series data modeling algorithms. The models to be compared are conventional machine learning methods such as Linear Regression (LR) [42], Decision Tree Regression (DTR) [43], Support Vector Regression (SVR) [44], and Multilayer Perceptron (MLP) [45]. As a deep learning model, we used the widely used Convolutional Neural Network (CNN) [46]. In addition, we used the Gated Recurrent Unit (GRU) [47], which is widely used for extracting features from time series data, and the LSTM [35]. The proposed model showed the highest correlation among all models for all measurement sites. We showed the usefulness of the proposed model for the problem of estimating muscle potential from camera images and inertial sensors. *Sensors* **2022**, *22*, x FOR PEER REVIEW 10 of 15

> greater (*p* < 0.001), and the proposed model had a positive correlation with the measured EMG value. These results confirm that the value could be estimated. The evaluation data included data acquired during various chair work movements that occur in wheelchair sports; thus, we believe that the proposed model exhibits good generalizability for various

> In addition, we confirmed that the average value of the correlation coefficient value for all subjects was 0.5 or greater. In addition, the proposed model was able to estimate a value having a positive correlation with the measured EMG value. Thus, the proposed method can be applied to various subjects. Similarly, the average correlation coefficient value for each myoelectric site achieved a correlation coefficient of 0.5 or greater for all myoelectric sites, and the proposed model could estimate a value that had a positive correlation with the measured EMG value. Thus, we have confirmed that the proposed

> In a previous study [48], the myoelectric potential during a dynamic loading task was estimated by solving the inverse dynamics and optimization problems using optical motion capture data and electromagnetic tracker data as inputs. It was reported that the deltoid muscle demonstrated an accuracy of 0.53 with the measured EMG, and the biceps brachii showed an accuracy of 0.61 with the measured EMG. The target motion in the current study is different from that of the previous study; thus, it is not possible to make a general comparison. However, the proposed model obtained the same degree of accu-

> The maximum prediction error of the proposed method was 7.78%. Similar to this study, in a previous study where kinematic data were estimated from kinematic data using deep learning, the results of estimating the skeletal muscle forces of the rectus femoris, soleus, and tibialis anterior muscles for walking movements indicated that the prediction error was less than 10% [49]. Note that we cannot make a general comparison with previous studies because the input kinematic data were acquired using a motion capture system, and the motions and myoelectric sites of the subjects differ; however, a simple measuring device was used in our study. Thus, we consider that the same degree of accuracy

> Since the myoelectric potential can be estimated without limiting the movement of the body by a simple method using only the camera image and the inertial sensor data, it is considered that it can be applied to the performance evaluation of wheelchair sports. To estimate muscle potential, muscle activity, muscle strength, and joint torque accurately, the three-dimensional posture data of a person acquired using a motion capture system and mechanical data generated during exercise are required [49]. To acquire mechanical data while operating a wheelchair, a wheelchair [50] equipped with a torque

**Figure 10.** Comparison of correlation coefficients for each model. **Figure 10.** Comparison of correlation coefficients for each model.

method can handle various myoelectric sites.

racy as the previous study using a simple measuring instrument.

**5. Discussion**

was obtained.

chair work movements.

#### **5. Discussion**

As shown in Table 1, we found that the results of the proposed estimate model for each subject and each myoelectric potential site had a correlation coefficient of 0.5 or greater (*p* < 0.001), and the proposed model had a positive correlation with the measured EMG value. These results confirm that the value could be estimated. The evaluation data included data acquired during various chair work movements that occur in wheelchair sports; thus, we believe that the proposed model exhibits good generalizability for various chair work movements.

In addition, we confirmed that the average value of the correlation coefficient value for all subjects was 0.5 or greater. In addition, the proposed model was able to estimate a value having a positive correlation with the measured EMG value. Thus, the proposed method can be applied to various subjects. Similarly, the average correlation coefficient value for each myoelectric site achieved a correlation coefficient of 0.5 or greater for all myoelectric sites, and the proposed model could estimate a value that had a positive correlation with the measured EMG value. Thus, we have confirmed that the proposed method can handle various myoelectric sites.

In a previous study [48], the myoelectric potential during a dynamic loading task was estimated by solving the inverse dynamics and optimization problems using optical motion capture data and electromagnetic tracker data as inputs. It was reported that the deltoid muscle demonstrated an accuracy of 0.53 with the measured EMG, and the biceps brachii showed an accuracy of 0.61 with the measured EMG. The target motion in the current study is different from that of the previous study; thus, it is not possible to make a general comparison. However, the proposed model obtained the same degree of accuracy as the previous study using a simple measuring instrument.

The maximum prediction error of the proposed method was 7.78%. Similar to this study, in a previous study where kinematic data were estimated from kinematic data using deep learning, the results of estimating the skeletal muscle forces of the rectus femoris, soleus, and tibialis anterior muscles for walking movements indicated that the prediction error was less than 10% [49]. Note that we cannot make a general comparison with previous studies because the input kinematic data were acquired using a motion capture system, and the motions and myoelectric sites of the subjects differ; however, a simple measuring device was used in our study. Thus, we consider that the same degree of accuracy was obtained.

Since the myoelectric potential can be estimated without limiting the movement of the body by a simple method using only the camera image and the inertial sensor data, it is considered that it can be applied to the performance evaluation of wheelchair sports.

To estimate muscle potential, muscle activity, muscle strength, and joint torque accurately, the three-dimensional posture data of a person acquired using a motion capture system and mechanical data generated during exercise are required [49]. To acquire mechanical data while operating a wheelchair, a wheelchair [50] equipped with a torque meter that can measure the torque generated on the wheels via push operations is required. However, such wheelchairs are custom products and are not widely used; thus, so it is extremely difficult to acquire such wheelchairs. Therefore, in this study, we thought that the wheelchair would move as a result of the movement of wheelchair operation, and we used the wheelchair acceleration and angular velocity data obtained from the inertial sensor as pseudo mechanical data. As a result, the input values can be used as the three-dimensional posture data of the person during wheelchair operation and the mechanical data generated by movement. Thus, we believe that the proposed method contributes to myoelectric potential estimation. The proposed model is a unique approach that effectively exploits the specific characteristics of wheelchair sports.

In the field of wheelchair sports, running tests (e.g., straight run, turn, and zigzag run) on a defined course have been frequently conducted for the purpose of agility tests and wheelchair work skill evaluation, but the evaluation has been limited to running time only. Due to the limitation of high expertise and the high price of the measurement equipment, kinematic and muscle activity evaluations have not been conducted. The proposed model

made it possible to estimate muscle potential using only simple devices, such as cameras and inertial sensors. Therefore, in the running test, it became possible to evaluate skills not only in terms of running time, which is the result of movements, but also the process of movements, especially the process of force generations. By evaluating the process of movements, it is possible to provide specific coaching. In addition, the proposed model makes it possible to assess the muscle fatigue caused by wheelchair operations.

The limitations of the proposed method are described as follows. As mentioned previously, the proposed method was designed based on the premise that the acceleration and angular velocity data of the wheelchair obtained from the inertial sensor are used as pseudo mechanical data. Therefore, if the movement is unrelated to the wheelchair operation, e.g., the swing movement of a tennis racket or the shooting movement of a basketball, the acceleration and angular velocity data of the wheelchair obtained by the inertial sensor do not change; thus, the myoelectric potential will be incorrect. In addition, as basic research, we created a model of basic wheelchair movements with the advice of experts.

However, it is difficult to apply the proposed model to actual competition scenes, because competition-specific movements are added to the basic wheelchair manipulation movements. For example, if a wheelchair is operated while holding a tennis or badminton racket, the EMGs may change. To solve this problem, we can collect additional EMGs during competition-specific motions and transfer learning the proposed model to cope with competition-specific motions. In order to solve this problem, we have been developing a model for estimating EMGs corresponding to competition-specific motions with the cooperation of Paralympic athletes playing wheelchair tennis and wheelchair badminton. In this study, we created a model to estimate the EMGs for each individual. Therefore, it is necessary to collect data using EMG for each person and use data as training to create the EMG estimation model. This point may be a barrier to the widespread use of this method in sports fields. The characteristics of EMGs, which are biometric information, differ slightly from person to person. Therefore, we will collect EMG data from a large number of subjects during wheelchair operations to expand the dataset. We will train the proposed model using the dataset and develop an EMG estimation model considering individual differences.

Figure 11 shows a plot of the electromyographic measurements and the estimated values obtained by the proposed model for the pectoralis major muscle of subject 1. Here, the horizontal axis plots the results of each sample such that the EMG measurement values are shown in ascending order. The vertical axis is the % MVC of the myoelectric potential. As shown in Figure 11, when the % MVC value is large, i.e., during a high-intensity push operation, the difference between the plots of the measured EMG values and the estimated values is large; thus, the estimation accuracy is poor. This tendency was also confirmed in other subjects and other myoelectric sites. It is considered that this is because the ratio of high-strength push motion was smaller than that of low-strength push motion in the data collection test conducted in this study. In other words, the feature quantity extraction effective for estimating the high-intensity push motion did not proceed in the learning process of the model because the data of the high-intensity push motion were scarce. To increase the proportion of high-intensity push motion data, data expansion by signal processing and additional data acquisition processes are required, which we plan to investigate in the future.

In the proposed model, a simple method was used to calculate a subject's threedimensional posture information from a camera image [32,33]. Compared to optical motion capture, the position accuracy of the three-dimensional posture is low. Thus, we expect to improve the accuracy of myoelectric potential estimation by improving the method used to calculate the 3D posture information from the camera images, which will improve position accuracy.

in the data collection test conducted in this study. In other words, the feature quantity extraction effective for estimating the high-intensity push motion did not proceed in the learning process of the model because the data of the high-intensity push motion were scarce. To increase the proportion of high-intensity push motion data, data expansion by signal processing and additional data acquisition processes are required, which we plan

In the proposed model, a simple method was used to calculate a subject's three-dimensional posture information from a camera image [32,33]. Compared to optical motion capture, the position accuracy of the three-dimensional posture is low. Thus, we expect to improve the accuracy of myoelectric potential estimation by improving the method used to calculate the 3D posture information from the camera images, which will improve po-

This study was the first attempt to estimate EMGs during wheelchair operation using the camera and inertial sensors. As a proof of concept, we showed that we applied deep learning to this estimation task and achieved a model with a moderate correlation to EMG measurements. In particular, we showed the usefulness of a deep learning model with 1D-CNN and fully connected layers. We believe that this realized effective feature extraction. However, the performance of a 1D-CNN may be inferior to that of LSTM [35] when long-term time information is required by estimating myoelectric potential because the time width that can be considered is fixed by the window size. Therefore, in the future, we will consider designing a neural network model that fuses the two methods, e.g., by applying the 1D-CNN for data close to the input and applying LSTM in the higher-order region.

Recall that the proposed myoelectric potential estimation model is an in-subject model. In other words, the model ignores individual differences. By training the proposed model on a dataset with an increased number of subjects and additional training data, we believe that an intersubject model that considers individual differences can be realized in

to investigate in the future.

sition accuracy.

the future.

**Figure 11.** Plots of measured EMG and estimated value of the pectoralis in subject 1. **Figure 11.** Plots of measured EMG and estimated value of the pectoralis in subject 1.

**6. Conclusions** In order to realize a simple method for measuring EMGs as a proof of concept study, we proposed a model for estimating EMGs during wheelchair operations using deep learning from camera images and inertial sensor data. We designed a deep learning model using a 1D-CNN and trained the model using a collected dataset. We found that the proposed model can estimate myoelectric potential that correlates with the measured EMG values (showing a correlation coefficient of 0.5 or greater). In addition, we achieved the same level of accuracy as a previous myoelectric potential estimate method that acquires values by a motion capture system using inverse dynamics and optimization problems. This study was the first attempt to estimate EMGs during wheelchair operation using the camera and inertial sensors. As a proof of concept, we showed that we applied deep learning to this estimation task and achieved a model with a moderate correlation to EMG measurements. In particular, we showed the usefulness of a deep learning model with 1D-CNN and fully connected layers. We believe that this realized effective feature extraction. However, the performance of a 1D-CNN may be inferior to that of LSTM [35] when longterm time information is required by estimating myoelectric potential because the time width that can be considered is fixed by the window size. Therefore, in the future, we will consider designing a neural network model that fuses the two methods, e.g., by applying the 1D-CNN for data close to the input and applying LSTM in the higher-order region.

We have demonstrated a new approach to estimating myoelectric potential during Recall that the proposed myoelectric potential estimation model is an in-subject model. In other words, the model ignores individual differences. By training the proposed model on a dataset with an increased number of subjects and additional training data, we believe that an intersubject model that considers individual differences can be realized in the future.

#### **6. Conclusions**

In order to realize a simple method for measuring EMGs as a proof of concept study, we proposed a model for estimating EMGs during wheelchair operations using deep learning from camera images and inertial sensor data. We designed a deep learning model using a 1D-CNN and trained the model using a collected dataset. We found that the proposed model can estimate myoelectric potential that correlates with the measured EMG values (showing a correlation coefficient of 0.5 or greater). In addition, we achieved the same level of accuracy as a previous myoelectric potential estimate method that acquires values by a motion capture system using inverse dynamics and optimization problems. We have demonstrated a new approach to estimating myoelectric potential during wheelchair operation using only kinematic data and deep learning. We have also suggested the possibility of estimating EMG using only camera image and inertial sensor data without using the EMG. Since this method can estimate the myoelectric potential without limiting the movement of the body, it is considered that it can be applied to the performance evaluation of wheelchair sports.

In the future, we plan to improve the accuracy of myoelectric potential estimation and to develop an estimation model that considers individual differences.

**Author Contributions:** Conceptualization, S.A. and A.S.; methodology, S.A. and R.S.; software, S.A., R.S. and T.S.; validation, R.S., R.M. and T.S.; formal analysis, S.A. and R.S.; investigation, S.A. and R.S.; resources, S.A.; data curation, R.S., R.M. and T.S.; writing—original draft preparation, S.A.; writing—review and editing, S.A. and R.S.; visualization, S.A.; supervision, S.A. and A.S.; project administration, S.A. and A.S.; funding acquisition, S.A. and A.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** All subjects gave their informed consent for inclusion before they participated in the study. The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the Ethics Committee of Nagaoka University of Technology (protocol code: 2017-38).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** All available data can be obtained by contacting the corresponding author.

**Acknowledgments:** We thank the participants for taking part in the study.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


**Ryu Nakabayashi <sup>1</sup> and Masanori Eguchi 2,\***


**Abstract:** Many dielectrophoretic (DEP) devices for biomedical application have been suggested, such as the separation, concentration, and detection of biological cells or molecules. Most of these devices utilize the difference in their DEP properties. However, single-cell analysis is required to evaluate individual properties. Therefore, this paper proposed a modified isomotive insulator-based DEP (iDEP) creek-gap device for straightforward single-cell analysis, which is capable of measurement at a wide frequency band. The proposed iDEP device generates more constant particle velocity than the previous study. The insulator was fabricated using backside exposure for accurate forming. We measured the distribution of the particle velocity and frequency property, using homogeneous polystyrene particles to verify the effectiveness of the proposed device. The results show that the particle velocity distribution was consistent with the distribution of the numerically calculated electric field square (∇*E* 2 rms). Furthermore, the velocity measurement, at a wide frequency band, from 10 Hz to 20 MHz, was performed because of the long distance between electrodes. These results suggest that the prop-erties of various particles or cells can be obtained by simple measurement using the proposed device.

**Citation:** Nakabayashi, R.; Eguchi, M. Evaluation of Isomotive Insulator-Based Dielectrophoretic Device by Measuring the Particle Velocity. *Sensors* **2022**, *22*, 1533. https://doi.org/10.3390/s22041533

Academic Editor: Shiyang Tang

Received: 19 January 2022 Accepted: 14 February 2022 Published: 16 February 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

**Keywords:** single-cell analysis; isomotive dielectrophoresis; insulator-based dielectrophoresis; particle tracking velocimetry; particle characterization

#### **1. Introduction**

Dielectrophoresis (DEP) is an electrokinetic phenomenon, which is the movement of a polarizable microparticle suspended in a medium under a non-uniform alternating current (AC) electric field [1]. The movement of particles toward a high electric field is called positive DEP (pDEP), whereas their movement away from this field is called negative DEP (nDEP). The DEP force acting on the particles depends on the radius of the particles, complex permittivity of the particle and medium, and gradient of electric field square (∇*E* 2 rms).

The properties of the particles driven by DEP, such as biological cells, are characterized by the DEP collection rate [2], equilibrium point between DEP force and gravity [3,4], cross-over frequency [5], and terminal velocity of particles [6]. Generally, particles have unique DEP properties with frequency dependence, due to their structural and electrical properties. Therefore, these properties can be evaluated through the DEP properties. Moreover, there are various biomedical applications of DEP utilizing the difference in the DEP properties of particles, such as the separation of human cancer cells from normal cells [7] and live and dead cells [8], the concentration of human cancer cells [9], and detection of deoxyribonucleic acid labeled to microbeads [10]. Therefore, the measurement of the DEP properties of particles or cells is one of the subjects for DEP application.

Cellular heterogeneity in isogenic cell populations has been widely observed, e.g., in stem cells [11]. Thus, it is crucial for cell analysis to measure at a single-cell level, representing individual property, instead of averaged properties, through bulk measurement [12]. In the electrical perspective, the frequency properties of cells contain much information, such as structural and electrical properties [13]. The literature stated that the electrical properties of cell membranes might be prognostic markers for tumor detection and treatments because they are remarkably changed by malignant transformation, affecting their growth [14].

Recently, several methods have been developed to characterize particles or cells using DEP [15,16], traveling wave DEP (twDEP) [15], and electrorotation (ROT) [17–19]. These devices are electrode-based (eDEP) systems. Therefore, the measurement at low frequency is limited, due to the electric double layer (EDL) at the interface between electrode and medium. Moreover, cells may be damaged by a high electric field [20], such as near the electrode. However, the 3DEP platform is a rapid and high-accuracy measurement method for the DEP properties of cells [21]. Further, it is unsuitable for single-cell analysis, since it achieves high-accuracy measurements by averaging the results derived from multiple samples.

More recently, isomotive DEP (isoDEP) devices were developed as the straightforward single-cell analysis method. isoDEP devices have two modes: eDEP and insulator-based DEP (iDEP). On iDEP devices, an electric field is generated by electrodes, and a non-uniform electric field is formed by insulators. Therefore, iDEP devices can easily be prevented from the damage of cells [22] because the test region of iDEP device is away from the electrodes. The pDEP-exhibited particles adhere to the edge of the electrode in eDEP device, due to the concentration of the electric field to electrodes. Meanwhile, it is less of a problem in iDEP [23]. In isoDEP, particles move on the constant ∇*E* 2 rms, leading to a constant particle velocity. This concept was first suggested by Pohl [24]. It was later demonstrated with polar and rectangular coordinate systems by Allen et al. [25] and Tada et al. [23,26], respectively. The particles moving on isomotive field are characterized by their velocity, and their electrical properties are enabled to evaluate by the velocity. The device with a polar coordinate system has a chamber with a 120◦ bend angle, and the particles move toward radius. In contrast, the device with a rectangular coordinate system has a creek-gap-shaped insulator, and the particles move on the centerline between the insulators.

In this study, we proposed a modified isomotive iDEP creek-gap device. Figure 1 shows the concept of the proposed device. The electrodes generate an electric field, and a non-uniform electric field is formed by changing the gap between creek-gap insulators at the center of the chamber. This concept of the proposed device is the same as in our previous study [23]. However, the design guideline of the device was modified to realize a more constant ∇*E* 2 rms. Furthermore, the validity of numerically calculated ∇*E* 2 rms was confirmed through the consistency of the distribution between the measured particle velocity and ∇*E* 2 rms. After that, the particle velocity measurement was performed at a wide frequency band from 10 Hz to 20 MHz. Velocity decay was observed at a very low frequency (≤100 Hz). Additionally, the cause was explained using the equivalent circuit of the proposed device, including the EDL between the electrode and medium.

**Figure 1.** Illustrations of the proposed isomotive iDEP creek-gap device. (**a**) The isometric view of the entire device. (**b**) The enlarged top view of the chamber. A non-uniform electric field was formed by changing the gap between creek-gap insulators. **Figure 1.** Illustrations of the proposed isomotive iDEP creek-gap device. (**a**) The isometric view of the entire device. (**b**) The enlarged top view of the chamber. A non-uniform electric field was formed by changing the gap between creek-gap insulators.

#### **2. Theory and Device Designing**

follows:

square; Re[*CM*\*

*\** and *ε*<sup>m</sup>

where *ε*<sup>p</sup>

frequency.

**2. Theory and Device Designing**  *2.1. Particle Velocity Induced by the Dielectrophoretic Force*

*2.1. Particle Velocity Induced by the Dielectrophoretic Force*  The time-averaged dielectrophoretic force, 〈*F*DEP〉, is calculated according to [27], as The time-averaged dielectrophoretic force, h*F*DEPi, is calculated according to [27], as follows:

$$
\langle F\_{\rm DEP} \rangle = 2\pi \varepsilon\_0 \varepsilon\_m R^3 \text{Re}[\![\![\!M^\*]\!]\!] \nabla E\_{\rm rms}^2 \tag{1}
$$

*\**

〈*F*DEP〉 = 2π*ε*0*ε*m*R*<sup>3</sup> Re[*CM*\* ]∇*E*rms <sup>2</sup> (1) where *ε*0 and *ε*m are the free-space permittivity and relative permittivity of suspending where *ε*<sup>0</sup> and *ε*<sup>m</sup> are the free-space permittivity and relative permittivity of suspending medium, respectively; *R* is the particle radius; ∇*E* 2 rms is the gradient of the electric field square; Re[*CM*∗ is the real part of Clausius–Mossotti factor, *CM*∗ , which is expressed by:

$$\text{CM}^\* = \frac{\varepsilon\_p^\* - \varepsilon\_m^\*}{\varepsilon\_p^\* + 2\varepsilon\_m^\*} \tag{2}$$

*CM*\* <sup>=</sup>*ε*<sup>p</sup> *\** െ *ε*<sup>m</sup> *ε*p *\* +* 2*ε*<sup>m</sup> *\** (2) where *ε* ∗ *<sup>p</sup>* and *ε* ∗ *<sup>m</sup>* are the complex permittivity of the particle and suspending medium, respectively. The complex permittivity, *ε* ∗ , is given by:

$$
\varepsilon^\* = \varepsilon - j\frac{\sigma}{\omega} \tag{3}
$$

*ε\* = ε* െ *j σ <sup>ω</sup>* (3) where *ε* and *σ* are the permittivity and conductivity, respectively; *j* <sup>2</sup> <sup>=</sup> <sup>−</sup>1; *<sup>ω</sup>* is the angular frequency.

where *ε* and *σ* The diameter of the semi-circle is proportional to; *j*2 = −1; *ω* is the angular The drag force, *F<sup>d</sup>* , acting on spherical particles is given by Stokes' law, as follows:

The drag force, *F*d, acting on spherical particles is given by Stokes' law, as follows:

$$F\_d = 6\pi\eta Rv \tag{4}$$

<sup>d</sup>*t* = 〈*F*DEP〉 െ *<sup>F</sup>*d(*v*(*t*)) െ *<sup>F</sup>*<sup>f</sup> (6)

<sup>3</sup> (*ρ*<sup>p</sup> *- ρ*m)*µg*ቋ ൜<sup>1</sup> െ exp ൬- *<sup>t</sup>*

*τ*

൰ൠ (7)

*F*d = 6π*ηRv* (4) where *η* is the viscosity of the medium, and *v* is the relative velocity of the particle. Additionally, the friction force, *F<sup>f</sup>* , acting on the particle surface is given by:

$$F\_f = \frac{4}{3}\pi \text{R}^3 (\rho\_p - \rho\_m) \mu \text{g} \tag{5}$$

*Ff* = <sup>4</sup> 3 π*R*<sup>3</sup> (*ρ*<sup>p</sup> െ *<sup>ρ</sup>*m)*µg* (5) where *ρ<sup>p</sup>* and *ρ<sup>m</sup>* are the mass density of the particle and suspending medium, respectively; *µ* is the coefficient of the dynamic friction, and *g* is the gravitational acceleration.

where *ρ*<sup>p</sup> and *ρ*m are the mass density of the particle and suspending medium, respec-Suppose *F*DEP and *F<sup>f</sup>* are constant. The governing equation for the particle moving on a plane is expressed as a function of time, *t*, as follows:

*<sup>R</sup>*<sup>3</sup> <sup>d</sup>*v*(*t*)

൧ ∂*E*rms 2 <sup>∂</sup>*<sup>x</sup>* െ <sup>2</sup>

tively; *µ* is the coefficient of the dynamic friction, and *g* is the gravitational acceleration.

When the initial velocity of the particle is zero and *y*-component of DEP force is bal-

$$\frac{4}{3}\pi\rho\_p\mathcal{R}^3\frac{dv(t)}{dt} = \left<\mathcal{F}\_{\rm DEP}\right> - \left.F\_d(v(t))\right> - \left.F\_f\right>\tag{6}$$

<sup>3</sup>*<sup>η</sup>* <sup>ቊ</sup>*ε*0*ε*mReൣ*CM*\*

4 3 π*ρ*<sup>p</sup>

When the initial velocity of the particle is zero and *y*-component of DEP force is balanced on the *x*-axis, due to its symmetry, the particle velocity is described as follows:

$$v(t) = \frac{\mathcal{R}^2}{3\eta} \left\{ \varepsilon\_0 \varepsilon\_m \text{Re} [\text{CM}^\*] \frac{\partial E\_{\text{rms}}^2}{\partial \mathbf{x}} - \frac{2}{3} (\rho\_{\text{p}} - \rho\_m) \mu \mathbf{g} \right\} \left\{ 1 - \exp \left( -\frac{t}{\tau} \right) \right\} \tag{7}$$

where *τ* is the time constant in the particle motion on the device, which is given by:

$$
\pi = \frac{2\rho\_p \mathcal{R}^2}{9\eta} \tag{8}
$$

If the frictional force is negligible because it is smaller than the x-component of the DEP force and *t τ*, then *v* is given by:

$$v(t) = \frac{\varepsilon\_0 \varepsilon\_{\rm m} R^2 \text{Re}[\rm CM^\*]}{3\eta} \frac{\partial E\_{\rm rms}^2}{\partial x} \tag{9}$$

Under the above assumption, the particle velocity is proportional to *∂E* 2 rms/*∂x* (i.e., on the constant *∂E* 2 rms/*∂x*, the particle behavior is the isomotive movement).

#### *2.2. Design of the Proposal Isomotive iDEP Device*

#### 2.2.1. Analysis of the Device Having Parallel Insulators

eDEP and iDEP devices form an isomotive field by changing the gap between the electrodes and insulators, respectively. There are clear theories for the electrode design of eDEP devices, such as *E* = *V*/*d* (where *V* is the applied voltage and *d* is the distance between electrodes at a parallel-plate electrode), but not for the insulator of iDEP devices. The device with parallel insulators (Figure 2) was numerically calculated to analyze the properties of the generated electric field between insulators using electromagnetic field simulation software (Ansys Maxwell, ANSYS, USA). It has the same construction as the device shown in Figure 1; however, only the shape of insulators was changed from creek-gap to parallel insulator. The device's materials are the substrate (glass with 0.5 mm-thickness), electrodes (aluminum), insulator (SU-8 with 70 µm-thickness), and cover (silicone rubber with 100 µm-thickness). More detailed material information is presented in Section 3. For the device's parameters, the gap of the insulators is 40 µm; the distance between the same potential electrode is 540 µm; the length of each side of the electrode uncovered with the insulator is 500 µm; the insulator length is defined as the variable. The voltage applied to the electrodes was setup as shown in Figure 2. Herein, the calculated fields were read at the height of 10 µm from the substrate because the radius of the used particle was 10 µm.

The ideal property of the generated electric field on the centerline between insulators is "uniform" because the intensity of the electric field line does not change, according to the concept of the iDEP device. However, three patterns of the properties (Figure 3) were observed and called "unimodal", which has one point of maximal electric field (Figure 3a), i.e., "uniform", which partly has a uniform electric field (Figure 3b) and "bimodal", which has two points of maximal electric field (Figure 3c) by changing *l*<sup>i</sup> . The uniform distribution was observed when 90 µm ≤ *l*<sup>i</sup> ≤ 210 µm. If *l*<sup>i</sup> is shorter than 90 µm, the property is unimodal, due to the lack of insulator length for gathering the electric field line. Meanwhile, if *l*<sup>i</sup> is longer than 210 µm, the property is bimodal, due to the concentration of the electric field around the sidewall of the insulator, at near *x* = *l*i/2. Furthermore, the properties of the generated electric field did not almost change when the device was scaled up. Thus, the properties were seemingly determined by the ratio of the device's parameter, e.g., the gap of insulators and insulator length.

**Figure 2.** Top view of the device with parallel insulators for analyzing the generated electric field properties. **Figure 2.** Top view of the device with parallel insulators for analyzing the generated electric field properties. **Figure 2.** Top view of the device with parallel insulators for analyzing the generated electric field properties.

**Figure 3.** Generated electric field properties between the centerline and parallel insulators of the device. (**a**) The "unimodal" electric field at *l*i = 40 µm. The distribution has one maximal point of the electric field. (**b**) The "uniform" electric field at *l*i = 150 µm. The distribution partly has a uniform electric field. (**c**) The "bimodal" electric field at *l*i = 400 µm. The distribution has two maximal points of the electric field. **Figure 3.** Generated electric field properties between the centerline and parallel insulators of the device. (**a**) The "unimodal" electric field at *l*i = 40 µm. The distribution has one maximal point of the electric field. (**b**) The "uniform" electric field at *l*i = 150 µm. The distribution partly has a uniform electric field. (**c**) The "bimodal" electric field at *l*i = 400 µm. The distribution has two maximal points of the electric field. **Figure 3.** Generated electric field properties between the centerline and parallel insulators of the device. (**a**) The "unimodal" electric field at *l*<sup>i</sup> = 40 µm. The distribution has one maximal point of the electric field. (**b**) The "uniform" electric field at *l*<sup>i</sup> = 150 µm. The distribution partly has a uniform electric field. (**c**) The "bimodal" electric field at *l*<sup>i</sup> = 400 µm. The distribution has two maximal points of the electric field.

#### 2.2.2. Design of Isomotive iDEP Creek-Gap Device 2.2.2. Design of Isomotive iDEP Creek-Gap Device 2.2.2. Design of Isomotive iDEP Creek-Gap Device

given as [26]:

given as [26]:

Following the above analyses, the "uniform" electric field is generated on the device, shown in Figure 2, when 90 µm ≤ *l*<sup>i</sup> ≤ 210 µm. Furthermore, the properties were determined by the ratio of the device's parameter. Therefore, the isomotive iDEP creek-gap device was re-designed by matching the ratio of each parameter to the device, with a parallel insulator using the gap between insulators as the reference. First, the shape of creekgap insulators was determined. The range of 90 µm ≤ *li* ≤ 210 µm is also described as 2.25 ≤ *l*i/*g* ≤ 5.25, using a gap of insulators, *g* = 40 µm. If the insulator length is 250 µm, 2.25 ≤ *l*i/*g* ≤ 5.25 is denoted as 48 µm ≤ *g* ≤ 111 µm (i.e., 2.25 = *l*i/*g*, then *g* = 250 µm/2.25 ≈ 48 µm, due to the insulator length of 250 µm). The shape of creek-gap insulators fol-Following the above analyses, the "uniform" electric field is generated on the device, shown in Figure 2, when 90 µm ≤ *l*<sup>i</sup> ≤ 210 µm. Furthermore, the properties were determined by the ratio of the device's parameter. Therefore, the isomotive iDEP creek-gap device was re-designed by matching the ratio of each parameter to the device, with a parallel insulator using the gap between insulators as the reference. First, the shape of creekgap insulators was determined. The range of 90 µm ≤ *li* ≤ 210 µm is also described as 2.25 ≤ *l*i/*g* ≤ 5.25, using a gap of insulators, *g* = 40 µm. If the insulator length is 250 µm, 2.25 ≤ *l*i/*g* ≤ 5.25 is denoted as 48 µm ≤ *g* ≤ 111 µm (i.e., 2.25 = *l*i/*g*, then *g* = 250 µm/2.25 ≈ 48 µm, due to the insulator length of 250 µm). The shape of creek-gap insulators fol-Following the above analyses, the "uniform" electric field is generated on the device, shown in Figure 2, when 90 µm ≤ *l*<sup>i</sup> ≤ 210 µm. Furthermore, the properties were determined by the ratio of the device's parameter. Therefore, the isomotive iDEP creekgap device was re-designed by matching the ratio of each parameter to the device, with a parallel insulator using the gap between insulators as the reference. First, the shape of creek-gap insulators was determined. The range of 90 µm ≤ *l*<sup>i</sup> ≤ 210 µm is also described as 2.25 ≤ *l*i/*g* ≤ 5.25, using a gap of insulators, *g* = 40 µm. If the insulator length is 250 µm, 2.25 ≤ *l*i/*g* ≤ 5.25 is denoted as 48 µm ≤ *g* ≤ 111 µm (i.e., 2.25 = *l*i/*g*, then *g* = 250 µm/2.25 ≈ 48 µm, due to the insulator length of 250 µm). The shape of creek-gap

lowed the equation of the design guideline of isomotive eDEP creek-gap device and was

lowed the equation of the design guideline of isomotive eDEP creek-gap device and was

insulators followed the equation of the design guideline of isomotive eDEP creek-gap device and was given as [26]:

*Sensors* **2022**, *22*, x FOR PEER REVIEW 6 of 14

$$y = \frac{V\_{\rm rms}}{2\sqrt{a\chi + b}}\tag{10}$$

where *V*rms is the root mean square of the applied voltage; *a* is *∂E* 2 rms/*∂x*, and *b* is the constant determined by the boundary conditions in the isomotive eDEP device. Equation (10) is also used for the isomotive iDEP creek-gap device. where *V*rms is the root mean square of the applied voltage; *a* is ∂*E*rms <sup>2</sup> /∂*x*, and *b* is the constant determined by the boundary conditions in the isomotive eDEP device. Equation (10) is also used for the isomotive iDEP creek-gap device.

Next, we determined the distance between the same potential electrode, *d*e, and length of each side of the electrode uncovered with the insulator, *l*e. According to the device with parallel insulators, and since *d*<sup>e</sup> = 540 µm and *l*<sup>e</sup> = 500 µm, we obtained *d*e/*g* = 13.5 and *l*e/*g* = 12.5. Thus, the shape of the electrode is determined as *d*e≈ 650 µm and *l*<sup>e</sup> ≈ 600 µm on the part of the narrower *g*, as well as *d*<sup>e</sup> ≈ 1500 µm and *l*<sup>e</sup> ≈ 1390 µm on the part of the wider *g*. Next, we determined the distance between the same potential electrode, *d*e, and length of each side of the electrode uncovered with the insulator, *l*e. According to the device with parallel insulators, and since *d*e = 540 µm and *l*e = 500 µm, we obtained *d*e/*g* = 13.5 and *l*e/*g* = 12.5. Thus, the shape of the electrode is determined as *d*<sup>e</sup> ≈ 650 µm and *l*<sup>e</sup> ≈ 600 µm on the part of the narrower *g*, as well as *d*<sup>e</sup> ≈ 1500 µm and *l*<sup>e</sup> ≈ 1390 µm on the part of the wider *g*.

Figure 4a,b show the design guideline of electrodes and creek-gap insulators, respectively. The electrodes and insulators were designed as mentioned above. The creek-gap insulators were calculated using Equation (10). Figure 4c shows the distribution of numerically calculated *E*rms and *∂E* 2 rms/*∂x* when the electrodes apply a voltage of 1 *V*rms. Furthermore, *∂E* 2 rms/*∂x* was approximately constant within ±2.5%, in the range of *x* = 110 µm to 195 µm. In contrast, it was approximately constant within ±25% in the previous study [23]. In the experiment, the silicone rubber cover was changed to dimethylpolysiloxane, due to the promotion of the adhesion between the insulator and cover. Figure 4a,b show the design guideline of electrodes and creek-gap insulators, respectively. The electrodes and insulators were designed as mentioned above. The creek-gap insulators were calculated using Equation (10). Figure 4c shows the distribution of numerically calculated *E*rms and ∂*E*rms <sup>2</sup> /∂*x* when the electrodes apply a voltage of 1 *V*rms. Furthermore, ∂*E*rms <sup>2</sup> /∂*x* was approximately constant within ±2.5%, in the range of *x* = 110 µm to 195 µm. In contrast, it was approximately constant within ±25% in the previous study [23]. In the experiment, the silicone rubber cover was changed to dimethylpolysiloxane, due to the promotion of the adhesion between the insulator and cover.

**Figure 4.** (**a**) Top view of the designed electrode shape of isomotive iDEP creek-gap device. (**b**) Top view of the designed shape of the creek-gap insulators. (**c**) Distribution of the magnitude of electric field, *E*rms , and ∂*E*rms <sup>2</sup> /∂*x* on the centerline of the creek-gap insulators. ∂*E*rms <sup>2</sup> /∂*x* was approximately constant within ±2.5%, in the range of *x* = 110 µm to 195 µm. **Figure 4.** (**a**) Top view of the designed electrode shape of isomotive iDEP creek-gap device. (**b**) Top view of the designed shape of the creek-gap insulators. (**c**) Distribution of the magnitude of electric field, *E*rms, and *∂E* 2 rms /*∂x* on the centerline of the creek-gap insulators. *∂E* 2 rms /*∂x* was approximately constant within ± 2.5%, in the range of *x* = 110 µm to 195 µm.

We used a glass wafer of 2 inch-diameter and 0.5 mm-thickness as the device's substrate. The aluminum of 400 nm-thickness was deposited using vacuum vapor deposition

3.1.1. Patterning of the Electrodes with Conventional Process

**3. Materials and Methods** 

*3.1. Fabrication of the Isomotive iDEP Device* 

#### **3. Materials and Methods**

*3.1. Fabrication of the Isomotive iDEP Device*

3.1.1. Patterning of the Electrodes with Conventional Process

We used a glass wafer of 2 inch-diameter and 0.5 mm-thickness as the device's substrate. The aluminum of 400 nm-thickness was deposited using vacuum vapor deposition (SVC-700TMSG, Sanyu Electron, Tokyo, Japan) and coated with the positive photoresist (S1805G, The Dow Chemical Company, Midland, MI, USA). After pre-bake on a hot plate at 90 ◦C for 3 min, the photoresist was exposed to the electrode pattern using Maskless Aligner (MLA150, Heidelberg Instruments, Heidelberg, Germany). The exposed photoresist was developed using 2.2% tetramethylammonium hydroxide (TMAH; MF-319, The Dow Chemical Company, Midland, MI, USA) and post-baked at 130 ◦C for 3 min. The aluminum electrode, uncovered with photoresist, was etched by mixed acid (Kanto Chemical, Tokyo, Japan) at 40 ◦C. Then, the photoresist was removed using AZ REMOVER 100 (Merck, Darmstadt, Germany).

#### 3.1.2. Patterning of the Insulators with Backside Exposure

The fabrication of the insulators should be highly accurate and consistent with the designing guideline (shown in Figure 4), in order to obtain a constant *∂E* 2 rms/*∂x*. The insulator was fabricated using backside exposure to form close to the design guideline (Figure 5). *Sensors* **2022**, *22*, x FOR PEER REVIEW 8 of 14

**Figure 5.** Illustrations of the patterning of insulators with backside exposure. (**a**) Patterning of the copper used as a photomask. (**b**) Backside exposure using the mask aligner. SU-8 covered with aluminum or copper is unexposed. (**c**) Selective exposure using the Maskless Aligner. (**d**) After developing SU-8, the copper photomask was etched. **Figure 5.** Illustrations of the patterning of insulators with backside exposure. (**a**) Patterning of the copper used as a photomask. (**b**) Backside exposure using the mask aligner. SU-8 covered with aluminum or copper is unexposed. (**c**) Selective exposure using the Maskless Aligner. (**d**) After developing SU-8, the copper photomask was etched.

The copper of 250 nm-thickness, used as photomask, was deposited on the substrate; these processes were the same as aluminum electrodes (Figure 5a). The copper was etched using Pure Etch C200 (Hayashi Pure Chemical, Osaka, Japan). After copper patterning, the substrate was baked as dehydration at 200 ◦C for 5 min. The baked substrate was coated with hexamethyldisilazane (HMDS; OAP, Tokyo Ohka Kogyo, Kanagawa, Japan) to promote adhesion between the glass substrate and insulator. Then, it was baked at 200 ◦C for 5 min. The negative photoresist (SU-8 3050, Kayaku Advanced Materials, Westborough, MI, USA), used as an insulator, was dripped 2 mL and coated by a spin coater to obtain an insulator with a thickness of 70 µm. The photoresist was soft-baked at 65 ◦C and 95 ◦C for 5 and 45 min, respectively. Then, it was cooled on the hot plate to room temperature (23 ◦C). The exposure was conducted in two steps, as follows: (1) the exposure of the photoresist on the glass from the backside of the substrate using Mask Aligner (BA100, Nanometric Technology, Tokyo, Japan) (Figure 5b); (2) exposure of the photoresist on the

**Figure 6.** Microscope image of the fabricated device. The shape of the fabricated device was mostly consistent with the designed parameters shown in Figure 4. (**a**) The entire chamber of the device, with the objective lens focused on the electrode. (**b**) The enlarged image of the creek-gap insulator, with the objective lens focused on the top of the insulator. (**c**) The depth synthetic image of the

We used monosized polystyrene particles of 20 µm-diameter (4220A, Thermo Fisher Scientific, Waltham, MA, USA) to measure the particle velocity, due to their uniformity and homogeneity. The diameter of the particles had the coefficient of variation of 1.1 % (written on the package). The particles were suspended in deionized (DI) water, with the conductivity of 0.3 mS/m, then filled in the chamber. The conductivity of DI water was measured using a conductivity meter (DS-72E, HORIBA, Ltd., Kyoto, Japan). A voltage was applied to the electrodes using a function generator (WF1968, NF Corp, Kanagawa, Japan), and the applied voltage was measured using an oscilloscope (GDS-2104, GW

narrower insulator gap. (**d**) The depth synthetic image of the wider insulator gap.

.

*3.2. Measurement of the Particle Velocity and Device Impedance* 

aluminum electrode from the topside of the substrate using Maskless Aligner (Figure 5c). The photoresist was post-baked at 65 ◦C and 95 ◦C for 5 min each; it was cooled on the hot plate to room temperature. In the photoresist development, we used propylene glycol methyl ether acetate (PGMEA)-based developer (SU-8 Developer, Kayaku Advanced Materials, Westborough, MI, USA). The developed photoresist was hard-baked at 65 ◦C for 5 min, 95 ◦C for 5 min, and 130 ◦C for 30 min; it was cooled on the hot plate to room temperature. The copper photomask was etched using the Pure Etch C200 (Figure 5d).

Figure 6 shows the image of the fabricated device. Figure 6a,b were captured using a microscope (OLS4500, Olympus, Tokyo, Japan), and Figure 6c,d show the depth synthetic image, captured by tilting the device with a high-speed microscope system (VW-9000, Keyence, Osaka, Japan). The shape of the fabricated device was mostly consistent with the designed parameters, shown in Figure 4. **Figure 5.** Illustrations of the patterning of insulators with backside exposure. (**a**) Patterning of the copper used as a photomask. (**b**) Backside exposure using the mask aligner. SU-8 covered with aluminum or copper is unexposed. (**c**) Selective exposure using the Maskless Aligner. (**d**) After developing SU-8, the copper photomask was etched.

*Sensors* **2022**, *22*, x FOR PEER REVIEW 8 of 14

**Figure 6.** Microscope image of the fabricated device. The shape of the fabricated device was mostly consistent with the designed parameters shown in Figure 4. (**a**) The entire chamber of the device, with the objective lens focused on the electrode. (**b**) The enlarged image of the creek-gap insulator, with the objective lens focused on the top of the insulator. (**c**) The depth synthetic image of the narrower insulator gap. (**d**) The depth synthetic image of the wider insulator gap. **Figure 6.** Microscope image of the fabricated device. The shape of the fabricated device was mostly consistent with the designed parameters shown in Figure 4. (**a**) The entire chamber of the device, with the objective lens focused on the electrode. (**b**) The enlarged image of the creek-gap insulator, with the objective lens focused on the top of the insulator. (**c**) The depth synthetic image of the narrower insulator gap. (**d**) The depth synthetic image of the wider insulator gap.

.

#### *3.2. Measurement of the Particle Velocity and Device Impedance 3.2. Measurement of the Particle Velocity and Device Impedance*

We used monosized polystyrene particles of 20 µm-diameter (4220A, Thermo Fisher Scientific, Waltham, MA, USA) to measure the particle velocity, due to their uniformity and homogeneity. The diameter of the particles had the coefficient of variation of 1.1 % (written on the package). The particles were suspended in deionized (DI) water, with the conductivity of 0.3 mS/m, then filled in the chamber. The conductivity of DI water was measured using a conductivity meter (DS-72E, HORIBA, Ltd., Kyoto, Japan). A voltage was applied to the electrodes using a function generator (WF1968, NF Corp, Kanagawa, Japan), and the applied voltage was measured using an oscilloscope (GDS-2104, GW We used monosized polystyrene particles of 20 µm-diameter (4220A, Thermo Fisher Scientific, Waltham, MA, USA) to measure the particle velocity, due to their uniformity and homogeneity. The diameter of the particles had the coefficient of variation of 1.1 % (written on the package). The particles were suspended in deionized (DI) water, with the conductivity of 0.3 mS/m, then filled in the chamber. The conductivity of DI water was measured using a conductivity meter (DS-72E, HORIBA, Ltd., Kyoto, Japan). A voltage was applied to the electrodes using a function generator (WF1968, NF Corp, Kanagawa, Japan), and the applied voltage was measured using an oscilloscope (GDS-2104, GW Instek, New Taipei City, Taiwan). The particle behavior and position were observed and recorded using a high-speed microscope system (VW-9000, Keyence, Osaka, Japan), with the frame rate of 250 fps. The particle velocities were measured using motion analyzer software with the microscope system.

Additionally, we used an impedance analyzer (4294A, Agilent Technologies, Santa Clara, CA, USA) to measure the device impedance. Prior to the measurement, the impedance analyzer was short- and open-calibrated by short- and open-circuited wires, respectively. The impedance was measured at 201 points on a logarithmic scale, from 40 Hz to 100 MHz, at a signal voltage of 0.5 Vrms. The measurement results were recorded to the floppy disk and analyzed on a personal computer. The conductivity of the medium was adjusted by mixing phosphate-buffered saline (PBS; 314-90185, Nippon Gene, Tokyo, Japan). MHz, at a signal voltage of 0.5 Vrms. The measurement results were recorded to the floppy disk and analyzed on a personal computer. The conductivity of the medium was adjusted by mixing phosphate-buffered saline (PBS; 314-90185, Nippon Gene, Tokyo, Japan).

Instek, New Taipei City, Taiwan). The particle behavior and position were observed and recorded using a high-speed microscope system (VW-9000, Keyence, Osaka, Japan), with the frame rate of 250 fps. The particle velocities were measured using motion analyzer

Additionally, we used an impedance analyzer (4294A, Agilent Technologies, Santa Clara, CA, USA) to measure the device impedance. Prior to the measurement, the impedance analyzer was short- and open-calibrated by short- and open-circuited wires, respectively. The impedance was measured at 201 points on a logarithmic scale, from 40 Hz to 100

#### **4. Results and Discussion 4. Results and Discussion**

#### *4.1. Distribution of the Particle Velocity 4.1. Distribution of the Particle Velocity*

software with the microscope system.

*Sensors* **2022**, *22*, x FOR PEER REVIEW 9 of 14

The electrical properties of particles or biological cells are evaluated by *∂E* 2 rms/*∂x*. *∂E* 2 rms/*∂x* is numerically calculated from the electromagnetic field simulation software. Therefore, the precision of the calculation is directly affected, in order to evaluate the electrical properties. In this section, we confirmed the calculation precision by comparing the calculated *∂E* 2 rms/*∂x* and measured particle velocity. The electrical properties of particles or biological cells are evaluated by ∂*E*rms <sup>2</sup> /∂*x*. ∂*E*rms <sup>2</sup> /∂*x* is numerically calculated from the electromagnetic field simulation software. Therefore, the precision of the calculation is directly affected, in order to evaluate the electrical properties. In this section, we confirmed the calculation precision by comparing the calculated ∂*E*rms <sup>2</sup> /∂*x* and measured particle velocity.

Figure 7 shows the behavior of the nDEP-exhibited particle. The applied voltage and frequency were 40 Vp-p and 1 MHz, respectively. The particle moved on the centerline between insulators at the single-particle level. Before applying the voltage, the particle settled down on the substrate, and the particle was driven by the DEP force, without levitation, after applying the voltage. Figure 8 shows the distribution of *∂E* 2 rms/*∂x* and particle velocity. One particle was measured at a time, and five particles were, thus, measured. The error bar of the plot shows the maximum and minimum velocity in this measurement. Both values were normalized at the position of 130 µm for comparison, since the particle velocity is proportional to *∂E* 2 rms/*∂x*, according to Equation (9). At the position of 130 µm, *∂E* 2 rms/*∂<sup>x</sup>* was 2.04 <sup>×</sup> <sup>10</sup><sup>12</sup> <sup>V</sup> <sup>2</sup>/m<sup>3</sup> , and the particle velocity was 19.3 µm/s. The distribution of *∂E* 2 rms/*∂x* was consistent with that of the particle velocity. This result indicates that *∂E* 2 rms/*∂x* was calculated with high precision, and it is possible to evaluate the electrical properties of particles using the calculated *∂E* 2 rms/*∂x*. Equation (9) assumes that the particle moves on a constant *∂E* 2 rms/*∂x*. However, the time constant shown in Equation (8) is sufficiently small, which it is considered negligible. Moreover, the distribution of particle velocity from 110 to 180 µm was approximately constant. This velocity profile means that the particles moved on the isomotive field. This isomotive movement enables particle characterization with straightforward measurement. Figure 7 shows the behavior of the nDEP-exhibited particle. The applied voltage and frequency were 40 Vp-p and 1 MHz, respectively. The particle moved on the centerline between insulators at the single-particle level. Before applying the voltage, the particle settled down on the substrate, and the particle was driven by the DEP force, without levitation, after applying the voltage. Figure 8 shows the distribution of ∂*E*rms <sup>2</sup> /∂*x* and particle velocity. One particle was measured at a time, and five particles were, thus, measured. The error bar of the plot shows the maximum and minimum velocity in this measurement. Both values were normalized at the position of 130 µm for comparison, since the particle velocity is proportional to ∂*E*rms <sup>2</sup> /∂*x*, according to Equation (9). At the position of 130 µm, ∂*E*rms <sup>2</sup> /∂*x* was 2.04 × 1012 V2/m3, and the particle velocity was 19.3 µm/s. The distribution of ∂*E*rms <sup>2</sup> /∂*x* was consistent with that of the particle velocity. This result indicates that ∂*E*rms <sup>2</sup> /∂*x* was calculated with high precision, and it is possible to evaluate the electrical properties of particles using the calculated ∂*E*rms <sup>2</sup> /∂*x*. Equation (9) assumes that the particle moves on a constant ∂*E*rms <sup>2</sup> /∂*x*. However, the time constant shown in Equation (8) is sufficiently small, which it is considered negligible. Moreover, the distribution of particle velocity from 110 to 180 µm was approximately constant. This velocity profile means that the particles moved on the isomotive field. This isomotive movement enables particle characterization with straightforward measurement.

**Figure 7.** Behavior of the nDEP-exhibited particle. The applied voltage and frequency were 40 Vp-p and 1 MHz, respectively. The particle individually moved on the centerline between insulators without levitation. **Figure 7.** Behavior of the nDEP-exhibited particle. The applied voltage and frequency were 40 Vp-p and 1 MHz, respectively. The particle individually moved on the centerline between insulators without levitation.

#### *4.2. Particle Velocity against Applied Frequency*

In this section, we discussed the measurement result of the frequency property of particle velocity. We applied a sinusoidal voltage of 40 Vp-p and frequency from 10 Hz to 20 MHz to the electrode. Additionally, we measured the velocity of five particles at each frequency. Then, the velocity was calculated from the travel time, using *x* = 135 to 115 µm, as shown in Figures 4 and 7. At frequencies of 10, 30, and 60 Hz, the particle tracking software of the microscope system was used to measure the particle velocity because the particles moved, vibrating in time with the sinusoidal wave, and the average velocity was

calculated from the tracked velocity. In all frequencies, the particle exhibited nDEP because the permittivity and conductivity of polystyrene are lower than DI water. *Sensors* **2022**, *22*, x FOR PEER REVIEW 10 of 14

**Figure 8.** Distribution of the calculated ∂*E*rms <sup>2</sup> /∂*x* and measured particle velocity. The applied voltage and frequency were 40 Vp-p and 1 MHz, respectively. The error bar of the plot shows the maximum and minimum velocity in the measurement. **Figure 8.** Distribution of the calculated *∂E* 2 rms /*∂x* and measured particle velocity. The applied voltage and frequency were 40 Vp-p and 1 MHz, respectively. The error bar of the plot shows the maximum and minimum velocity in the measurement.

*4.2. Particle Velocity against Applied Frequency*  In this section, we discussed the measurement result of the frequency property of particle velocity. We applied a sinusoidal voltage of 40 Vp-p and frequency from 10 Hz to 20 MHz to the electrode. Additionally, we measured the velocity of five particles at each frequency. Then, the velocity was calculated from the travel time, using *x* = 135 to 115 µm, as shown in Figures 4 and 7. At frequencies of 10, 30, and 60 Hz, the particle tracking software of the microscope system was used to measure the particle velocity because the particles moved, vibrating in time with the sinusoidal wave, and the average velocity was Figure 9 shows the particle velocity against the applied frequency. The error bar of the plot shows the maximum and minimum velocity in the measurement. The velocity measurement of a wide frequency band was performed using the proposed isomotive iDEP device. Generally, the velocity measurement using eDEP devices is not easy below 10 kHz because of the EDL between the medium and electrode. However, the influence of the EDL in iDEP occurs at a very low frequency because the electrode distance of the iDEP device is longer than that of the eDEP device. In the proposed device, the influence occurs below 100 Hz. Further discussions are described in the next section. *Sensors* **2022**, *22*, x FOR PEER REVIEW 11 of 14

20 MHz. The DEP-induced particle velocity is dependent on the unique electrical properties of the particles, such as biological cells, and can be utilized to identify the cell types. Ad-**Figure 9.** Particle velocity against the applied frequency. The velocity was measured from 10 Hz to 20 MHz. The error bar of the plot shows the maximum and minimum velocity in the measurement. The velocity was calculated from the travel time, using *x* = 135 to 115 µm, as shown in Figures 4 and 7. **Figure 9.** Particle velocity against the applied frequency. The velocity was measured from 10 Hz to 20 MHz. The error bar of the plot shows the maximum and minimum velocity in the measurement. The velocity was calculated from the travel time, using *x* = 135 to 115 µm, as shown in Figures 4 and 7.

] of the homogene-

ditionally, the new DEP properties of biological cells at low frequencies, through singlecell analysis, can be revealed by measuring the cell velocity using the proposed device. *4.3. Impedance Measurement of the Proposed Device*  In this section, the impedance of the proposed device was investigated to elucidate the influence of the EDL at the interface between the medium and electrode at low frequency. Figure 10a shows the equivalent circuit, considering elements consisting of the According to Equation (9), the electrical properties of the particles are reflected in the frequency property of the particle velocity by Re[*CM*∗ . The Re[*CM*∗ of the homogeneous particles has a low- and high-frequency limit and is dominated by conductivity and permittivity, respectively [28]. The limit values were at 100 Hz to 100 kHz and 5 MHz to 20 MHz.

device at the measurement frequency. Here, *R*E, *R*EDL, and *R*m are the resistance of the electrode, EDL, and medium, respectively; *C*EDL and *C*m are the capacitance of the EDL and medium, respectively. The subscript of "\_S" and "\_L" mean the electrodes with a smaller The DEP-induced particle velocity is dependent on the unique electrical properties of the particles, such as biological cells, and can be utilized to identify the cell types.

The impedance measurement using the medium of low conductivities was impossible due to the limitation of the impedance analyzer. Accordingly, the influence of the EDL, when DI water was used, was predicted by the impedance measurement results, using the medium of high conductivities. Figure 10a shows the measurement results, expressed by a complex impedance locus (also called Cole–Cole plots). At a very high frequency, the measured impedance was nearly 0 Ω because *C*m and *C*EDL were considered short-circuited. As the frequency decreased, the impedance of the medium was observed, due to *C*<sup>m</sup> ≪ *C*EDL. The locus of a parallel circuit, consisting of a resistance and capacitance, is represented by a semi-circle. With further decrease in the frequency, the component of the imaginary part was significantly decreased, and the measured impedance was approximately equal to *R*m. The diameter of the semi-circle is proportional to *σ*mିଵ (Figure 11a). At a very low frequency, the impedance of *C*EDL was measured with the sum of *R*m. Accordingly, the EDL influences the measurement of particle velocities below the frequency, which is the contact point between the semi-circle by the medium and locus by the EDL. Figure 11b shows the frequencies of the contact point, *f*cp, against the conductivity of the medium. There was a linear relationship between *f*cp and *σ*m. Additionally, *f*cp (predicted by the linear approximation) is 131 Hz at the conductivity of DI water (0.3 mS/m). This frequency was almost consistent with the frequency where particle velocity began to decay; thus, suggesting the decay is due to the EDL. At the frequency below *f*cp, the equivalent circuit of the proposed device is more simplified as a series circuit of *R*m and *C*EDL. The cut-off frequency, *f*co, where the square of the voltage of *R*m is half of the square of the

voltage applied to the device, is given by:

The equivalent circuit is simplified as shown Figure 10b.

the EDL.

and larger area, respectively, at the following conditions:

Additionally, the new DEP properties of biological cells at low frequencies, through singlecell analysis, can be revealed by measuring the cell velocity using the proposed device.

#### *4.3. Impedance Measurement of the Proposed Device Sensors* **2022**, *22*, x FOR PEER REVIEW 12 of 14

In this section, the impedance of the proposed device was investigated to elucidate the influence of the EDL at the interface between the medium and electrode at low frequency. Figure 10a shows the equivalent circuit, considering elements consisting of the device at the measurement frequency. Here, *R*E, *R*EDL, and *R*<sup>m</sup> are the resistance of the electrode, EDL, and medium, respectively; *C*EDL and *C*<sup>m</sup> are the capacitance of the EDL and medium, respectively. The subscript of "\_S" and "\_L" mean the electrodes with a smaller and larger area, respectively, at the following conditions: *f* co<sup>=</sup> <sup>1</sup> 2π*C*EDL*R*<sup>m</sup> (11) According to Equation (11), *f*co decreases as *R*m increases. In the iDEP device, *R*m is higher than the general eDEP devices because of the distance between electrodes, which


**Figure 10.** Equivalent circuit of the proposed device. (**a**) Equivalent circuit, considering elements consisting of the device at the measurement frequency. (**b**) Simplified equivalent circuit. **Figure 10.** Equivalent circuit of the proposed device. (**a**) Equivalent circuit, considering elements consisting of the device at the measurement frequency. (**b**) Simplified equivalent circuit.

The equivalent circuit is simplified as shown Figure 10b.

**Figure 11.** Measurement results of the proposed device. (**a**) Complex impedance locus with different σm. The diameter of the semi-circle is proportional to the σmିଵ. (**b**) The frequency, which is the contact point between the semi-circle by the medium and locus by the EDL against σm. **5. Conclusions**  In this paper, we proposed the isomotive iDEP creek-gap device for single-cell analysis. The design guideline of the proposed device was based on the observations from the The impedance measurement using the medium of low conductivities was impossible due to the limitation of the impedance analyzer. Accordingly, the influence of the EDL, when DI water was used, was predicted by the impedance measurement results, using the medium of high conductivities. Figure 10a shows the measurement results, expressed by a complex impedance locus (also called Cole–Cole plots). At a very high frequency, the measured impedance was nearly 0 Ω because *C*<sup>m</sup> and *C*EDL were considered short-circuited. As the frequency decreased, the impedance of the medium was observed, due to *C*<sup>m</sup>*C*EDL. The locus of a parallel circuit, consisting of a resistance and capacitance, is represented by a semi-circle. With further decrease in the frequency, the component of the imaginary part was significantly decreased, and the measured impedance was approximately equal to *R*m. The diameter of the semi-circle is proportional to *σ*<sup>m</sup> −1 (Figure 11a). At a very low frequency, the impedance of *C*EDL was measured with the sum of *R*m. Accordingly, the EDL influences the measurement of particle velocities below the frequency, which is the contact point between the semi-circle by the medium and locus by the EDL. Figure 11b shows the frequencies of the contact point, *f* cp, against the conductivity of the medium. There was a linear relationship between *f* cp and *σ*m. Additionally, *f* cp (predicted by the linear approximation) is 131 Hz at the conductivity of DI water (0.3 mS/m). This frequency was almost consistent with the frequency where particle velocity began to decay; thus, suggesting the decay is due to the EDL. At the frequency below *f* cp, the equivalent circuit of the proposed device is more simplified as a series circuit of *R*<sup>m</sup> and *C*EDL. The cut-off frequency, *f* co, where the square of the voltage of *R*<sup>m</sup> is half of the square of the voltage applied to the device, is given by:

device. Furthermore, the distribution of the measured particle velocity was consistent

equivalent circuit model of the proposed device, with the impedance measurement. These

$$f\_{\rm co} = \frac{1}{2\pi \mathcal{C}\_{\rm EDL} R\_{\rm m}} \tag{11}$$

<sup>2</sup> . Additionally, the wide frequency band measurement of

with that of the calculated ∇*E*rms

**Figure 10.** Equivalent circuit of the proposed device. (**a**) Equivalent circuit, considering elements

consisting of the device at the measurement frequency. (**b**) Simplified equivalent circuit.

*f*

quency, were measured in the iDEP device.

co<sup>=</sup> <sup>1</sup> 2π*C*EDL*R*<sup>m</sup>

According to Equation (11), *f*co decreases as *R*m increases. In the iDEP device, *R*m is higher than the general eDEP devices because of the distance between electrodes, which is applied via high and low voltage in the iDEP device, is longer than that of the eDEP devices. Therefore, the particle velocities at a wide frequency band, including low fre-

(11)

**Figure 11.** Measurement results of the proposed device. (**a**) Complex impedance locus with different σm. The diameter of the semi-circle is proportional to the σmିଵ. (**b**) The frequency, which is the contact point between the semi-circle by the medium and locus by the EDL against σm. **Figure 11.** Measurement results of the proposed device. (**a**) Complex impedance locus with different *σ*m. The diameter of the semi-circle is proportional to the *σ*<sup>m</sup> −1 . (**b**) The frequency, which is the contact point between the semi-circle by the medium and locus by the EDL against *σ*m.

**5. Conclusions**  In this paper, we proposed the isomotive iDEP creek-gap device for single-cell analysis. The design guideline of the proposed device was based on the observations from the generated electric field properties of the simplified iDEP device. The ∇*E*rms 2 generated on the centerline between the creek-gap insulators is more constant than that of the previous According to Equation (11), *f* co decreases as *R*<sup>m</sup> increases. In the iDEP device, *R*<sup>m</sup> is higher than the general eDEP devices because of the distance between electrodes, which is applied via high and low voltage in the iDEP device, is longer than that of the eDEP devices. Therefore, the particle velocities at a wide frequency band, including low frequency, were measured in the iDEP device.

#### device. Furthermore, the distribution of the measured particle velocity was consistent **5. Conclusions**

with that of the calculated ∇*E*rms <sup>2</sup> . Additionally, the wide frequency band measurement of the particle velocity was performed from 10 Hz to 20 MHz. The velocity is decayed below 100 Hz, due to the EDL between the electrode and medium. This is explained using the equivalent circuit model of the proposed device, with the impedance measurement. These In this paper, we proposed the isomotive iDEP creek-gap device for single-cell analysis. The design guideline of the proposed device was based on the observations from the generated electric field properties of the simplified iDEP device. The ∇*E* 2 rms generated on the centerline between the creek-gap insulators is more constant than that of the previous device. Furthermore, the distribution of the measured particle velocity was consistent with that of the calculated ∇*E* 2 rms. Additionally, the wide frequency band measurement of the particle velocity was performed from 10 Hz to 20 MHz. The velocity is decayed below 100 Hz, due to the EDL between the electrode and medium. This is explained using the equivalent circuit model of the proposed device, with the impedance measurement. These results suggest that various particle or cell properties are obtained by simple measurement using the proposed device.

Because of the simple principle, inexpensive fabrication, and high versatility of isoDEP, it is considered a major method of single-cell analysis and micromaterial evaluation. However, the proposed device has challenges with low throughput. Thus, new explorations are urgently needed that combine microfluidics, multiple arrays, automatic video processing, and frequency sweeping.

**Author Contributions:** Conceptualization, R.N. and M.E.; methodology, R.N. and M.E; software, R.N.; validation, R.N. and M.E.; formal analysis, R.N.; investigation, R.N.; resources, M.E.; writing original draft preparation, R.N.; writing—review and editing, M.E.; visualization, R.N.; supervision, M.E.; project administration, M.E.; funding acquisition, M.E. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported in part by JSPS KAKENHI, Grant Number JP17H04704.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** The authors would like to thank Matsumura for providing the initial conditions for fabricating devices using back exposure.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Data Analysis and System Development for Medical Professionals on Sleep Apnea Syndrome and Orthostatic Dysregulation by Processing-Healthcare Professionals and Patients**

**Miyori Shirasuna**

National Institute of Technology, Tsuruoka College, Yamagata 997-8511, Japan; m-shirasuna@tsuruoka-nct.ac.jp

**Abstract:** This paper presents the details of our research and the activities involved. Japan is one of the most advanced countries in medicine worldwide. However, in terms of technology, knowledge sharing, and successor development, Japanese medicine lags behind other developed countries, and these matters require addressing. The country is also facing a shortage of doctors, among other things, and this medical problem will surely become critical in the near future. In this study, we aim to help solve such problems from the medical engineering viewpoint, analyze and create systems based on the experience of doctors from the engineering viewpoint, and make it easy for patients to understand orthodox and general statistical analysis methods. We perform a visualization and quantitative medical data analysis and examine diagnostic support. We consider sleep apnea syndrome (SAS), and orthostatic dysregulation (OD) in children in this study. This research aims to detect SAS early, identify people with pre-SAS who are likely to become SAS in the near future, and identify OD. We analyze and identify these diseases through statistics and a multivariate analysis and create a dedicated analysis system for them. Our research and system development will allow specialists to make informed diagnoses, reproduce empirical rules, improve work efficiency, and improve patients' health awareness. This research has only looks at two diseases, but these methods can be expected to be applied to other diseases.

**Keywords:** medical engineering; medical assistance; multivariate analysis; sleep apnea syndrome; orthostatic regulation disorder

#### **1. Introduction**

Although Japan is at the forefront of medical technology worldwide, it has many problems and lags behind other developed countries [1,2]. There are many medical support systems in engineering. For example, systems for medical professionals include cancer diagnosis, brain infarction diagnosis, and community medical support [3–6]. Additionally, for non-medical personnel, there are health management systems using smart watches, AIbased disease identification applications, and applications for diagnosing the possibility of diabetes [7–9]. Some challenges currently faced by the country are declining birthrate, aging population, shortage of doctors, lack of accumulation and sharing of medical expertise, and immaturity of training successor doctors. These problems will become even more critical in the future.

To help address these problems, we are collaborating with medical professionals to examine medical diagnostic support from the biomedical engineering standpoint. The methods we used are orthodox and popular statistical analysis and time-frequency analysis, such as data mining and wavelet transformation. Our study focuses on sleep apnea syndrome (SAS) and orthostatic dysregulation (OD) in children.

The possible diagnosis of SAS, including in the reserve group, was performed only through voice. In current SAS diagnostics, patients are diagnosed based on a device attached to the body, which is stressful for them and makes accurate measurement difficult, but the voice is recorded and analyzed with an integrated circuit (IC) recorder. This method

**Citation:** Shirasuna, M. Data Analysis and System Development for Medical Professionals on Sleep Apnea Syndrome and Orthostatic Dysregulation by Processing-Healthcare Professionals and Patients. *Sensors* **2022**, *22*, 1254. https://doi.org/10.3390/s22031254

Academic Editor: Andrea Facchinetti

Received: 24 November 2021 Accepted: 28 January 2022 Published: 7 February 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

causes less stress on the patient's mind and body. Based on data of blood pressure and heart rate, OD reproduces the subclasses that doctors use as a rule of thumb. This is because many off-the-shelf data analysis systems and tools are often difficult to maintain by a diagnostician or analyst who uses them [10,11]. This classification is important because there are several types of subclasses and the treatment methods for patients differ depending on the type, but this is largely due to the experience of doctors. Thus, we collect data analysis using a simple and easy to understand system, and maintenance and version upgrades are easy owing to its simplicity. The system presents the ability for an analysis of quantitative results about diagnosis results and rules of thumb. Because raw and standardized data are stored in a database, the system performs the analysis and obtains quantitative results. Hence, the analysis results are accurate and consistent. In addition, the data of patients and those who had no health problems used in this study were fully explained by specialists to the subjects or their parents, and the contents of the study were understood. Additionally, the personal information is provided to us in an anonymous state.

In addition, our study considers and proposes methods to present easy-to-understand diagnostic results to patients. We analyze and reproduce the experience of specialist doctors, as the visual and quantitative realization of diagnostic results is not yet popular in the medicine field of Japan. For example, SAS diagnostic results are shown only in letters and numbers, and OD results are too complex for elementary to high school patients. Therefore, solutions to the medical problems mentioned previously should be determined.

The remainder of this paper is organized as follows. Section 2 describes our approach to medical data analysis. Section 3 describes our collaboration with medical specialists. Section 4 describes the SAS, and in Section 5 OD research cases. Section 6 describes the medical support system in development. Finally, Section 7 provides a summary of this study.

#### **2. Medical Data Analysis**

Recently, medical data analysis has undergone rapid development, which can be attributed to the advancement of machine learning and artificial intelligence (AI) technology. For example, since the onset the coronavirus disease 2019, which continues to infect people worldwide, the daily elucidation of virus details and simulations of its infection status have been reported [12,13]. Further, biomedical engineering has been applied to the elucidation and analayse many diseases. For example, new medications, artificial organs, limb and dental implant construction have been developed.

For biomedical engineering, a data analysis is performed on time series, audio, images, other data types, or a combination thereof. Related studies are progressing in advanced medical countries; nevertheless, this remains an urgent requirement in Japan in the future. Therefore, this field should be developed and disseminated in Japan.

However, using machine learning and AI in these studies involves some problems. First, these methods require large data and cannot be applied to data with a small number of samples. Second, even if the accuracy of the analysis results is acceptable, many points should be clarified because the analysis process of these methods is a "black box". Third, when these methods are used, the analysis accuracy is highest during delivery (during initial analysis/creation), but this accuracy declines as time passes. To maintain the accuracy of the analysis, analysts need to perform continuous maintenance or upgrades [14,15]. Fourth, many of these analytical methods are difficult to understand because understanding analysis requires a good knowledge of statistics and mathematics. Therefore, the results of the analysis cannot be easily and accurately understood by everyone, aside from engineers who are familiar with them [16]. Fifth, machine learning and AI-based data analyses, such as AI-based automatic blood pressure monitors that have been reported to be medically unreliable, leading to credibility issues [17].

The medical data that we consider in this study do not contain the required amount of data for machine learning and AI, and such cases are common in medical data. Therefore, we must analyze a limited number of data, and the ideal analysis of such cases is a "white open" analysis, which is a synonym of "black box".

#### **3. Cooperation between Doctors and Data Scientists**

Doctors' expertise, experience, and intuition are invaluable, and their diagnoses are difficult to automate. Thus, we collaborated with doctors who specialize in SAS and OD and those who are familiar with medical data analysis. During our meetings and interviews with them, third-party doctors made unclear points regarding the study results reported, medical equipment used, and analysis methods used for medical data analysis.

For example, the data of medical devices used in studies are often unavailable to users, including doctors. Even if these data are acquired, expensive software is often needed for data export. Therefore, many physicians, especially those who are not involved in large projects or research groups, lack the tools, data, and time needed to acquire and analyze certain data. Typically, they also find it difficult to analyze the data using such equipment. This is a common problem for small- and medium-sized medical institutions that are privately owned by doctors in Japan.

There are often significant gaps between clients and analysts seeking analysis sup-port. This is because analysts tend to try new or complex analytical techniques, and clients who are presented with the results obtained from these analytical methods often do not understand the content. This problem is also related to machine learning and medical-data analysis that uses AI. Although these analytical methods are convenient, they it is difficult to visualize the analytical process using them because these analysis processes are black boxes in many cases. Therefore, our study focuses on how our clients, both professional physicians and analysts, can understand, share, and discuss the analytical methods and implications of the results.

#### **4. Sleep Apnea Syndrome (SAS) Research**

#### *4.1. About SAS*

SAS is a lifestyle-related disease that has attracted scholarly attention worldwide [18]. For example, traffic accidents caused by sleep apnea in patients with SAS are occasionally reported. Many SAS patients are not treated because they have few subjective symptoms and do not have a sense of crisis, and are known as "hidden SAS patients". It is considered that SAS patients make up a few percent of the Japanese population. In addition, SAS is a common disease in middle-aged and elderly people; nonetheless, the number of patients is increasing in the younger generation. Therefore, a simple and accurate diagnosis of SAS is required. However, there are many problems associated with SAS treatment. In Japan, as diagnosing SAS is time consuming because of the following procedure:


This process can take a month or longer, and the patient may have to be out of work for an SAS diagnosis, which is difficult considering the Japanese working style. Therefore, even with an SAS diagnosis, many patients find this process grueling and abandon the SAS diagnosis or discontinue treatment prematurely. This is also a problem for medical institutions. In addition, the current SAS diagnostics are inefficient. Sleep medicine professionals spend approximately 6 h per subject to manually and visually analyze more than 10 types of data, such as blood pressure and pulse, from PSG, which is accompanied by an overnight hospital stay [19,20]. This manual and visual analysis also has problems in that the diagnostic results are inconsistent.

### *4.2. Our SAS Research*

#### 4.2.1. Our SAS Diagnosis

We evaluated the possibility of SAS using only sleep apnea sound data, and the PSG did not obtain breath-sound data during sleep. This is because the data for sleep breath sounds are long and noisy, and breath sounds during 6–8 h of sleep might be inconsistent. In addition, the duration of sleep is as long as 6–8 h, and thus, retaining and analyzing these data are challenging. The World Health Organization (WHO) has established criteria for SAS diagnosis; however, these criteria do not include breath sounds during sleep. Sleep sounds of patients with SAS are characteristic. For example, the sleep sound of SAS is greater than for a person without health issues, the sound when resuming breathing from apnea is very loud, and the breathing intervals are not uniform [21,22]. Considering the SAS patients under diagnosis, there were many cases in which the possibility of SAS was identified by family members and acquaintances who noticed abnormalities in their sleep apnea sounds, which prompted them to go to the hospital. Therefore, many studies have been conducted on the possibility of diagnosing SAS using breath sounds during sleep. Recently, studies using machine learning and AI have been reported [23,24].

Our study for determining the potential of SAS using sleep apnea sounds was simple. Because we wanted to make the SAS diagnosis processes easier, as identified in the previous section, we developed a method that would allow subjects to perform SAS diagnosis at home using a simple IC recorder for the benefit of both the subject and the analyst. Because the analysis is simple, the analysis time can be shortened. In addition, this method addresses the potential problem of difficulty sleeping in a non-home sleeping environment without the device being worn on the subject's body.

Specifically, our study record breath sounds during sleep with an IC recorder, determines and analyzes the characteristics of SAS sound, and determines the probability of SAS using a simple method of multivariate analysis. From the characteristics of SAS sounds, 15 s centered on breath sounds during sleep with high sound pressure were cut out, and five sounds data were created for each person. Additionally, there were a total of 42 subjects, 37 SAS patients, 5 healthy individuals, and adult men and women, and there was no missing data. Therefore, 210 sounds data were created and we included a flag to determine whether their data suggests them to be an SAS patient or Non SAS. Then, we cluster-analyzed these sound data from 4 to 7 by the k-means method Euclidean distance. The results for six clusters presented a silhouette coefficient of 0.81 at the highest, followed by 0.69 with five clusters. In addition, we showed that SAS patients contained one of the three healthy data clustered into these six, and patients without health problems did not contain these three data. With this method, the probability diagnosis of SAS was 100%. Additionally, the results were promising, and the diagnosis time can be reduced to about 1/40 per person [22].

In the current study, we diverted and developed analytical methods and new devices that combine the above methods with signal processing methods.

Figure 1 shows the diagnostic procedure for SAS.

#### 4.2.2. SAS Sound Visualization

Figure 2 shows an example of visualizing the respiratory state of breathing sounds during the sleep of an SAS-affected person. The presence or absence of breath sounds during sleep recorded on the IC recorder was binarized with 1 and 0 for breathing and no breathing every second, respectively. Thereafter, the values were added to the time series of each second, and the added value of each second was graphed. The section between the red dashed lines is the WHO-defined apnea state for 8 s or longer [25]. Therefore, if the slope of this graph is 0, apnea occurs. This graphing method is simple and is easier for the affected individuals to understand than the diagnostic results reported in medical terminology and numbers.

**Figure 1.** Step comparison of SAS diagnosis: (**A**) current medical institutions and (**B**) proposed procedure.

**Figure 2.** Example of visualizing the respiratory state of breathing sound during sleep of an SASaffected person.

In addition, current SAS treatments often do not clearly indicate changes in pathological conditions either visually or quantitatively. Therefore, both doctors and patients cannot clearly and quantitatively understand how the condition has improved.

In another example, we used a continuous wavelet transform (CWT) to visualize breath sounds during sleep and to communicate the condition to the patient. CWT is a useful analysis method (considering time and frequency) and is expressed by Equation (1) [23,24].

$$\mathcal{W}(\mathfrak{a}, \mathfrak{f}) = \int\_{-\infty}^{\infty} \frac{1}{\sqrt{\alpha}} \overline{\psi \left( \frac{t - \beta}{\alpha} \right)} f(t) dt \tag{1}$$

Here, *α*, *β*, *t* ∈ *R* (*R* is the real number), *α* > 0, *β* ≥ 0, where 1/*α* represents frequency, and *β* represents time. The value of *W* (α, β) obtained from this equation is known as the wavelet coefficient. *ψ* (*t*) in Equation (1) is the mother wavelet, which is the core of the decomposition of the CWT. The Gabor wavelet (GW), which is one of the Mother Wavelet (MW)s, is expressed by Equation (2) [26–28].

$$
\psi^G(t) = \frac{1}{\pi^{1/4}\sqrt{\delta}} e^{-\frac{t^2}{2(\delta)^2}} e^{i2\pi t} \tag{2}
$$

Here, *t*, *δ* ∈ *R* and *δ* > 0 in Equation (2). The GW is the most popular mother wavelet because it has the highest resolution among the many proposed MWs and is compatible with human vision. We improved the GW to create an SAS-only MW. The equation is given by Equation (3).

$$\psi^{SMS}(t) = \frac{1}{\pi^{\frac{1}{4}}\sqrt{\delta\_D}\left(\sqrt{A} + \sqrt{B}\right)} \left\{ e^{-\frac{\left(\delta I\right)^2}{2\left(\delta\_D\right)^2}} + e^{-\frac{\left(\delta I\right)^2}{2\left(\delta\_D\right)^2}} \right\} e^{i2\pi} \tag{3}$$

Considering Equation (3), *t*, *δD*, *A*, *B* ∈ *R*, and *δD*, *A*, *B* > 0. This equation is known in this paper as the Designed MW for SAS, and we defined it as DMS. An outline of this function is presented in Figure 3. DMS is a combination of two GWs and is looser and longer than GWs.

**Figure 3.** (**A**) is example of the DMS function outline. And (**B**) is the example of the DMS function external shape, the solid red line is a real number, and the broken blue line is an imaginary number.

Figure 4 is a visualization of SAS sleep and non-SAS sleep breath sounds with GW at the top, and WC calculated from the DMS at the bottom. This is the waveform of the analyzed breath sounds. Each parameter in this figure is *δ* = 1.0 in Equation (2), and *A* = 1.0, *B* = 0.5 and *δ<sup>D</sup>* = 0.5 in Equation (3). Figure 4 shows that the warmer the color, the stronger the unique characteristics of the waveform. Compared to GW, DMS has more concentrated warm colors and shows a more focused reaction. In addition, the DMS has a weaker non-SAS response than the GW. Because SAS is a disease with few subjective symptoms, it is useful to visually indicate the condition of the patient as described above.

**Figure 4.** Comparison of breath sounds during sleep between SAS and non-SAS using GW and DMS.

#### **5. Orthostatic Dysregulation (OD) Analysis**

#### *5.1. About OD*

Approximately 700,000 elementary to high school students in Japan have OD disease, and the incidence in this age group is approximately 7% [29]. Orthostatic dysregulation is a disorder of the autonomic nervous system, specifically a disorder in which blood pressure and pulse are not well regulated. The main symptoms of OD include difficulty waking up in the morning, headache, nausea, and lack of motivation. Severe symptoms can lead to school refusal and can cause multiple social problems [29,30]. In addition, because such symptoms are not felt seriously by parents and school teachers, children with OD cannot be diagnosed, and they may be warned that they have "lazy disease". However, OD can be recovered with proper treatment and requires early diagnosis, support, and treatment.

Recently, OD has become well known in Japan, but with few specialist doctors. Therefore, many potentially affected children are diagnosed in internal medicine facilities near their homes. When we interviewed physicians and pediatricians, they indicated that OD was difficult to diagnose. Therefore, we aim to propose a simple method for diagnosing the possibility of OD. Our suggestion facilitates a simple diagnosis before seeing a doctor specialized in OD.

#### *5.2. Possible Diagnosis of OD*

For the diagnosis of OD, the patient performed a new standing test in the presence of a specialist doctor. The subject stood up on his own from the bed-rest state and measured his/her blood pressure and heart rate. Because OD is a disease of the autonomic nervous system in which blood pressure and heart rate are not well linked, blood pressure and heart rate are not well linked in this test as well. Conversely, for those who have no health problems, blood pressure and heart rate are linked to simple exercise scenes in daily life. The Fisher's correlation coefficient between the two is 0.59, and the variance is ±0.11 [31].

Based on the data from the new standing test of patients with OD, the correlation coefficient between blood pressure and heart rate was obtained for 3 m at 3 s intervals during bed rest and standing, as shown on the positioning map in Figure 5 [32].

**Figure 5.** Possibility of OD determined by obtaining Fisher's correlation coefficient of the bed rest-standing state using a positioning map.

Figure 5 shows the correlation coefficient between blood pressure and heart rate during bed rest and standing in a new standing test of 35 males and females, aged 10–18 who were patients with OD, and there was no missing data. The circles in the graph represent the positions of the correlation coefficients of OD patients. The two dashed lines in each of the vertical and horizontal directions shown by the broken lines in this map have a Fisher's correlation coefficient of 0.59 ± 0.11. They are lightly colored squares in between the two straight lines on the vertical and horizontal axes. If the blue circle mark position is located near to the red square area, the subject is considered to have no health problem. Then, the patient closest to the red square area is indicated by an arrow. The perpendicular distance between this patient and the square is plus 0.013. The position of the correlation of this OD patient is located outside this square. Since the data of people with no health problems are taken from the paper, it is not possible to set the error range and boundaries with OD patients, but in the future, we will collect data of people with no health problems and analyze them.

Considering this map, OD-affected persons have large individual differences in the correlation coefficient in both the bed rest and the standing state, and none of them correspond to those who have no health problems. Because people with OD often have low blood pressure, the possibility of OD can be easily diagnosed based on both the blood pressure rate and the positioning map. In addition, similar to the case of SAS in Figure 4, if this map is created for each treatment, the medical condition of the affected person can be confirmed, and it becomes easier for both the doctor and the patient to understand the situation.

#### **6. Medical Support System**

#### *6.1. About the Medical Support System*

This section describes the medical support system that we are developing. The process of analysis is important in the system creation.

The Faculty of Engineering is good at using mathematical formulas as tools, but tends to depend on them. For example, engineering students can effortlessly use a library of data analysis software, but they do not have sufficient control or understanding of the functional meaning of the library and the interpretation of the results. This "file/folder thinking", which is an image of binding knowledge and formulas into a file and stored in a folder of fields and subjects, is a feature common to Japanese engineering students.

We aim to develop a system that can provide medical support. For example, when doctors and medical professionals seek to clarify various hypotheses and thoughts in the process of analyzing patient measurement data, if the situation can be visualized during their analysis, this will be a strong support for them.

#### *6.2. Medical Support System and Functional Demand*

Currently, medical-support research uses machine learning and AI in many cases. However, these cannot be applied to data with a small number of samples, such as those that we have considered in this study. In addition, because the analysis process of these methods is a "black box", the examination and verification of outliers occur after the analysis, and the analyst cannot observe the calculation process. When these methods are used, the accuracy of the analysis is the highest during the initial analysis and creation, but this accuracy declines as time passes [14,15]. Moreover, many of these analytical methods are difficult to understand, and it is not easy for anyone, other than a technician who is familiar with them, to accurately understand the results of the analysis [16].

#### *6.3. Our Medical Support System*

Considering some cases, the proposed system does not use the development language library; hence, the calculation process of the analysis can be clarified. This creation process has two merits. First, the system creator's understanding of the analysis method is deepened. Second, cases exist where new ideas are born from a deeper understanding of the analysis method. The MW shown in Figure 3 is one such example, which is our medical system. Our collaborative doctor is interested in analyzing the data he holds. OD specialists make OD diagnoses based on the guidelines advocated by The Japanese Society of Psychosomatic Pediatrics, but many patients fall into exceptions and vague boundaries. Therefore, the visualization and verification of this boundary is required.

We actively exchanged opinions with doctors specializing in collaborative studies and researchers in other fields and discussed the demands of specialist doctors, objective opinions, and ideas that can be applied in other fields.

Our collaborating doctors are interested in a frequency analysis of biological signals and wish to visualize them simply by the FFT, so we are also investigating FFT methods. Figure 6 is an example of the verification how Overlap Processing (OP) is performed when the fast Fourier transform (FFT) is multiplied by the window function [33,34]. The purpose of the window function is to perform the FFT in a finite period. When a function is multiplied by a window function, a section of the function is cut out. Therefore, when the function multiplied by the window function is had FFT, FFT for a finite period can be performed. In this example, the continuity of the sine waveform of the test signal is lost, owing to the correction of the window function. Therefore, the effect of the inverse Fourier transform on the test waveform is verified.

**Figure 6.** Example of our system.

#### **7. Conclusions**

In this paper, we briefly described our research approach to medical data analysis in collaboration with specialist doctors, two practical examples of activities, the creation of medical support systems, and their important features. Regarding research on SAS, we showed the possibility of diagnosis by incorporating our proposal and described how to visually present it to patients. We also proposed a simple diagnosis of OD for research on OD. We provided some examples of the development of medical support systems.

Advancement in machine learning and AI enhances daily medical and biomedical engineering techniques that are useful for data analysis. Meanwhile, only a few studies were conducted on visualization and quantification of medical treatment, expert experience, and human senses, and we hope that this topic will attract scholarly attention. Additionally, we would like to study the method used in this paper to obtain more accurate results by conducting an analysis with different parameters from those used in this paper. We will continue to follow up on what has been described in this study and collaborate with doctors and experts in various fields to help improve healthcare in Japan and across the world. We will continue our research with a view of applying SAS research to sudden death in newborns and OD research to aging hypotension.

**Funding:** This study was supported by JSPS KAKENHI (Grant Number 20K20220).

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki, and approved by the Institutional Ethics Committee of Toyohashi University of Technology (protocol code 2020-10 and 28 July 2020).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to the subjects who cooperated in this study provided them on the premise that the data will not be published.

**Acknowledgments:** We are grateful to Norio KAZUMA (KAZUMA KODOMO Clinic, Saitama, Japan), YASUHIRO ISHKAWA (ISHIKAWA Clinic, Saitama, Japan), and SHO Tadashi (Hiroshima Institute of Technology, Hiroshima, Japan) for their efforts in this study.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Effects of Social Implementation Education for Assistive Device Engineers at NIT (KOSEN) through the Development of a Digital Reading Device for the Visually Impaired**

**Kimiyasu Kiyota 1,\* , Takaaki Ishibashi <sup>2</sup> , Manabu Shimakawa <sup>1</sup> and Kazuyuki Ito <sup>3</sup>**


**Abstract:** Assistive technology (AT) is any item, device, software, or product system used to enhance, maintain, or improve the functional abilities of people with disabilities. There are many people with disabilities in the world, including the visually impaired, the hearing impaired, and the physically impaired. We established the National KOSEN Support Equipment Development Network (KOSEN-AT) with technical college faculty members 10 years ago to assist these disabled and elderly people. However, Japan is facing the challenge of a rapidly aging society, and the digital transformation of assistive device development for people with disabilities has not been adequately addressed. A major reason for this is the lack of engineers in Japan who can develop products with an understanding of the needs of people with disabilities and the elderly. In this paper, we describe a new initiative of the GEAR 5.0 program, a practical engineer education program that will enable the development of assistive devices for the physically challenged and the elderly, which started in 2020 at the National Institute of Technology in Japan. We believe that it is necessary to educate technicians not only with conventional specialized skills, but also with a full understanding of the concept of disability and basic skills in assistive technology. Next, we developed "Touch Talker", a digital text reading system for the visually impaired. As a part of the GEAR 5.0 program, we conducted an evaluation experiment in which students from a technical college experienced visual impairment in the same blindfolded environment as visually impaired people to evaluate the developed assistive device. To verify its importance, we developed a digital text-to-speech system for the visually impaired, "Touch Talker", as part of the GEAR 5.0 program. We thought that by conducting evaluation experiments in the same blindfolded environment as visually impaired people, we could make technical college students aware of the difficulties of operating digital devices due to visual impairment. The results of the experiment showed that the developed "Touch Talker" was effective for both the visually impaired group and the blindfolded technical college student group. The evaluation results also showed a similar trend, confirming that the evaluation by blindfolded technical college students is effective for the development of assistive devices for the visually impaired. In addition, the technical college students who participated in the evaluation experiment were able to understand the difficulty of operating digital devices by experiencing visual impairment. It was suggested that the perspective of the people involved is important in the development of assistive devices.

**Keywords:** assistive technology; visually impaired support; education of engineers; text reading

#### **1. Introduction**

Recently, the enhancement of welfare and medical care for the super-aging society has become an issue not only in Japan but also on a global scale. The Japanese Ministry of

**Citation:** Kiyota, K.; Ishibashi, T.; Shimakawa, M.; Ito, K. Effects of Social Implementation Education for Assistive Device Engineers at NIT (KOSEN) through the Development of a Digital Reading Device for the Visually Impaired. *Sensors* **2022**, *22*, 1047. https://doi.org/10.3390/ s22031047

Academic Editors: Marco Leo and Toshiyo Tamura

Received: 6 December 2021 Accepted: 25 January 2022 Published: 28 January 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Education, Culture, Sports, Science and Technology (MEXT) recently imposed an obligation on all employers to employ people with disabilities at a rate higher than the legal employment rate, to encourage "people with disabilities to live and work together as members of the community" [1]. In response to this social situation, the MEXT has taken the lead in promoting policies in various schools that promote the early realization of an inclusive society based on rational consideration [2,3]. Under these circumstances, the demand for AT devices for the disabled and elderly is increasing in the welfare equipment industry and educational institutions, and there is an urgent need to train engineers who can develop assistive devices.

National colleges of technology (NCTs) are higher education institutions that accept graduates of junior high schools and provide an integrated education for 5 years (5.5 years for merchant marine technical colleges) to train engineers required by society. Many engineers with specialties in mechanical, electrical, information, biochemical, architectural, and civil engineering have come from NCTs [4]. However, the development of AT devices necessitates not only traditional, specialized skills but also new technical skills (AT skills) that take into account people's needs (disabled and elderly users). In other words, it is preferable to introduce technical education that employs a social implementation model (needs-based) for the development and evaluation of devices and orthotics in collaboration with specialists from various fields such as medical and welfare care. In response to such requests from special support schools and medical institutions in each prefecture, the faculty members of 13 NCTs took the lead in establishing the National KOSEN Assistive Technology (AT) Development Network (KOSEN-AT Net). In this network, each technical college has developed various devices and applications for special needs [5–12].

#### **2. Enhancement of Technical College Education (AT Future Technology Human Resource Development Model for a Sustainable Society)**

*2.1. GEAR 5.0 Project and eAT (Extended Assistive Technology)*

In Japan, the National Institute of Technology (KOSEN) started its latest research project, called GEAR 5.0, in May 2020. AT and medical engineering are key disciplines selected as the focus of the GEAR 5.0 project. This project aims to realize the next-generation AT (extended-AT (eAT)) that combines AT and digital technologies (AI, IoT, robotics, big data, and mobility, etc.) to realize Society 5.0. Here, "Society 5.0" refers to Japan's 5th Science and Technology Basic Plan, which refers to an ultra-smart society that utilizes digital technologies such as IoT and AI that integrate cyber and physical space.

Figure 1 shows the conceptual idea of the eAT (extended-AT) project in GEAR 5.0. We will encourage market-creating research that will result in increased employment opportunities for people with disabilities, improved quality of life for patients and people with disabilities, and increased healthy life expectancy for the elderly. Simultaneously, students at KOSEN(NIT colleges) will participate in the process of social implementation, acquiring a practical knowledge of advanced technologies through problem-solving activities and the development of prototype devices and services, and developing as engineers with an AT mindset who can solve problems from the perspective of people with disabilities. To promote the above research and education, we will accumulate and share the know-how of the design data and customization of the next-generation ATs developed by the technical colleges participating in GEAR 5.0 as an AT library that can be shared by all technical colleges in Japan, to improve the development efficiency of AT devices and services and promote the social implementation of ATs. Furthermore, we will establish a joint research network with medical and welfare institutions, companies, and local governments, among others, to share the voices of AT users, on-site needs, and issues, and to jointly create better AT devices and services, as well as to build a research system for industry–academia–government collaboration. The goal of this project is to produce engineers from technical colleges across the country who can realize "the creation of a symbiotic society by using eAT even for people with disabilities" [13]. These pilot initiatives that incorporate new technologies to extend

healthy life expectancy and improve wellbeing are at the beginning stages of consideration as a challenge not only for Japan but also for the world [14,15]. are at the beginning stages of consideration as a challenge not only for Japan but also for the world [14,15].

jointly create better AT devices and services, as well as to build a research system for industry–academia–government collaboration. The goal of this project is to produce engineers from technical colleges across the country who can realize "the creation of a symbiotic society by using eAT even for people with disabilities" [13]. These pilot initiatives that incorporate new technologies to extend healthy life expectancy and improve wellbeing

*Sensors* **2021**, *21*, x FOR PEER REVIEW 3 of 14

**Figure 1.** Conceptual diagram of the eAT (extended-AT) in the GEAR 5.0 project. **Figure 1.** Conceptual diagram of the eAT (extended-AT) in the GEAR 5.0 project.

#### *2.2. Enhancement of Technical College Education 2.2. Enhancement of Technical College Education*

Figure 2 shows the concept of the advancement of technical college engineer education in this GEAR 5.0 project (nursing care and medical engineering fields). In the field of AT, we aim for a new integration of education, research, and practice for the development of technical college education. In other words, we will practice the fusion of education, research, and social implementation that is being promoted at many universities. AT has so far focused on the production of assistive devices to overcome or complement disabilities. In the next generation, however, it will be essential to have a good understanding of people with disabilities when developing high-mix, low-volume assistive devices for various disabilities using digital technology. Therefore, this is a gap market where no major companies have been able to enter. Until now, the technical colleges collaborating in the nationwide KOSEN-AT network have been working individually with local special-needs schools, NPOs in Japan, and medical institutions to develop and research assistive devices. Figure 2 shows the concept of the advancement of technical college engineer education in this GEAR 5.0 project (nursing care and medical engineering fields). In the field of AT, we aim for a new integration of education, research, and practice for the development of technical college education. In other words, we will practice the fusion of education, research, and social implementation that is being promoted at many universities. AT has so far focused on the production of assistive devices to overcome or complement disabilities. In the next generation, however, it will be essential to have a good understanding of people with disabilities when developing high-mix, low-volume assistive devices for various disabilities using digital technology. Therefore, this is a gap market where no major companies have been able to enter. Until now, the technical colleges collaborating in the nationwide KOSEN-AT network have been working individually with local special-needs schools, NPOs in Japan, and medical institutions to develop and research assistive devices. *Sensors* **2021**, *21*, x FOR PEER REVIEW 4 of 14

**Figure 2.** Conceptual diagram of the eAT (extended-AT) in GEAR 5.0 project. **Figure 2.** Conceptual diagram of the eAT (extended-AT) in GEAR 5.0 project.

dents cannot easily read documents using digital terminals. In recent years, tablets for the visually impaired have been developed. BLITAB allows users to read entire pages on a tactile screen. BLITAB enables blind users for the first time to learn, work, and play on a single mobile device, providing digital access to information in real time. BLITAB converts any document into braille text. However, this system is for users who have access to braille

The following tablet terminal applications have been developed for use by the blind and elderly who cannot use braille. AccessNote for the iOS platform is designed particularly for VoiceOver users looking for an efficient, feature-rich note-taking experience. An inexpensive alternative to traditional note-takers, it allows users to combine efficient notetaking with the other features and functions of the iOS devices, making it accessible to blind and visually impaired people [17]. AccessWorld has also been optimized for iOS VoiceOver and accessibility features. Amazon Kindle allows users to use their iPhones and iPads as a Kindle device and access their e-books purchased from Amazon as well as download e-books directly to their devices [17]. However, with Amazon Kindle, visually impaired users have a hard time finding the part they want to read because the text is read aloud from the beginning of the document. The environment for ICT-assisted reading support described above is not sufficient for the visually impaired, especially those with intermediate visual impairment. For this reason, there is an urgent need to develop assistive devices based on Society 5.0 that can be used with an understanding of the situation of users with disabilities. Especially for the visually impaired, retrieving the necessary information from a vast amount of information is a very important task for users who

The goal of this research is to create a digital smart terminal that allows visually impaired people to read tactilely by "finger tracing" the freely displayed text characters without any visual feedback. To realize this assistive technology, we propose a new system (dubbed "Touch Talker") that outputs text at a specific position from a large number

**3. New Concept of the "Touch Talker" System**  *3.1. Overview and Background of the Study* 

listen from voice output devices.

[16].

#### **3. New Concept of the "Touch Talker" System**

#### *3.1. Overview and Background of the Study*

People with disabilities who suddenly lose their sight due to diseases or traffic accidents cannot easily read documents using digital terminals. In recent years, tablets for the visually impaired have been developed. BLITAB allows users to read entire pages on a tactile screen. BLITAB enables blind users for the first time to learn, work, and play on a single mobile device, providing digital access to information in real time. BLITAB converts any document into braille text. However, this system is for users who have access to braille [16].

The following tablet terminal applications have been developed for use by the blind and elderly who cannot use braille. AccessNote for the iOS platform is designed particularly for VoiceOver users looking for an efficient, feature-rich note-taking experience. An inexpensive alternative to traditional note-takers, it allows users to combine efficient note-taking with the other features and functions of the iOS devices, making it accessible to blind and visually impaired people [17]. AccessWorld has also been optimized for iOS VoiceOver and accessibility features. Amazon Kindle allows users to use their iPhones and iPads as a Kindle device and access their e-books purchased from Amazon as well as download e-books directly to their devices [17]. However, with Amazon Kindle, visually impaired users have a hard time finding the part they want to read because the text is read aloud from the beginning of the document. The environment for ICT-assisted reading support described above is not sufficient for the visually impaired, especially those with intermediate visual impairment. For this reason, there is an urgent need to develop assistive devices based on Society 5.0 that can be used with an understanding of the situation of users with disabilities. Especially for the visually impaired, retrieving the necessary information from a vast amount of information is a very important task for users who listen from voice output devices.

The goal of this research is to create a digital smart terminal that allows visually impaired people to read tactilely by "finger tracing" the freely displayed text characters without any visual feedback. To realize this assistive technology, we propose a new system (dubbed "Touch Talker") that outputs text at a specific position from a large number of text documents displayed on a smart terminal's screen by "tracing" the screen with a finger as shown in Figure 3. *Sensors* **2021**, *21*, x FOR PEER REVIEW 5 of 14 of text documents displayed on a smart terminal's screen by "tracing" the screen with a finger as shown in Figure 3.

**Figure 3.** Conceptual diagram of text-to-speech using "finger tracing". **Figure 3.** Conceptual diagram of text-to-speech using "finger tracing".

The number of visually impaired people in Japan is approximately 316,000 (Ministry of Health, Labor and Welfare, 2013). Many of these visually impaired people are pursuing specialized education in physical therapy courses at national facilities and schools for the The number of visually impaired people in Japan is approximately 316,000 (Ministry of Health, Labor and Welfare, 2013). Many of these visually impaired people are pursuing specialized education in physical therapy courses at national facilities and schools for the

blind, to become professionally self-sufficient by obtaining national qualifications such as acupuncture and moxibustion therapists. However, only 9.2% of the blind use braille, and 50% of the blind do not use writing instruments. The use of text-to-speech software, which

visually impaired; however, the PC usage rate for general braille users is only 10.7%, and the rate for the visually impaired remains around 5% (about 15,000 people). In addition, the rapid spread of smart terminals in recent years has created a new digital divide for the visually impaired, as they are unable to use soft key operations on the screen. To improve these problems, we thought that we could provide employment opportunities for the vis-

From our previous research, we have developed a pen input system for the visually impaired that allows them to input Japanese into a computer using the same method as their usual writing (Fundamental Research (C) Representative: Kiyota: FY 2007-2008). In our previous research, we have commercialized a pen-based Japanese character input system for Japanese document processing. This research is an effort to demonstrate a means of obtaining text information at high speed using a digital terminal in order to solve the

For the development of the "Touch Talker" system, we used a development environment called Xcode (Apple's Integrated Development Environment (IDE) for software development). We also used Swift, a programming language provided by Apple for developing iOS applications. As a voice output for the function of reading out text and PDF documents from a computer, we used a speech synthesizer called "VoiceOver" (Apple's proprietary VoiceOver technology), which is compatible with iOS. When importing text documents, the Japanese morphological analysis software "MeCab" was used for pre-processing separate sentences into parts of speech and to convert sentences around instruc-

ually impaired by making the effective use of smart terminals possible for them.

digital divide for the visually impaired.

tions into word units for tracing and reading.

*3.2. Development Environment* 

blind, to become professionally self-sufficient by obtaining national qualifications such as acupuncture and moxibustion therapists. However, only 9.2% of the blind use braille, and 50% of the blind do not use writing instruments. The use of text-to-speech software, which reads out text information on the screen, has improved the PC usage environment for the visually impaired; however, the PC usage rate for general braille users is only 10.7%, and the rate for the visually impaired remains around 5% (about 15,000 people). In addition, the rapid spread of smart terminals in recent years has created a new digital divide for the visually impaired, as they are unable to use soft key operations on the screen. To improve these problems, we thought that we could provide employment opportunities for the visually impaired by making the effective use of smart terminals possible for them.

From our previous research, we have developed a pen input system for the visually impaired that allows them to input Japanese into a computer using the same method as their usual writing (Fundamental Research (C) Representative: Kiyota: FY 2007–2008). In our previous research, we have commercialized a pen-based Japanese character input system for Japanese document processing. This research is an effort to demonstrate a means of obtaining text information at high speed using a digital terminal in order to solve the digital divide for the visually impaired.

#### *3.2. Development Environment*

For the development of the "Touch Talker" system, we used a development environment called Xcode (Apple's Integrated Development Environment (IDE) for software development). We also used Swift, a programming language provided by Apple for developing iOS applications. As a voice output for the function of reading out text and PDF documents from a computer, we used a speech synthesizer called "VoiceOver" (Apple's proprietary VoiceOver technology), which is compatible with iOS. When importing text documents, the Japanese morphological analysis software "MeCab" was used for pre-processing separate sentences into parts of speech and to convert sentences around instructions into word units for tracing and reading.


#### *3.3. Specifications*

Using a smart device, start the "Trace Reading" application. Trace the file list in the text folder from top to bottom, and VoiceOver will read out the individual file titles. Next, open the text document that you want to read by selecting the text file you want to read. Begin reading by tracing from above with your index finger. The text document you want to read can also be added or deleted as a text file or PDF file from "iTunes" or "folder". Table 1 and Figure 4 show the list of settings and screenshots for "Touch Talker". In the setting mode, the user can set each language (Japanese, English, German, Italian, French), reading speed, pitch, reading units, (sentences, clauses), line spacing, font size, etc., according to the user's preference. Depending on the size of the user's hands and their senses, there are different settings for font size and line spacing according to their preference. This system uses text-to-speech software to read out the settings and allow the user to change the settings, but in this case, we used the default settings that the subjects had practiced with before the experiment, and set them all to the same settings. Here, I used a font size of 28 points, a line spacing of 5 points, and a reading speed of 5 (middle speed). Figure 5 shows a screenshot of the loaded text document. After booting up the system, the user uses their index finger to trace down from the top of the screen to read the word closest to their fingertip. When the user's index finger reaches the bottom of the screen, a double-tap with the index and middle fingers brings up the next page (this is a type of gesture control).

**Table 1.** Settings and contents for "Touch Talker".


**Figure 4.** Screenshot of settings and contents in "Touch Talker". **Figure 4.** Screenshot of settings and contents in "Touch Talker". The example screen in Figure 4 shows the Touch Talker settings menu. From the top, it shows "Language Selection", "Reading Speed", "Pitch (character spacing)", "Reading Unit–Sentence or Section", and "Font Size". In addition, the rectangle at the bottom is an example of a sentence display.

**Figure 5.** View the text document of a file in Japanese**.** 

We conducted an evaluation experiment to verify the effectiveness of our proposed

system with word-reading function by finger tracing. The experiment required two groups of six visually impaired people and six blindfolded healthy people to participate. Table 2 shows the profiles of the six visually impaired people, and Table 3 shows the profiles of six healthy, blindfolded people (NIT, Kumamoto college students). As a result, as

**4. Functionality Evaluation Experiment** 

*4.1. Experiment Participants* 

**Figure 4.** Screenshot of settings and contents in "Touch Talker".

**Figure 5.** View the text document of a file in Japanese**. Figure 5.** View the text document of a file in Japanese. The Japanese text in Figure 5 is a part of a novel called "Kappa" written by a Japanese author named Ryunosuke Akutagawa.

#### **4. Functionality Evaluation Experiment 4. Functionality Evaluation Experiment**

#### *4.1. Experiment Participants 4.1. Experiment Participants*

We conducted an evaluation experiment to verify the effectiveness of our proposed system with word-reading function by finger tracing. The experiment required two groups of six visually impaired people and six blindfolded healthy people to participate. Table 2 shows the profiles of the six visually impaired people, and Table 3 shows the profiles of six healthy, blindfolded people (NIT, Kumamoto college students). As a result, as We conducted an evaluation experiment to verify the effectiveness of our proposed system with word-reading function by finger tracing. The experiment required two groups of six visually impaired people and six blindfolded healthy people to participate. Table 2 shows the profiles of the six visually impaired people, and Table 3 shows the profiles of six healthy, blindfolded people (NIT, Kumamoto college students). As a result, as part of their social implementation education, these students were allowed to experience the difficulties of people with disabilities. Table 2 shows the participants' visual impairment status as well as the digital devices they typically use. Furthermore, we asked subjects of a young age to participate to avoid the effects of limb disability.




**Table 3.** Profiles of the six blindfolded, healthy subjects that participated in the experiment.

In general, visually impaired people obtain information by using digital IC recorders (hereinafter referred to as "IC.R") for voice playback, and by using text-to-speech software on PCs and tablet terminals.

#### *4.2. Comparison Experiment with the Information Search Task*

For the evaluation experiment, we prepared a task of retrieving arbitrary information from a huge amount of information. As shown in Figure 6, we also created a text file with 190 countries and their capitals, one on each line. The prepared text file was then loaded into the developed "Touch Talker" system, as shown in Figure 6. In this experiment, the specifications of the initial system settings in Table 1 were set as follows:

/Language: Japanese /Reading speed: 4 steps (middle speed) /Pitch 1.0 (character spacing) /Reading units: phrase /Line space: 14 points /Font size: 14 points *Sensors* **2021**, *21*, x FOR PEER REVIEW 9 of 14


**Figure 6.** A part of the text file (6 pages) of the list of names and capitals of the countries of the world (190 countries) was prepared for the evaluation experiment. **Figure 6.** A part of the text file (6 pages) of the list of names and capitals of the countries of the world (190 countries) was prepared for the evaluation experiment.

*4.3. Information Search Task*  We obtained informed consent from each subject before they participated in the experiment. In the evaluation experiment, the experimenter first explained how to operate With the speech settings described above, the names of all 190 countries and their capitals were read out loud using the VoiceOver function of the iPad from the top to the end of the text and recorded on a digital IC recorder.

the digital IC.R (Digital IC recorder) and the iPad. Then, the subjects were asked to prac-

that lists the world's countries (190 countries) and capitals alphabetically at one-line in-

To test the finger tracing function's effectiveness, we first loaded text data into "Touch Talker" and recorded the same voice data read out at the same speed on a digital recording device. The subjects were instructed to locate the capital of the country using

Figure 7 shows the flow of capital search by tracing with your index finger. Using a smart device, launch the "Trace and Read" application. When the user traces their index finger down from the top of the iPad screen, the name of the country at their fingertip will be read out by the iPad's VoiceOver function. Then, moving the user's finger to the right

For each question, the experimenter read out the target country. Then, the measurement was started and the time taken by the subject to find the capital of the country was

1st time: Measurement of the mean search time for a total of 10 countries

**Question 1**: Find the names of the capitals of the following countries. **Task 1** "Touch Talker" → IC.R (Digital IC recorder) (5 countries)

**Task 2** IC.R (Digital IC recorder) → "Touch Talker" (5 countries)

will read out the capital of the country, as shown in Figure 7.

/Tasks for the evaluation experiment

(1). United Arab Emirates → **Abu Dhabi**

(4). Kingdom of Saudi Arabia → **Riyadh**

(2). Canada → **Ottawa** (3). Grenada → **St. George's**

(5). Czech Republic → **Prague**

(6). State of Israel → **Jerusalem** (7). Republic of Estonia → **Tallinn** (8). Republic of Cameroon → **Yaounde** (9). Solomon Islands → **Honiara**

tice the operation method for about 10 min.

tervals, as shown in Figure 6.

**[First Time Trial]** 

measured.

#### *4.3. Information Search Task* 2nd time: Measurement of the mean search time for a total of 10 countries.

**[Second Time Trial]** 

We obtained informed consent from each subject before they participated in the experiment. In the evaluation experiment, the experimenter first explained how to operate the digital IC.R (Digital IC recorder) and the iPad. Then, the subjects were asked to practice the operation method for about 10 min. **Question 2**: Find the name of the capital of the following countries. **Task 3** ("Touch Talker" only) (1). United Arab Emirates → **Abu Dhabi** (2). Canada → **Ottawa**

The second trial was completed the same as the first time, with the experimenter reading out the target country. Then, the measurement was started and the time taken by

*Sensors* **2021**, *21*, x FOR PEER REVIEW 10 of 14

the subject to find the capital of the country was measured.

(10). Republic of Chile → **Santiago**

To test the finger tracing function's effectiveness, we first loaded text data into "Touch Talker" and recorded the same voice data read out at the same speed on a digital recording device. The subjects were instructed to locate the capital of the country using "Touch Talker" and the IC.R. In this study, we created a six-page textbook for evaluation that lists the world's countries (190 countries) and capitals alphabetically at one-line intervals, as shown in Figure 6. (3). Grenada → **St. George's** (4). Kingdom of Saudi Arabia → **Riyadh** (5). Czech Republic → **Prague** (6). Israel → **Jerusalem** (7). Estonia → **Tallinn**

Figure 7 shows the flow of capital search by tracing with your index finger. Using a smart device, launch the "Trace and Read" application. When the user traces their index finger down from the top of the iPad screen, the name of the country at their fingertip will be read out by the iPad's VoiceOver function. Then, moving the user's finger to the right will read out the capital of the country, as shown in Figure 7. (8). Republic of Cameroon → **Yaounde** (9). Solomon Islands → **Honiara** (10). Chile → **Santiago**

**Figure 7.** List of capitals of 190 countries used in the evaluation experiments. **Figure 7.** List of capitals of 190 countries used in the evaluation experiments.

/Tasks for the evaluation experiment

#### *4.4. Experimental Results*  **[First Time Trial]**

Table 4 and Figure 8 show the experimental results of the average search time and standard deviation of IC.R and "Touch Talker" for the first trial of Task 1, Task 2, and the For each question, the experimenter read out the target country. Then, the measurement was started and the time taken by the subject to find the capital of the country was measured.

folded, healthy people when compared to the digital IC recorder. Furthermore, when compared to the first trials of Task 1 and Task 2, the average search time for both groups

second trial of Task 3 by six visually impaired people. Table 5 and Figure 9 show the experimental results of the mean search time and standard deviation for six blindfolded 1st time: Measurement of the mean search time for a total of 10 countries **Question 1**: Find the names of the capitals of the following countries. **Task 1** "Touch Talker" → IC.R (Digital IC recorder) (5 countries)


#### **[Second Time Trial]**

The second trial was completed the same as the first time, with the experimenter reading out the target country. Then, the measurement was started and the time taken by the subject to find the capital of the country was measured.

2nd time: Measurement of the mean search time for a total of 10 countries. **Question 2**: Find the name of the capital of the following countries. **Task 3** ("Touch Talker" only)


#### *4.4. Experimental Results*

Table 4 and Figure 8 show the experimental results of the average search time and standard deviation of IC.R and "Touch Talker" for the first trial of Task 1, Task 2, and the second trial of Task 3 by six visually impaired people. Table 5 and Figure 9 show the experimental results of the mean search time and standard deviation for six blindfolded technical college students simulating intermediate visual impairment. From the comparison of Tables 4 and 5, there was no significant difference in the mean search time using the digital IC recorder between the visually impaired group and the blindfolded group of students, which were recorded as 55.8 and 53.2 s, respectively (a p-value less than 0.01 s was considered statistically significant). The developed "Touch Talker", on the other hand, reduced the mean search time for both the visually impaired people and blindfolded, healthy people when compared to the digital IC recorder. Furthermore, when compared to the first trials of Task 1 and Task 2, the average search time for both groups of "Touch Talker" became shorter in Task 3, the second trial, as they became more familiar with the operation of the device. According to the results of this experiment, the standard deviation for each individual decreased.

**Table 4.** Results of an experiment comparing information retrieval between IC.R (Digital IC recorder) and "Touch Talker" by a visually impaired person.


deviation for each individual decreased.

deviation for each individual decreased.

corder) and "Touch Talker" by a visually impaired person.

*Sensors* **2021**, *21*, x FOR PEER REVIEW 11 of 14

corder) and "Touch Talker" by blindfolding a healthy person.

corder) and "Touch Talker" by a visually impaired person.

**Figure 8.** Results of an experiment comparing information retrieval between IC.R (Digital IC recorder) and "Touch Talker" by six visually impaired persons. **Figure 8.** Results of an experiment comparing information retrieval between IC.R (Digital IC recorder) and "Touch Talker" by six visually impaired persons. 40.3 40.0

\*\*:*p*<0.01

with the operation of the device. According to the results of this experiment, the standard

**Table 4.** Results of an experiment comparing information retrieval between IC.R (Digital IC re-

**Task Avg. (Sec.) SD (Sec.)** 

**Table 5.** Results of an experiment comparing information retrieval between IC.R (Digital IC re-

**Table 4.** Results of an experiment comparing information retrieval between IC.R (Digital IC re-

with the operation of the device. According to the results of this experiment, the standard

**Task Avg. (Sec.) SD (Sec.)** 

**Task Avg. (Sec.) SD (Sec.)** 

① IC.R (1st time) 55.8 16.5 ② Touch (1st time) 40.3 13.2 ③ Touch (2nd time) 35.9 13.9

① IC.R (1st time) 53.2 7.2 ② Touch (1st time) 30.5 7.1 ③ Touch (2nd time) 21.3 3.0

① IC.R (1st time) 55.8 16.5 ② Touch (1st time) 40.3 13.2 ③ Touch (2nd time) 35.9 13.9

80.0 **Table 5.** Results of an experiment comparing information retrieval between IC.R (Digital IC recorder) and "Touch Talker" by blindfolding a healthy person. 35.9 20.0


**Figure 9.** Results of an experiment comparing information retrieval between IC.R (digital IC recorder) and "Touch Talker" by blindfolding six healthy persons. **Figure 9.** Results of an experiment comparing information retrieval between IC.R (digital IC recorder) and "Touch Talker" by blindfolding six healthy persons.

#### **5. Discussion**

30.0

Although the subjects were relatively the same age in this evaluation experiment, the average search time of the healthy group of technical college students who usually use smartphones and PCs became faster in the second trial of Task 3. This is thought to be due to the user's familiarity with the operation of the iPad on which "Touch Talker" is installed. However, there was no difference in the retrieval time of the first trial for subject B in the visually impaired group when using either "Touch Talker" or the digital IC recorder.

Rather, the second trial of "Touch Talker" tended to take more time as shown in Figure 10. Subject B is a 30-year-old blind female, as shown in Table 2. She usually uses most of the information terminals, including a smartphone. Therefore, she is familiar with information terminals. In the questionnaire after the experiment, she commented on the operation of "Touch Talker" as follows: From her comments, we think that the main reason for the unresponsive state is due to the poor response of tapping on the iPad, and we would like to improve our next application.

"I couldn't tell whether my finger was on a letter, a space, or a blank line."

Although the subjects were relatively the same age in this evaluation experiment, the average search time of the healthy group of technical college students who usually use smartphones and PCs became faster in the second trial of Task 3. This is thought to be due to the user's familiarity with the operation of the iPad on which "Touch Talker" is installed. However, there was no difference in the retrieval time of the first trial for subject B in the visually impaired group when using either "Touch Talker" or the digital IC re-

Rather, the second trial of "Touch Talker" tended to take more time as shown in Figure 10. Subject B is a 30-year-old blind female, as shown in Table 2. She usually uses most of the information terminals, including a smartphone. Therefore, she is familiar with information terminals. In the questionnaire after the experiment, she commented on the op-

"In many cases, I could not read the text even if my finger was on the text, so I just

*Sensors* **2021**, *21*, x FOR PEER REVIEW 12 of 14

**5. Discussion** 

corder.

eration of "Touch Talker" as follows:

kept moving my finger."

**Figure 10.** Results of the capital search experiment for subject B with visual impairment. **Figure 10.** Results of the capital search experiment for subject B with visual impairment.

On the other hand, the participating technical college students said that operating the "I couldn't tell whether my finger was on a letter, a space, or a blank line."

tablet device while blindfolded was completely different from their normal work and that they had a hard time just operating the touch screen. From this evaluation experiment, it "In many cases, I could not read the text even if my finger was on the text, so I just kept moving my finger."

was found that even for technical college students who are used to operating tablets, when blindfolded, there was a tendency to operate the tablet in a similar way as the visually impaired. Furthermore, we were able to make the technical college students understand From her comments, we think that the main reason for the unresponsive state is due to the poor response of tapping on the iPad, and we would like to improve our next application.

the importance of considering the situation of the subject as an engineer when developing assistive devices, and we confirmed the effectiveness of social implementation education. **6. Conclusions**  In this paper, we described the necessity of the next-generation AT (extended-AT, eAT thereafter) that integrates AT (Assistive Technology) and digital technologies (AI, IoT, robotics, big data, mobility, etc.) for the realization of Society 5.0. Furthermore, we described GEAR 5.0, a project to train the next generation of engineers to support people On the other hand, the participating technical college students said that operating the tablet device while blindfolded was completely different from their normal work and that they had a hard time just operating the touch screen. From this evaluation experiment, it was found that even for technical college students who are used to operating tablets, when blindfolded, there was a tendency to operate the tablet in a similar way as the visually impaired. Furthermore, we were able to make the technical college students understand the importance of considering the situation of the subject as an engineer when developing assistive devices, and we confirmed the effectiveness of social implementation education.

#### with disabilities through technological measures, to expand employment opportunities **6. Conclusions**

for people with disabilities, and to improve the quality of life of patients and people with In this paper, we described the necessity of the next-generation AT (extended-AT, eAT thereafter) that integrates AT (Assistive Technology) and digital technologies (AI, IoT, robotics, big data, mobility, etc.) for the realization of Society 5.0. Furthermore, we described GEAR 5.0, a project to train the next generation of engineers to support people with disabilities through technological measures, to expand employment opportunities for people with disabilities, and to improve the quality of life of patients and people with disabilities. Next, we created "Touch Talker" to help visually impaired people read text using a digital terminal, and we described an evaluation experiment with visually impaired people conducted by technical college students using the prototype. We asked the technical college students to simulate visual impairment by performing the same task as the visually impaired while blindfolded in the evaluation experiment.

From the evaluation experiments, we confirmed the effectiveness of our proposed "Touch Talker", which reads out text traced with the index finger. However, we discovered some flaws in the application, such as the tap control function. We intend to improve the application and turn it into a commercial device that can be used by visually impaired people based on the information obtained from these evaluation experiments and questionnaires.

#### **7. Patents**

Part of this study has been granted Japan patent JP6391064B, "Audio output processing device, audio output processing program, and audio output processing method", Japan, 31 August 2018. Additionally, "Touch Talker" is registered as a trademark under JP6368537, 25 March 2021.

**Author Contributions:** Conseptualisation, K.K. and T.I.; methodology, K.K. and M.S.; resources, K.I.; data curation, K.K. and K.I.; writing—original draft preparation, K.K. and T.I.; review and editin, M.S. and K.I.; project administration, K.K. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by supported by Grant-in-Aid for Scientific Research, Grant-in-Aid for Scientific Research (B), 18H03563.

**Institutional Review Board Statement:** The evaluation experiment was conducted after this study was ethically reviewed and approved by the Ethics Review Committee for Research on Human Subjects at Kumamoto National College of Technology. In accordance with Article 9 of the Regulations of the Ethics Review Committee for Research Involving Human Subjects of NIT, Kumamoto College, the decision was made to approve the project.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Not Applicable.

**Conflicts of Interest:** The autors declare no conflict of interest.

#### **References**


## *Article* **Tracking and Classification of Head Movement for Augmentative and Alternative Communication Systems**

**Carlos Wellington P. Gonçalves <sup>1</sup> , Rogério A. Richa <sup>2</sup> and Antonio P. L. Bo 3,\* ,†**


**Abstract:** The use of assistive technologies can mitigate or reduce the challenges faced by individuals with motor disabilities to use computer systems. However, those who feature severe involuntary movements often have fewer options at hand. This work describes an application that can recognize the user's head using a conventional webcam, track its motion, model the desired functional movement, and recognize it to enable the use of a virtual keyboard. The proposed classifier features a flexible structure and may be personalized for different user need. Experimental results obtained with participants with no neurological disorders have shown that classifiers based on Hidden Markov Models provided similar or better performance than a classifier based on position threshold. However, motion segmentation and interpretation modules were sensitive to involuntary movements featured by participants with cerebral palsy that took part in the study.

**Keywords:** human–computer interface; human movement analysis; cerebral palsy; hidden Markov model; assistive technology

#### **1. Introduction**

The use of computers, tablets, and similar devices is generalized in society. By enabling people to expand their communication possibilities and improve productivity at work, such systems are increasingly part of our daily lives. Nevertheless, to operate conventional keyboards and mouses on computers and laptops, and touchscreen interfaces on smart mobile devices, finger dexterity and range of movement is often a requirement. Hence, millions of individuals worldwide, who lack the ability to control the upper limbs, may not fully experience the functions provided by these devices. The implications of this may be particularly important for children and young adults [1].

Solutions that allow the use of computer systems by a person with disability in order to communicate are referred to as Augmentative and Alternative Communication (AAC). Each individual will adapt himself and use a AAC device based on assessment of cognitive and physical condition. Mechanical switches, switch activated mouses, and sipand-puff devices are examples of popular systems that deliver input to human–computer interfaces (HCI) that control virtual keyboards and other applications providing means to communication (written or with synthesized speech). In addition to enable use of computer systems, these sensor modalities may also enable control of powered wheelchairs and smart environments, for instance [2]. Nonetheless, despite their ease-of-use, robustness, and reduced cost, mechanical switches present limitations, such as the need to re-position the sensor if the body posture changes, and the requirement of complex mounting structures fixed to the wheelchair [2].

**Citation:** Gonçalves, C.W.P.; Richa, R.A.; Bo, A.P.L. Tracking and Classification of Head Movement for Augmentative and Alternative Communication Systems. *Sensors* **2022**, *22*, 435. https://doi.org/ 10.3390/s22020435

Academic Editors: Kimiyasu Kiyota, Akira Shionoya and Mitsuteru Inoue

Received: 18 November 2021 Accepted: 23 December 2021 Published: 7 January 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

For those conditions in which the individual presents only head and facial functional movements, systems based on eye and head tracking are valuable alternatives, particularly if no markers are required. Among the markerless solutions already available for the user, some require specialized hardware in order to track eye movement, such the Tobii eye tracking systems (Tobii, Sweden). However, despite positive evaluation from parents [3] in a pilot study, they require the user to maintain firm sitting position and avoid gross head movements for suitable operation. Hence, for some individuals, such as those with dystonia, spasticity, or pathological tremor, these solutions are not functional. Furthermore, since most computer systems currently feature onboard cameras and thus enabling the design of an AAC device with no additional hardware, such AAC systems are costly when compared to other alternatives.

Concerning markerless systems that employ a conventional webcam, software is also available that enables head tracking and estimating the corresponding cursor movement. Indeed, systems that rely on cameras to enable HCI using head tracking feature additional advantages, mostly related to the wide availability of cameras in modern devices. However, they share similar limitations to eye tracking systems. Hence, solutions, such as the Headmouse (Universitat de Lleida, Spain) [4] require fine control of head movements. These limitations may likely persist even if novel sensing technologies (e.g., Kinect, as in [5]) or computer vision tracking tools (e.g., [6–8]) are used with this purpose.

Based on the literature, for users who feature involuntary movements or disability that interfere with head movement, there is no viable alternative to the best of authors knowledge of computer vision based systems enabling effective AAC. In this scenario, in this work we propose applying head tracking using computer vision alongside gesture recognition methods for enabling robust detection of specific head movements that are then used for AAC. In particular, two user intent detection approaches are proposed and compared experimentally in this paper: while one relies on simple position thresholds, the other is based on Markov modeling to characterize and recognize appropriate movements. Experimental studies have been conducted in two groups, namely participants with disabilities and participants presenting no neurological impairments. The experimental protocol has been designed to enable evaluating the overall feasibility of the method, as well as its performance, particularly in terms of minimizing false positives.

This paper is organized as follows. Section 2 presents the method, including information on participants and experimental protocol, as well as a detailed description of the proposed HCI system based on computer vision and intention classification. Experimental evaluation is presented in Section 3, including data from individuals with cerebral palsy (CP) and participants with no neurological disorder. These results are discussed in Section 4. Finally, we draw conclusions and discuss future works in Section 5.

#### **2. Materials and Methods**

The AAC system proposed in this paper relies on the coordinated performance of distinct methods, which are described in this Section. Information regarding recruitment and experimental protocol are also provided here.

#### *2.1. Subjects and Protocol*

Six subjects in total were recruited for this study, four participants with no neurological disorder (group A, three males) and two participants with cerebral palsy (group B, two males). The research was approved by the Ethics Committee at the SARAH Network of Rehabilitation Hospitals and registered in Brazilian National Committee in Research Ethics (CONEP) with CAAE number 15055513.6.0000.0022, in accordance with the Helsinki Declaration. All volunteers signed an informed consent form.

Participants from group B who participated in the study presented heterogeneous features in terms of movement disorder. Although participant B1 featured involuntary movement mostly described as choreodystonia, participant B2 presented mostly choreoathetosis. Both participants had previous experience in using mechanical switches for AAC. Group B participants were explicitly recruited to evaluate system performance when users present intense dystonia. Group A participants were selected to enable assessing system performance when users present no involuntary movement, a condition that represents several targeted clinical populations.

The study consisted of trials in which participants were given explanatory information on the protocol and then seated comfortably in front of a computer screen with a webcam (640 × 480 pixels resolution). A mechanical switch was positioned next to the participant's head, in order to control a virtual keyboard with scanning control scheme. Each participant then performed head movements that triggered the mechanical switch in order to write a predefined sentence. Sentences were defined such that the expected duration to type it was similar for participants in groups A and B. Participants were instructed to focus on head movements, particularly side-bending and flexion, but trunk movements were also allowed. Group A participants executed complementary movements guided by a orange sphere randomly placed next to the participant's head by the researcher conducting the experiment. This additional step was included to enable evaluating head detection and tracking performance in the presence of head motion that should not generate writing (i.e., invalid movements). All movement was registered, as well the reference valid movements obtained from the mechanical switch activation, for further analyzing the proposed HCI system.

#### *2.2. Head Detection and Tracking*

Face detection is the first stage within the head tracking method employed in this work. Its implementation is obtained using a Viola–Jones classifier trained to vertical face recognition [9].

Among the candidates for the user's face, only the largest rectangle is analyzed. We assume that the user is positioned in front of the webcam and thus he is the person closer to the camera. This approach allows multiple individuals to be present in the image, with no compromise to function.

At the end of this stage, both the rectangle dimensions that delimits the user's face, *sh*<sup>0</sup> = (*wh*<sup>0</sup> , *hh*<sup>0</sup> ), and the corresponding position of this rectangle, *ph*<sup>0</sup> = (*xh*<sup>0</sup> , *yh*<sup>0</sup> ), are obtained. For each different position of the user in the image, either in the plane (*x*, *y*) or in relation to distance from the camera, different *sh*<sup>0</sup> and *ph*<sup>0</sup> may be obtained. However, this variability is unsuitable for tracking. For this reason, two additional operation are performed on the reference image: scaling and centering. The scale factor is calculated to generate a region of interest whose width is three times *wh*<sup>0</sup> . Both operations are then performed for every frame of the acquired video.

Head tracking is then implemented using primarily the mean shift algorithm, particularly employing the posterior probability measure as a similarity measure [10] modified to work with colored images coded in HSV domain. Among the implementation choices in this stage, the rectangle dimensions are reduced in 20% to partially remove the user's hair and the search region is defined as 50% larger than the original scaled rectangle.

Furthermore, since this estimate will often be affected by noise and other error sources related to lighting and inherent limitations of the method, a Kalman filter is used for additional filtering. In this case, a constant-velocity model is employed, and the corresponding covariance matrices were selected empirically to improve motion segmentation performance. Figure 1 illustrates both reference detected position and estimated tracked position.

#### *2.3. Motion Segmentation and Classification*

Motion segmentation is used with the goal of reducing the amount of data that require classification and also to determine the segments candidates that represent functional movements. Our approach was conceived based on the methods proposed in [11,12].

Considering head velocity estimates provided by the Kalman filter, segmentation is performed based on a zero velocity cross approach, where the following quantity is calculated to combine the effects of *x* and *y* trajectories:

$$
\boldsymbol{\varepsilon} = \dot{\mathfrak{x}}\_h^2 + \dot{\mathfrak{y}}\_h^2. \tag{1}
$$

In this work, only reaching movements followed by return to the reference position are considered as potential input to the system. Hence, we have implemented a finite-state machine to detect, based on *e*, one peak detected between two valleys, which then represents the segmented movement.

Two different methods were used to classify a movement as functional (also referred to as valid in this context). Although one method is based on a *x*-axis position threshold, the other is based on Markov modeling, which potentially enables more complex and accurate modeling of the functional movement.

For implementing the first method, i.e., detecting the functional reaching movement based on a horizontal threshold *x<sup>t</sup>* , motion segmentation is not required. *x<sup>t</sup>* , which is defined in relation to the reference head position *xh*<sup>0</sup> , is calculated during an initial calibration phase based on maximum displacements when performing the reaching movement, max (*x<sup>h</sup>* ). In this works, we have used the average when performing 10 reaching movements. The threshold *x<sup>t</sup>* is defined as 20 *pixels* closer to the reference position, as illustrated in Figure 1. Since the user may perform reaching movements towards any direction, both positive and negative *x<sup>t</sup>* are possible.

**Figure 1.** Illustration of head tracking and threshold-based movement classification. The green rectangle represents the reference position, denoted by *sh*<sup>0</sup> and *ph*<sup>0</sup> , while the red rectangle illustrates the current estimated position. (**a**) illustrates the initial position, (**b**) the definition of a position threshold *x<sup>t</sup>* (yellow line) based on max (*x<sup>h</sup>* ) (red line), and (**c**) the user moving his head to control the virtual keyboard.

The second method is based on the understanding that the reaching movements are composed of stages. The approach firstly involves obtaining the Hidden Markov Model (HMM) using segments representing valid movements. Based on these segments, clustering using k-means was employed to obtain the statistics of each stage that compose the functional movement sequence. Four-state HMMs were chosen to represent the desired movements, and hence the number of possible clusters was set to four. Next, the probability density functions of each cluster were used as an initial estimate of the state observation matrix **B**. Finally, the corresponding transition matrix **A** and the initial distribution *π* are estimated using the Baum-Welch algorithm. This procedure was applied for three different feature vectors evaluated in this work: head position (HMM-P), velocity (HMM-V), or both (HMM-PV).

Identifying valid movements using the HMM-based method is primarily accomplished using the log-likelihood, *l<sup>s</sup>* . Furthermore, early trials indicated influence of the segment length, *n<sup>s</sup>* . This joint measure *ψ* = [*l<sup>s</sup> n<sup>s</sup>* ] *T* is the basis for detecting valid segments. The normal distribution of *ψ* obtained using valid segments is calculated, and the corresponding Mahalanobis distance to *ψ* given by a measured segment is used for classification.

#### *2.4. Data Analysis*

The main outcome measure in this work was obtained using the receiver operating characteristic curve, or ROC curve. In a ROC curve, the corresponding threshold is varied to generate the corresponding true positive rate (TPR) against the false positive rate (FPR) , which were both calculated based on the ground truth (i.e., activation of the mechanical switch). In the HMM-based method, the threshold refers to confidence level employed on the null hypothesis testing using the Mahalanobis distance. Finally, the area under the curve (AUC) is used to provide a quantitative measure of each classifier.

Furthermore, regarding the HMM-based method, evaluation of preliminary results generated in this study was performed using 3-fold cross-validation, where a third of collected data were used for validation at each iteration.

#### **3. Results**

Table 1 lists the main result in this proof-of-concept study. The AUC calculated for every participant using both movement intent detection methods are listed. For the HMM-based method, results obtained for each evaluated feature vector are included.


**Table 1.** AUC for each participant and every user intent detection method.

Additionally, Figures 2–5 illustrate intermediate signals and ROC curves generated with the experimental data. In particular, Figure 2 serves to illustrate the performance obtained using the head tracking method proposed in this work.

Figures 3 and 4 depict, respectively, the motion segmentation method and the thresholdbased classification method. In Figure 3, the detection of each segment (described in Section 2.3) is illustrated by the segment counter, which is incremented once the finitestate machine detects a peak between two valleys in *e*. Regarding the threshold-based method, Figure 4a illustrates cases where the threshold-based method performs satisfactorily, while in Figure 4b one false positive occurs (first positive movement) among a total of six detected segments.

Considering the Table 1 and Figure 5, while the performance between HMM-based and threshold-based classifiers may seem similar at first, in some cases the HMM (and in particular HMM-PV and HMM-V) performance in terms of low FPR stands out an important feature.

Table 1 also shows similar AUC was obtained for participants A1, A2, and A3. For participant A3, the lower performance by HMM-P possibly is due to the lack of velocitydependent features. The corresponding transition matrices have shown that HMM-PV and HMM-V often generate classifiers with higher capacity of generalization.

For users A4, B1, and B2, issues were observed in the generation of training segments with a substantial number of samples. For A4 the main problem was possibly the proximity of the mechanical switch to the user's face. For B1 and B2, the presence of involuntary movements prevented the acquisition of longer segments, deteriorating the performance of the HMM-based method for all features (as illustrated in Figure 5b. Nevertheless, it may also be observed that the performance of the threshold-based classifier also deteriorated due to variations in range of valid motions. For participants in group B, this often occurred because the switch was often pushed with intense force, shifting its position. Since participant B2 featured a more controlled movement, a better performance was obtained for all classifiers when compared to participant B1.

**Figure 2.** Illustration of head tracking performance, both in terms of position (**top**) and velocity (**middle**). Corresponding activation of mechanical switch is also depicted (**bottom**).

**Figure 3.** Example of motion segmentation based on *e*, including corresponding segment counter.

**Figure 4.** Sample classification results when using the threshold-based method. In (**a**) four successive reaching movement are successfully detected, while in (**b**) false positives due to involuntary movements are depicted.

**Figure 5.** ROC curves obtained for participants (**a**) A2 and (**b**) B1. All classifiers are compared using 3-fold cross-validation.

#### **4. Discussion**

To the best of the authors knowledge, this paper presents a first attempt to implement an AAC system using head tracking and user intent recognition based on HMM. Nevertheless, while the proof-of-concept experiments helped us to evaluate the overall usability of the system, some of the potential limitations became evident.

The group of people who activate mechanical switches employing head movements to use a computer is the target population in this study, particularly the subgroup that also features involuntary head movements. This is also an important advantage of the approach proposed in this paper, since little training is expected to use the system, since similar movements as those required to operate the switch are employed as input in this work. Another fundamental feature of our solution regards the low technical requirements. Indeed, the two hardware elements required in the system are a simple webcam to acquire images of the user and a basic computer to run the software application that interprets the head movements to enable control of virtual keyboard.

One specific challenge in systems that enable control of virtual keyboards, particularly when sweep scanning mode is used, refers to false positives (e.g., selecting a wrong character when using a virtual keyboard). Indeed, often a high number of additional steps need to be performed by the user in virtual keyboards or general AAC software whenever a false positive occurs. This issue becomes a larger concern if we consider that voluntary activation often demands a high level of user concentration, as well as being physically demanding. These are the fundamental reasons of our concern to reduce the number of false positives generated by the system.

In this work, evaluation of performance was based on tests where participants were instructed to type predefined sentences using mechanical switches activated by their head movements. Video was also recorded, which enabled comparison between the various methods based on computer vision proposed in this work with the assistive device that is commonly used by this population (i.e., mechanical switches). Nevertheless, it also means that we have not formally evaluated each individual component of our algorithm, such as the tracking and segmentation routines. Although preliminary data indicating satisfactory tracking performance is presented in Section 3, analysis of segmentation performance is indirect, based on final classification results.

Regarding the classification in this work, we have proposed two different user intention detection systems. The first is based on a head position threshold, while the second is based on Markov models. Both systems are trained from movements labeled as valid, which are identified when the user is writing a sentence using a mechanical switch and an on-screen keyboard using sweep scanning mode. The threshold-based method draws clear inspiration on mechanical switches. The modeling of functional movement is performed based on the expected range of motion alone. However, involuntary movements of the same range are treated in the same way (e.g., Figure 4b), effectively limiting the usability of the method for certain populations who present levels of involuntary movement that overlap the assigned threshold.

Although the threshold-based method may rely on the horizontal estimate of head position only, this is not the same for the HMM-based method. Clearly, computer vision tracking systems can provide further information in addition to the *x*-axis position of an object. Using the Meanshift algorithm based on the PPM similarity measure, we estimated the user head position, estimating velocity and position in two dimensions. Although other techniques could be used to enable head tracking, often providing other variables, such as head orientation, in this work we hypothesized that a simple but robust tracking would provide sufficient performance.

The application of a Markov chain proved to be extremely flexible, while simultaneously requiring few initialization parameters. The main parameters defined were the number of states, i.e., four, and the use of observable data sampled from a continuous distribution. The model training employs data from valid movements alone, and the classification of a given movement by the model is performed using a continuous variable that represents the probability of this movement being represented by the chain.

For participants who do not experience involuntary movement (i.e., group A), the performance of the HMM classifiers surpassed in some cases the threshold-based classifier. That may have occurred due to the incorporation of an actual model of the valid movement, which is the basis for disregarding involuntary movements of similar amplitude. However, participants from group A who featured very small functional movements generated valid segments with few samples (e.g., A4). In these cases, the resulting model often presented more than one state that do not allow transition to others states.

For participants who present involuntary movements (i.e., group B), the training of the Markov model was affected by limitations in the segmentation algorithm employed in this work. Functional movements were split due to the sudden change in speed, such as depicted in Figure 4, generating once again segments of valid movement with few samples. The higher number of invalid segments generated by involuntary movements finally increased the number of false positives obtained for these data. We can infer that, based on these results, this use of this specific classifier for this population is unfeasible.

Based on these factors, one possible conclusion is that in this work we could not fully assess the hypothesis that the higher-dimensional representation of movement in HMMbased classifiers (i.e., displacement and speed in *x* and *y*, in comparison to displacement in *x* only) generates improved AAC systems. Involuntary movement not only compromised the performance of the threshold-based method, but also the HMM-based method, demon-

strating how the modeling adopted in this work may have been insufficient to detect the volitional movement. Indeed, users with disabilities featured head movement patterns that were barely perceptible to the naked eye, which also lead us to infer a higher-dimensional model might be required to provide correct classification.

In order to minimize the issues related to the segmentation performance, we understand that solutions that allow the creation of segments with a larger number of samples may enhance the classification accuracy of HMM-based methods, particularly decreasing the rate of false positives. Nevertheless, given the issues discussed here, further methods to take into account the effect involuntary movements should be considered to enable robust replacement of AAC based on mechanical switches, particularly for those individuals with severe movement disorders. One of such methods would involve modeling the involuntary movement itself, such as in [13,14], while another approach would involve providing a more comprehensive framework for defining head movements to operate the HCI, such as in [15].

Lastly, while a writing task was used in this work to evaluate the feasibility of the proposed HCI system, in our research group we are interested in evaluating similar methods in other AT scenarios. Some examples applications involve human–robot interaction [16,17] and wearable systems [18].

#### **5. Conclusions**

The availability of easy-to-use AAC systems based on tools from computer vision may produce a high impact on the quality of life of individual with severe motor disabilities. Nevertheless, alternatives are scarce for those who feature intense dystonia, a common manifestation on children and young adults with CP. In this work, we have presented a self-contained method developed using open-source libraries that enables tracking head movements and detecting user's intent, which may then be used to control a virtual keyboard or other system. An experimental proof-of-concept study was conducted involving participants with CP and subjects with no neurological disorders. The obtained results have indicated satisfactory outcome for both techniques compared in this paper, whereas the performance was reduced whenever intense involuntary movement occurred.

**Author Contributions:** Conceptualization, C.W.P.G., R.A.R., A.P.L.B.; methodology, C.W.P.G., R.A.R., A.P.L.B.; software, C.W.P.G.; data curation, C.W.P.G., A.P.L.B.; writing—original draft preparation, C.W.P.G., A.P.L.B.; writing—review and editing, C.W.P.G., R.A.R., A.P.L.B. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board of *Associação* das Pioneiras Sociais-DF/Rede SARAH (CAAE 15055513.6.0000.0022).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflicts of interest.

#### **Abbreviations**

The following abbreviations are used in this manuscript:


#### **References**


MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Sensors* Editorial Office E-mail: sensors@mdpi.com www.mdpi.com/journal/sensors

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34

www.mdpi.com ISBN 978-3-0365-6412-8