Human–Computer Interaction for Intelligent Systems

1. Introduction

The further development of human–computer interaction applications is still in great demand as users expect more natural interactions. For example, speech communication in many languages is expected as a basic feature for intelligent systems, such as robotic systems, autonomous vehicles, or virtual assistants. For this Special Issue, we invited submissions from researchers addressing the unique opportunities and challenges associated with human–computer interaction with intelligent systems. We encouraged authors to submit reports describing systems built for different languages and multilingual systems. We also invited submissions from researchers studying the linguistic, emotional, prosodic, and dialogue aspects of speech communication. We proposed to have a dialogue about other input and output modalities, including multimodal systems, fusion/fission algorithms, and deep learning methods. We encouraged the authors to report in detail the state-of-art results, and provide useful reviews and data used to build such systems to support development in those areas. The rapidly growing domain of virtual reality applications is of interest both as an application domain in which new interfaces and interaction methods are needed and as a potential testbed for evaluating speech and other interface modalities.

2. Short Presentation of the Papers

Every high-quality research started using a deep state-of-the-art review. We are proud to present great reviews in our collection about speech emotions [1], automatic spelling correction [2] and the usage of art in virtual reality [3].

2.1. Review Papers

Lieskovská et al. [1] presented a review of the recent development in speech emotion recognition and also examines the impact of various attention mechanisms on speech emotion recognition performance. Overall comparison of the systems was performed on a widely used IEMOCAP [4] benchmark database.

Hládek et al. [2] created a survey of automatic spelling correction algorithms. It follows from the previous work by Kukich [5] from 1992. It covers almost 20 years of research conducted since this paper. The article proposes a theoretical framework, overview of the approaches, benchmarks, and evaluation methods. It gives great insight for researchers after the last comprehensive survey because it is the first comprehensive survey paper about this topic after a long period.

Aldridge and Bethel [3] conducted an assessment of how art is being used in virtual reality (VR), and the feasibility of brain injury patients to participate in virtual art therapy was investigated. Studies included in this review highlight the importance of the artistic subject matter, sensory stimulation, and measurable performance outcomes for assessing the effect art therapy has on motor impairment in VR.

2.2. Research Papers

Stark et al. [6] described design and implementation of a new method for control and monitoring of mechatronic systems connected to the IoT network using a selected segment of extended reality to create an innovative form of the human–machine interaction. In the proposed solution, modern detection and recognition methods of 3D objects in augmented reality are used instead of conventional methods of control and monitoring of mechatronic IoT systems based on scanning QR codes.

Machová et al. [7] focus on increasing the effectiveness of the lexicon-based sentiment analysis. Within the research, two lexicons were built: the first was a big, domain-dependent lexicon created by translating and merging several existing dictionaries, and the second was a small, domain-dependent lexicon since it contained only words with the same meaning in different domains. These lexicons were labeled by assigning a degree of polarity to each word in the lexicon using Particle Swarm Optimization methods. The article also contains the results of experiments with the distribution of polarity values for different labeling techniques. The created lexicons were used in a new approach to sentiment analysis and evaluated. Sometimes, when the lexicon does not contain the words used in an analyzed text, the lexicon-based sentiment analysis itself fails. For such cases, it was supplemented with a machine learning model for sentiment analysis. This hybrid approach achieved very good results.

Szabóová et al. [8] paper is from the field of the analysis of emotions from a text that was obtained from a dialogue between a human and a robot and thus combines the field of sentiment analysis with HRI (human–robot interaction). Information about the emotional state of the person the robot is interacting with can help the robot choose the most appropriate response. Both a lexicon-based approach and machine learning were used for the emotion recognition (Naïve Bayes (multi-nomial, Bernoulli, and Gaussian), Support Vector Machine, and feed-forward neural network using various data representations, such as Bag-of-Words, TF-IDF, and sentence embeddings (ConceptNet Numberbatch)). The result of the experiments was an ensemble classifier consisting of the nine best models for each emotion. The model was demonstrated in four different scenarios with the humanoid robot NAO. Results concluded, that the best scenario for human acceptance is the one with emotional classification accompanied by emotional movements of the robot.

Shao et al. [9] presented the classification of dual-arm robot operator’s mental workload by using the heart rate variability (HRV) signal. Average classification accuracy of 98.77% was obtained using the K-Nearest Neighbor (KNN) method.

Ondáš et al. [10] introduced a novel pediatric audiometry application for hearing detection in the home environment. Conditioned play audiometry principles were adopted to create a speech audiometry application, where children help the virtual robot Thomas assign words to pictures. Several game scenarios together with the setting condition issues were created, tested, and discussed.

Agarwal et al. [11] focuses on designing a grammar detection system that understands both structural and contextual information of sentences for validating whether the English sentences are grammatically correct. The paper proposes a new Lex-Pos sequencing approach that contains both information, linguistic, as well as syntactic, of a sentence. Long Short-Term Memory (LSTM) neural network architecture has been employed to build the grammar classifier. The study conducts nine experiments to validate the strength of the Lex-Pos sequences. The results showed that the Lex-Pos-based models are observed as superior in giving more accurate predictions and they are more stable.

Trnka et al. [12] depicted a system for predicting the values of Activation and Valence (AV) directly from the sound of emotional speech utterances without the use of its semantic content or any other additional information. The system uses X-vectors to represent sound characteristics of the utterance and a Support Vector Regressor for the estimation of the AV values. The aim of the work was to test whether in each unseen database the predicted values of Valence and Activation will place emotion-tagged utterances in the AV space in accordance with expectations based on Russell’s circumplex model of affective space.

Gondi and Pratap [13] from Facebook AI Research presented an innovative performance evaluation of offline speech recognition on Raspberry Pi CPU compared to Jetson Nano GPU. It was shown that after PyTorch mobile optimization and quantization, the models can achieve real-time inference on the Raspberry Pi CPU with a small degradation to word error rate. On the other hand, the Jetson Nano GPU has inference latency three to five times better, compared to Raspberry Pi.

Seo and Kim [14] presented a self-attentive multi-layer aggregation with feature recalibration and deep length normalization for a text-independent speaker verification system. The ResNet with the scaled channel width and layer depth was used to reduce the number of model parameters as a baseline. A self-attention mechanism was applied to perform multi-layer aggregation with dropout regularizations and batch normalizations. Further, deep-length normalization was used on a recalibrated feature in the training process. Experimental results using the VoxCeleb1 [15] evaluation dataset showed that the performance of the proposed methods was comparable to that of state-of-the-art models.

Bačíková et al. [16] used the term domain usability (DU) to describe the aspects of the user interface related to the terminology and domain. A new method called ADUE (Automatic Domain Usability Evaluation) for the automated evaluation of selected DU properties on existing user interfaces was introduced. The authors executed ADUE on several real-world Java applications and report their findings.

Lin et al. [17] developed posting recommendation systems (RSs) to support users in composing reasonable posts and receiving effective answers. The posting RSs were evaluated by a user study containing 27 participants and three tasks to examine if users engaged more in the question-generation process. The results show that the proposed mechanism enables the production of question posts with better understanding, which leads experts to devote more attention to answering their questions.

Jinsakul et al. [18] presented an innovative approach to improve Thailand’s government’s systems to include handicraft products with a 3D display option for smartphones. The 1775 participants’ evaluation results in this study proved that the proposed 3D handicraft product application affected users by attracting their attention towards them.