Search Results (8)

Large language models (LLMs) exhibit strong general conversational capabilities, yet their deployment in domain-specific service environments such as hospitality remains limited by the absence of emotionally grounded datasets and validated end-to-end system architectures. This paper presents HOSPIT-LLM, an EU-funded euROBIN Technology Exchange Program pilot, as a complete, integrated pilot pipeline for human-centered conversational AI in hotel reception scenarios. We deploy a multimodal hotel-terminal assistant in a real hotel reception, capturing synchronized dual-camera video and audio to collect authentic guest–staff interactions. Speech is transcribed using Whisper, and emotion is extracted from the corresponding video segments via DeepFace, producing 582 real Greek guest–receptionist exchange examples. The resulting data are classified into eight Standard Operating Procedure (SOP) categories. To address data scarcity, we augment the corpus with 1269 synthetic dialogues generated by eight diverse LLMs through the OpenRouter API, yielding a total of 1851 dialogue records with explicit emotion-token annotation. We fine-tune Qwen3.5-35B-A3B using Low-Rank Adaptation (LoRA) through a two-stage process: supervised fine-tuning (SFT) on an 888-example conversation pool and Simple Preference Optimization (SimPO) on a 1899-pair preference pool, each split 80/10/10 into training, validation, and test. The resulting model is integrated into an interactive hotel-terminal system combining YOLO-based person detection, face-recognition-driven guest personalization, Kokoro neural text-to-speech (TTS), and a multi-service orchestration layer connected to the hotel Property Management System (PMS). Evaluation combines standard text metrics, emotion-aware scoring, and a large-model judge. The results indicate targeted improvements in the rule-based contextual emotion-policy match and staff-emotion policy compliance compared to the base model, while general response-quality gains remain more modest. In particular, the rule-based contextual policy-match score improves from 0.614 to 0.901, while forbidden staff-emotion outputs decrease from 0.142 to 0.018. The deployed pilot demonstrates the practical integration of a personalized, emotion-aware LLM assistant in an interactive hotel-terminal setting; end-to-end latency and fully hotel-side edge deployment were not evaluated and are left for future work. HOSPIT-LLM provides a reproducible framework for multimodal dataset creation, preference-based fine-tuning, and deployment of human-centered AI systems. A mobile robotic embodiment is planned as future work. Full article

Recent text-to-speech (TTS) systems can synthesize natural and intelligible speech, but adding controllable emotional expression to a pretrained model while preserving target-speaker identity remains challenging. This setting is especially constrained when the acoustic backbone is kept frozen and emotional adaptation relies on additional trainable modules. We study emotional adaptation for a frozen flow-matching TTS backbone and propose the dual-track residual framework (DTRF). The DTRF keeps the neutral-adapted Matcha-Base backbone unchanged, represents emotion as a neutral-anchor residual, and introduces two residual control paths: an asymmetric zero-initialized acoustic control branch for spectral vector field modulation and an emotional duration adapter (EDA) for duration-level prosody control. Rather than injecting emotion only into the acoustic path, the DTRF applies emotion control to both acoustic vector field prediction and phoneme duration prediction, jointly adjusting spectral realization and temporal prosody. A global neutral anchor converts absolute emotion embeddings into relative residuals so that the control signal describes the deviation from neutral speech toward the target emotion rather than an absolute style vector. During inference, a shared scalar factor $\alpha$ scales both residual paths, providing a practical residual strength interface for controllable emotion rendering. Moderate $\alpha$ values tend to increase emotional salience, whereas larger extrapolative values introduce trade-offs in naturalness, speaker similarity, and intelligibility. Experiments on the English subset of the Emotional Speech Dataset (ESD) show that the DTRF improves emotion-related metrics relative to the internal full-parameter updating and style token conditioning baselines, while maintaining a practical balance among speaker similarity, naturalness, and intelligibility. The emotion control modules contain approximately 27 M trainable parameters, corresponding to 29.61% of the full model parameters and a 70.39% reduction compared with full-parameter updating. These results suggest that jointly modeling acoustic and duration residuals can be an effective strategy for adding residual strength-controlled emotional rendering to a frozen flow-matching TTS model without full-backbone updating. Full article

Text-to-Speech (TTS) methods typically employ a sequential approach with an Acoustic Model (AM) and a vocoder, using a Mel spectrogram as an intermediate representation. However, in home environments, TTS systems often struggle with issues such as inadequate robustness against environmental noise and limited adaptability to diverse speaker characteristics. The quality of the Mel spectrogram directly affects the performance of TTS systems, yet existing methods overlook the potential of enhancing Mel spectrogram quality through more comprehensive speech features. To address the complex acoustic characteristics of home environments, this paper introduces AirSpeech, a post-processing model for Mel-spectrogram synthesis. We adopt a Generative Adversarial Network (GAN) to improve the accuracy of Mel spectrogram prediction and enhance the expressiveness of synthesized speech. By incorporating additional conditioning extracted from synthesized audio using specified speech feature parameters, our method significantly enhances the expressiveness and emotional adaptability of synthesized speech in home environments. Furthermore, we propose a global normalization strategy to stabilize the GAN training process. Through extensive evaluations, we demonstrate that the proposed method significantly improves the signal quality and naturalness of synthesized speech, providing a more user-friendly speech interaction solution for smart home applications. Full article

Bimodal emotion recognition based on audio and text is widely adopted in video-constrained real-world applications such as call centers and voice assistants. However, existing systems suffer from limited cross-domain generalization and monolingual bias. To address these limitations, a cross-lingual bimodal emotion recognition method is proposed, integrating Mamba-based temporal encoders for audio (Wav2Vec2.0) and text (Jina-v3) with a Transformer-based cross-modal fusion architecture (BiFormer). Three corpus-adaptive augmentation strategies are introduced: (1) Stacked Data Sampling, in which short utterances are concatenated to stabilize sequence length; (2) Label Smoothing Generation based on Large Language Model, where the Qwen3-4B model is prompted to detect subtle emotional cues missed by annotators, producing soft labels that reflect latent emotional co-occurrences; and (3) Text-to-Utterance Generation, in which emotionally labeled utterances are generated by ChatGPT-5 and synthesized into speech using the DIA-TTS model, enabling controlled creation of affective audio–text pairs without human annotation. BiFormer is trained jointly on the English Multimodal EmotionLines Dataset and the Russian Emotional Speech Dialogs corpus, enabling cross-lingual transfer without parallel data. Experimental results show that the optimal data augmentation strategy is corpus-dependent: Stacked Data Sampling achieves the best performance on short, noisy English utterances, while Label Smoothing Generation based on Large Language Model better captures nuanced emotional expressions in longer Russian utterances. Text-to-Utterance Generation does not yield a measurable gain due to current limitations in expressive speech synthesis. When combined, the two best performing strategies produce complementary improvements, establishing new state-of-the-art performance in both monolingual and cross-lingual settings. Full article

Significant advances have been made in emotional speech synthesis technology; however, existing models still face challenges in achieving fine-grained emotion style control and simple yet precise emotion intensity regulation. To address these issues, we propose Easy-Control Emotion Text-to-Speech (ECE-TTS), a zero-shot TTS model built upon the F5-TTS architecture, simplifying emotion modeling while maintaining accurate control. ECE-TTS leverages pretrained emotion recognizers to extract Valence, Arousal, and Dominance (VAD) values, transforming them into Emotion-Adaptive Spherical Vectors (EASV) for precise emotion style representation. Emotion intensity modulation is efficiently realized via simple arithmetic operations on emotion vectors without introducing additional complex modules or training extra regression networks. Emotion style control experiments demonstrate that ECE-TTS achieves a Word Error Rate (WER) of 13.91%, an Aro-Val-Domin SIM of 0.679, and an Emo SIM of 0.594, surpassing GenerSpeech (WER = 16.34%, Aro-Val-Domin SIM = 0.627, Emo SIM = 0.563) and EmoSphere++ (WER = 15.08%, Aro-Val-Domin SIM = 0.656, Emo SIM = 0.578). Subjective Mean Opinion Score (MOS) evaluations (1–5 scale) further confirm improvements in speaker similarity (3.93), naturalness (3.98), and emotional expressiveness (3.94). Additionally, emotion intensity control experiments demonstrate smooth and precise modulation across varying emotional strengths. These results validate ECE-TTS as a highly effective and practical solution for high-quality, emotion-controllable speech synthesis. Full article

Digital technologies are undergoing comprehensive integration across diverse domains and processes of the human economy, politics, culture, society, and ecological civilization. This integration brings forth novel concepts, formats, and models. In the context of the accelerated convergence between the digital and physical worlds, a discreet yet momentous transformation is being steered by artificial intelligence generated content (AIGC). This transformative force quietly reshapes and potentially disrupts the established patterns of digital content production and consumption. Consequently, it holds the potential to significantly enhance the digital lives of individuals and stands as an indispensable impetus for the comprehensive transition towards a new era of digital civilization in the future. This paper presents our award-winning project, a large language model (LLM)-powered 3D hyper-realistic interactive digital human system that employs automatic speech recognition (ASR), natural language processing (NLP), and emotional text-to-speech (TTS) technologies. Our system is designed with a modular concept and client–server (C/S) distributed architecture that emphasizes the separation of components for scalable development and efficient progress. The paper also discusses the use of computer graphics (CG) and artificial intelligence (AI) in creating photorealistic 3D environments for meta humans, and explores potential applications for this technology. Full article

Text-to-speech (TTS) synthesizers have been widely used as a vital assistive tool in various fields. Traditional sequence-to-sequence (seq2seq) TTS such as Tacotron2 uses a single soft attention mechanism for encoder and decoder alignment tasks, which is the biggest shortcoming that incorrectly or repeatedly generates words when dealing with long sentences. It may also generate sentences with run-on and wrong breaks regardless of punctuation marks, which causes the synthesized waveform to lack emotion and sound unnatural. In this paper, we propose an end-to-end neural generative TTS model that is based on the deep-inherited attention (DIA) mechanism along with an adjustable local-sensitive factor (LSF). The inheritance mechanism allows multiple iterations of the DIA by sharing the same training parameter, which tightens the token–frame correlation, as well as fastens the alignment process. In addition, LSF is adopted to enhance the context connection by expanding the DIA concentration region. In addition, a multi-RNN block is used in the decoder for better acoustic feature extraction and generation. Hidden-state information driven from the multi-RNN layers is utilized for attention alignment. The collaborative work of the DIA and multi-RNN layers contributes to outperformance in the high-quality prediction of the phrase breaks of the synthesized speech. We used WaveGlow as a vocoder for real-time, human-like audio synthesis. Human subjective experiments show that the DIA-TTS achieved a mean opinion score (MOS) of 4.48 in terms of naturalness. Ablation studies further prove the superiority of the DIA mechanism for the enhancement of phrase breaks and attention robustness. Full article

Due to an increase in the number of disabled people around the world, inclusive solutions are becoming a priority. People with disabilities may encounter many problems and may not be able to easily participate in various activities due to physical barriers, which may sometimes cause them to be frustrated and embarrassed. Recently, the emerging telepresence robot technology has been proposed to enable people with disabilities to increase their presence by incorporating information and communications technology (ICT) into robotics platforms. Therefore, in this paper we conduct a comprehensive analysis using comparative and elicitation studies to understand the current state of mobile telepresence robot systems and to identify the gaps that must be filled. This paper further contributes to the literature by proposing a novel telepresence robot system that adapts text-to-speech (TTS) and ICT technologies with robotics for its use as an assistant. To the authors’ knowledge, the proposed system is the first MRP system that supports speech impairment and introduces emotion components into its communication function. It includes an operator site (mobile) and a remote site (robot) to allow users to control the robot from a distance and communicate with others in remote locations. It allows the user to physically interact with people and show certain emotions through the robot in remote locations, or it can accompany them to speak on their behalf. It can provide agency for both remote and in-class users through emoji-based communication and audio–video streaming with recording functionality. As shown at the end of this paper, the system was tested with 30 people, some of whom had mobility or speech disabilities, showing that the user acceptance score was above 95% and that people with disabilities liked to interact with other people using the proposed system. The users appreciated having the ability to control the robot from a distance and praised the capability to show their emotions through the robot emoji motions and to control the audio–video streaming. From this study, we conclude that the proposed telepresence system could be an asset to people with speech and mobility disabilities and could help them feel physically present in various places. Full article