Audio–Visual Sound Source Localization and Tracking Based on Mobile Robot for The Cocktail Party Problem

Abstract

Locating the sound source is one of the most important capabilities of robot audition. In recent years, single-source localization techniques have increasingly matured. However, localizing and tracking specific sound sources in multi-source scenarios, which is known as the cocktail party problem, is still unresolved. In order to address this challenge, in this paper, we propose a system for dynamically localizing and tracking sound sources based on audio–visual information that can be deployed on a mobile robot. Our system first locates specific targets using pre-registered voiceprint and face features. Subsequently, the robot moves to track the target while keeping away from other sound sources in the surroundings instructed by the motion module, which helps the robot gather clearer audio data of the target to perform downstream tasks better. Its effectiveness has been verified via extensive real-world experiments with a 20% improvement in the success rate of specific speaker localization and a 14% reduction in word error rate in speech recognition compared to its counterparts.

1. Introduction

To achieve natural and effective human–robot Interaction (HRI), robots need to utilize multiple senses to communicate with humans. Among these senses, robot audition has become increasingly important in recent years. The concept of robot audition was first proposed by Nakadai et al. [1] and is defined by three key abilities, (1) the ability to locate sound sources in unknown acoustic environments, (2) the ability to move actively to obtain more information about the sound and (3) the ability to continuously perform acoustic scene analysis in noisy environments. Current research on robot audition focuses on Sound Source Localization (SSL) [2], Sound Source Separation (SSS) [3] and Automatic Speech Recognition (ASR) [4]. While these techniques have become increasingly mature in single-speaker scenarios after more than two decades of development, locating and tracking a particular speaker in a noisy environment where multiple speakers are communicating simultaneously as illustrated in Figure 1 remains a challenge, and is known as the cocktail party problem [5]. This problem involves two challenges [6], how to locate the sound of interest and how to collect a clear audio signal of the target from a variety of background noises.

The first challenge can be solved via SSL. At present, traditional SSL methods have the ability to locate the sound sources relying only on microphone arrays [7,8]. However, when the sound sources are far from the robots compared to the aperture of the microphone array (called far-field effects), these SSL methods cannot estimate the distance between the sources and the robots [9]. To deal with this problem, more and more studies attempt to combine visual Simultaneous Localization and Mapping (SLAM) with SSL [9,10,11,12], which can provide the relative positions of the sources and the robots. Although these SLAM-based SSL approaches can accurately estimate the locations of sound sources, they cannot determine which source a particular sound is emitted from. Specifically, suppose there are two people in a room, one is singing and the other is calling out. The above-mentioned SLAM-based SSL methods can estimate the relative position between these two active sound sources and the robot, but they do not know who is singing and who is making a phone call. Therefore, these techniques cannot be applied directly to resolve the cocktail party problem where the robot should locate the source of interest such as the one that is singing rather than all sound sources in the environment.

To meet the second challenge, there has been an amount of work investigating speech separation in cocktail party environments [3,13,14,15] and the most advanced techniques have been able to separate clear speech from mixed signals [3]. However, because of the front–back ambiguity of binaural arrays [16,17], these methods trained with binaural videos cannot be directly applied in a 3D cocktail party environment. Furthermore, most of them ignore the ability of robots to move actively to acquire more information about the target.

In this paper, we attempt to fill the aforementioned research gaps to some extent. We propose AV-SSLT (short for Audio–visual Sound Source Localization and Tracking), a framework that uses voiceprint and face features to identify the target speaker and locate her/his position and collects clear audio data of the target by moving towards it while staying away from other sound sources in the surroundings controlled via the motion module. The characteristics of AV-SSLT and our contributions can be summarized as follows:

• To identify and localize a specific sound source in a cocktail party environment, we propose an audio–visual approach since the vision can help the audition to distinguish the target speaker. Considering that voiceprint recognition and face recognition technologies are relatively mature, AV-SSLT combines these two types of techniques to find out the specific sound source and its position using microphones and RGB-D cameras. Compared with the existing work [18,19,20,21], the proposed approach is able to autonomously select the target of interest among several speakers. In addition, the method is well suited for situations where the users’ identities are relatively fixed and the audio–visual features can be simply recorded, such as routine company meetings.
• Nowadays, most studies on the cocktail party problem ignore the ability of the robot to move actively to obtain more information about sound sources. To make full use of this ability to acquire clearer audio of the target, a motion module is designed based on the Direction of Arrival (DoA) of the sounds. After locking on the target, the robot will move towards this speaker while keeping away from other interfering sound sources. To the best of our knowledge, this is the first work to improve the quality of the captured audio through robot motion.
• To verify the effectiveness of AV-SSLT, it was deployed on a mobile robot and real-world experiments were conducted with multi-speaker scenarios. Compared with its counterparts, AV-SSLT has a much higher success rate in localizing the particular target and significantly better accuracy concerning speech recognition at the end of tracking. To make our results reproducible, the source code is available at https://github.com/ZhanboShiSE/AV-SSLT (accessed on 10 May 2023).

2. Related Work

2.1. Audio–Visual Sound Source Localization (AV-SSL)

Audio–visual sound source localization techniques can be categorized into traditional methods and learning-based methods. The traditional ones mainly combine SSL with SLAM [9,10,11,12]. This type of method typically begins with estimating DoA using multi-channel audio data from microphone arrays and then calculates the relative distances between the sound sources and the robot referenced by SLAM. For instance, Emery et al. [10] used SEVD-MUSIC to estimate the DoA of the sound and RGB-D SLAM for mapping and localization. Subsequently, the extended Kalman filter was utilized to track the movement of the source. This method successfully located one static or moving speaker in the real world. Similarly, Chau et al. [11] made use of GSVD-MUSIC and EKF-SLAM to localize and track the sound sources. In addition, combined with OpenPose, a monocular camera-based human pose estimator, their system focused more on human–robot interaction. In contrast with the above two studies using just one robot, Michaud et al. [9] proposed a method using two robots referenced via RTAB-Map, a graph-based SLAM, to localize the sound source. In [9], each robot was equipped with a tower-shaped 16-mic 3D array. To locate and track the sound source, SRP-PHAT-HSDA and triangulation were used in their method. It is obvious that the aforementioned studies improve the accuracy of sound source localization by means of SLAM. In order to improve the robustness of SLAM to dynamic environments, an approach based on direct-path relative transfer functions was presented by Zhang et al. [12] to eliminate the moving objects.

With the development of machine learning, some researchers have been seeking learning-based solutions to locate sound sources. For instance, Masuyama et al. [22] introduced a self-supervised learning-based probabilistic spatial model for AV-SSL. To train this model, 360° images with multi-channel audio signals were fed into visual and auditory DNNs. Chen et al. [23] replaced the DNN with SVM and trained the models using 1-channel audio from a single microphone to simultaneously estimate the DoA and the distance between the sound source and the microphone. Regrettably, although the above systems can accurately locate multiple sound sources at the same time, they are unable to identify the sources and focus on a specific target, which is actually a highly desired capability of robot audition in cocktail party environments. To address this issue, SELDnet was proposed in [21] for Sound Event Localization and Detection (SELD) of overlapping sources to provide the robot with the ability to identify the temporal activities of each sound event and estimate their respective spatial location trajectories when active. In recent years, SELD has received increasing attention from researchers [24,25,26]. However, even the most advanced techniques of SELD only have a success rate of around $41.6\%$ in sound event detection and an angular error of approximately 18.5° in sound event localization [20]. Due to the low performance, these techniques cannot be applied directly to resolve the cocktail party problem.

2.2. Motion Planning for Robot Audition

With the development of robot path planning techniques [27,28], the majority of current research attempts to solve the front–back ambiguity of linear arrays by designing the path of the robot based on the information theory criteria. For example, Vinecent et al. [16] proposed a dynamic programming algorithm that minimized the entropy of a discrete occupancy grid to guide the movement of the robot to resolve ambiguities. Similarly, the belief entropy at each future time step was used to quantify the uncertainty in estimating the source location by Nguyen et al. [17]. After that, the motion of the robot was controlled via a Monte Carlo tree search algorithm. Later, in [29], Nguyen et al. extended their former work [17] to locate moving, intermittent sounds using the extended mixture Kalman filter. Unlike the previous methods that capture the sound signals with the Kinect 4-microphone array, a Monte Carlo exploration approach for active binaural localization was presented by Schymura et al. [30]. Bustamante and Danes [31] also proposed a robot motion control method based on information entropy using binaural microphones. Furthermore, in [32], Bustamante et al. took the rotation of the robot head into account to reduce the number of steps to find the source. For situations with strong reverberation and echo such as lecture hall, Sewtz et al. [33] introduced the motion model enhanced MUSIC to improve the accuracy of SSL. Unfortunately, the studies mentioned are mainly designed for a single sound source. To the best of our knowledge, no work has attempted to design a motion planning algorithm for robots to perceive sound sources in a cocktail party environment.

3. Methodology

3.1. Overview

In order to enable the mobile robot to selectively locate and track the sound source of interest in the cocktail party environments, we propose an audio–visual sound source localization and tracking system called AV-SSLT. The system first collects the audio data from the environment with an omnidirectional microphone. Acoustic features such as Mel-scale Frequency Cepstral Coefficients (MFCC) are subsequently extracted from the 1-channel audio data for detecting keywords of interest. When awakened by a keyword, the system extracts the voiceprint features from the audio. By comparing the voiceprint features with the voiceprint–face database, the system obtains the face features of the target. By doing so, the robot can identify the target speaker among multiple speakers in RGB-D images. As for the control of the robot’s movement, the system estimates the DoA using 4-channel audio data at first. After obtaining the DoA information, the robot rotates in these directions to capture the target speaker with the camera. Once the target is detected, the robot moves towards it while staying away from other interfering sound sources in the surroundings as much as possible under the control of the motion module. The overall framework of AV-SSLT is illustrated in Figure 2.

3.2. Specific Target Identification

To locate and track the sound source of interest in a cocktail party environment, the system first needs to identify the specific speaker uttering the keywords. To achieve this, we equip the robot with an omnidirectional microphone to constantly pick up sound signals in the environment. Meanwhile, the input signals are transformed into the complex domain using Short-Time Fourier transform (STFT) to extract the MFCC features. These MFCC features are then fed into a recurrent neural network transducer (RNN-T) [34,35,36] to detect the keywords. Once the system is awakened by a keyword, the part of the audio containing the keyword is fed into Tse-FV [37] for voiceprint feature extraction. The resulting features from Tse-FV consist of two parts, MFCC features and d-vector features. By comparing these features with the voiceprint–face matching database, the identity of the target speaker can be determined.

Note that when the target’s identity is retrieved from the database, her or his facial features are also retrieved. With such features, we further resort to ArcFace [38] to find the target face for the robot equipped with the RGB-D camera to locate and track the speaker of interest. It is easy for the above process if the target is within the camera’s field of view (FOV). However, if the target is out of view, the robot must first rotate to face the target, which is guided by the DoA estimation discussed in the next subsection.

3.3. DoA Estimation

In a real-world environment, the sound source of interest is often out of the FOV of the camera. To locate and track such a target, the robot should first estimate its DoA and then rotate to find it. In our AV-SSLT, the DoA estimation is based on the SEVD-MUSIC algorithm [18] using another microphone array. MUSIC takes the transfer function from sound to each microphone as a prior. Consider a microphone array with $N_m$ microphones (which is 4 in our system), the multichannel transfer function can be expressed as,

$$\mathit{H}(\theta ,\omega )={[h_1(\theta ,\omega ),\dots ,h_{N_m}(\theta ,\omega )]}^T,$$

(1)

where $\theta$ is the azimuth of the sound source relative to the microphone array in the 2D case, $\omega$ represents the frequency and $h_i,i\in (1,N_m)$ is the transfer function from the sound to the i-th microphone.

To perform sound source localization, we have to calculate the correlation matrix among the input signal channels. To achieve this, the signal vector in the frequency domain is firstly obtained via STFT of the input acoustic signal in $N_m$ channels as,

$$\mathit{X}(\omega ,f)={[X_1(\omega ,f),\dots ,X_{N_m}(\omega ,f)]}^T,$$

(2)

where f expresses the frame index. With $\mathit{X}(\omega ,f)$, the correlation matrix among its channels can be further defined as follows for every frame and every frequency,

$$\mathit{R}(\omega ,f)=\mathit{X}(\omega ,f)\mathit{X}{(\omega ,f)}^{*},$$

(3)

where ${(·)}^{*}$ denotes the conjugate transpose operator. In order to be robust against noise, in AV-SSLT, $\mathit{R}(\omega ,f)$ is time-averaged over $T_r$ frames via,

$$\mathit{R}{}^{\prime }(\omega ,f)=\frac{1}{T_r}\sum _{i=0}^{T_r-1}\mathit{R}(\omega ,f+i).$$

(4)

To separate the speech signal and noise signal from the mixed signal, standard eigenvalue decomposition of $\mathit{R}{}^{\prime }(\omega ,f)$ is conducted to decompose the signal space into the speech and noise subspaces,

$$\mathit{R}{}^{\prime }(\omega ,f)=\mathit{E}(\omega ,f)\mathbf{\Lambda }(\omega ,f){\mathit{E}}^{-1}(\omega ,f),$$

(5)

in which $\mathbf{\Lambda }(\omega ,f)=\mathrm{diag}({\lambda }_1(\omega ,f),\dots ,{\lambda }_{N_m}(\omega ,f))$ is a diagonal matrix composed of the eigenvalues of $\mathit{R}{}^{\prime }(\omega ,f)$ in descending order and the i-th column of $\mathit{E}(\omega ,f)=[e_1(\omega ,f),\dots ,$ $e_{N_m}(\omega ,f)]$ is the eigenvector of $\mathit{R}{}^{\prime }(\omega ,f)$ associated with $\mathit{R}{}^{\prime }(\omega ,f)$’s eigenvalue ${\lambda }_i(\omega ,f)$.

Subsequently, the MUSIC spectrum $\zeta$ for SSL can be written as follows,

$$\zeta (\theta ,\omega ,f)=\frac{|{\mathit{H}}^{*}(\theta ,\omega )\mathit{H}(\theta ,\omega )|}{{\sum }_{i=N_s+1}^{N_m}\left|{\mathit{H}}^{*}(\theta ,\omega )e_i(\omega ,f)\right|},$$

(6)

where $\mathit{H}(\theta ,\omega )$ is the multichannel transfer function defined in Equation (1) and $N_s$ is an empirical parameter considered as the number of sound sources in the SSL process to remove the noise from the time-averaged correlation matrix $\mathit{R}{}^{\prime }(\omega ,f)$.

Note that the MUSIC spectrum defined by Equation (6) is for every frequency. To make it applicable to human speech within a certain frequency range between $i_{low}$ and $i_{high}$, the response of the MUSIC spectrum can be calculated by summing out $\omega$ in Equation (6) as,

$$\overline{\zeta }(\theta ,f)=\frac{1}{i_{high}-i_{low}+1}\sum _{i=i_{low}}^{i_{high}}\zeta (\theta ,{\omega }_i,f).$$

(7)

At every frame f, the peaks of $\overline{\zeta }(\theta ,f)$ which have larger values than a threshold are detected as the direction of the active sound source.

In a cocktail party environment, there are typically several active sound sources present at the same time. To capture clearer speech of the speaker of interest in such an environment, it is natural to get close to that speaker while keeping away from other distracting sound sources, which can be achieved with the motion module in our AV-SSLT.

3.4. Motion Module

In order to be more robust against interfering sound sources in the cocktail party environment where multiple sound sources are active simultaneously, the motion module is designed to control the robot to move towards the target while maintaining a distance from other sound sources based on the estimated DoA. More specifically, the purpose of the motion module is to find an optimal direction, which is a weighted sum of the direction of the target sound source and the opposite directions of the other interfering sound sources. To calculate this direction, the directions of sound sources are firstly converted into unit vectors in the robot coordinate system by means of trigonometric functions as follows,

$$\begin{array}{cc}\hfill \mathit{V}=& [{(\cos {\theta }_t,\sin {\theta }_t)}^T,\hfill \\ \hfill & {(\cos {\theta }_0,\sin {\theta }_0)}^T,\dots ,{(\cos {\theta }_{N_o},\sin {\theta }_{N_o})}^T],\hfill \end{array}$$

(8)

where ${\theta }_t$ denotes the direction of the target sound source, $N_o$ is the number of other sound sources in the environment and ${\theta }_i,i\in (1,N_o)$ represents other sound source directions.

After that, the response of the MUSIC spectrum of the target sound source and the other sound sources for the current time frame f can be written as,

$$\overline{\mathbf{\zeta }}{}^{\prime }=[\overline{\zeta }({\theta }_t,f),\overline{\zeta }({\theta }_1,f),\dots ,\overline{\zeta }({\theta }_{N_o},f)],$$

(9)

which indicates the intensity of the effect of each sound source on the robot.

Thus, the direction of the robot’s next movement can be straightforwardly expressed as,

$${(x,y)}^T=\frac{\mathit{V}({\mathit{W}}^T\odot {\overline{\mathbf{\zeta }}{}^{\prime }}^T)}{\left|\right|\mathit{V}({\mathit{W}}^T\odot {\overline{\mathbf{\zeta }}{}^{\prime }}^T){\left|\right|}_2},$$

(10)

where $\mathit{W}=[w_t,w_1,\dots ,w_{N_o}]$ indicates the weights of the directions of these sound sources, which affect the length of the path to the target, and ⊙ denotes the element-wise product.

In order to instruct the robot to rotate, the direction obtained in Equation (10) is further converted into radians by the inverse trigonometric functions as,

$$\theta {}^{\prime }=arccos\frac{x}{\sqrt{x^2+y^2}}.$$

(11)

4. Experiments and Results

To validate the performance of our proposed AV-SSLT in cocktail party environments, we performed experiments to test its ability to accurately identify the target speaker in a noisy environment with several speakers and its accuracy of speech recognition when the robot is at the end of tracking. The experimental setup and the experimental results are as follows.

4.1. Experimental Setup

4.1.1. Experimental Site

The room where the experiments were conducted was an office of size 10.5 × 10.5 m². There was an open space of the size about 3 × 5 m² in the middle surrounded by tables and chairs. The Room Impulse Response (RIR) was measured according to [39] as shown in Figure 3. During the measurements, a four-channel microphone array was positioned in the center of the room 1 m above the ground. At the same height, a loudspeaker was installed. The loudspeakers were placed at positions on a circle with a 1 m radius around the microphone array. An initial position of the loudspeaker was marked as $0^{°}$, from which the measurements were conducted in a counter-clockwise direction at ${10}^{°}$ intervals. The Reverberation Time (RT60) of this room was about 1.52 s.

4.1.2. Hardware

The platform for experiments was a Handsfree stone robot equipped with an Orbbec Astro Pro RGB-D camera, a PlayStation Eye 4-channel microphone array and a conference-grade omnidirectional microphone. The configuration of the sensors is illustrated in Figure 4. Another high-resolution 32-channel spherical microphone array was used to generate the FOA format recordings. The processor hardware of the mobile robot is listed in

Table 1. The processor hardware of the mobile robot.

Hardware	Model
CPU	Intel Core i7-8700 @ 3.20 GHz
GPU	NVIDIA GeForce GTX 1650 4 G
Memory	Micron 8G DDR4 2400 MHz $\times 2$

.

4.1.3. Episode Definition

In the experiments, three speakers stood casually in the room and had a conversation. For each episode, one of the speakers uttering the keywords was treated as the target speaker. The other two speakers were presented as interfering sound sources. The sound of a television program was considered as the background noise and its level was about 55 dB measured 1.5 m away from the television. All the sound sources mentioned above can be considered as far-field signals introduced in [40]. The starting position of the robot was a random position in the room and the starting orientation of the robot was a random direction with the target speaker out of the FOV of the camera. An episode was considered successful if the robot was able to stop within 1.5 m of the speaker who spoke the keywords and keep the camera facing this speaker. As an example, a specific configuration of an episode in the experiments is presented in Figure 5.

4.1.4. Metrics

The average success rate of the robot in locating a speaker uttering the keywords, the average distance to the target at the end of tracking and the word error rate (WER) of speech recognition at the end of tracking serve as the evaluation metrics for the performance of the system. The first two correspond to the first challenge in the cocktail party problem, locating the sound of interest, while the latter corresponds to the second challenge, collecting a clear audio signal of the target from a multiple sound sources signal.

4.2. Results

4.2.1. The Performance of Locating the Sound of Interest

To verify the ability of our system in solving the first challenge in the cocktail party problem, we conducted experiments in the real world. We compared our AV-SSLT with three representative solutions in this field, the SEVD-MUSIC [18], the noise-robust GSVD-MUSIC [19] and the Multi-ACCDOA-based SELDnet [20,21]. The two MUSIC-based algorithms took the audio data recorded via the four-channel microphone array as input. Two strategies were applied to select the direction of the target sound source from all DoA estimations, a random direction or the direction with the highest MUSIC spectrum response. The SELDnet was trained on the STARSS22 dataset [41] and a dataset synthesized using the method proposed in [42] with a batch size of 128 and a dropout rate of 0.05. The model was then fine-tuned on the real-world FOA format recordings from the experimental room generated via the 32-channel spherical microphone array. The target sound source of SELDnet was selected as the sound source whose sound event category was human speech and which was active at the same time as the keywords were detected. More detailed parameters of the MUSIC-based methods and SELDnet are presented in

Table 2. Parameter list of the MUSIC-based algorithm.

Parameter	Value
Sampling Rate	16,000 Hz
Frame Length	512
Window Size	64
Dimension of Mel-spectrograms	13
Lower Bound Frequency	300 Hz
Higher Bound Frequency	2800 Hz
Max Number of Sounds	3
Interval of Azimuth	${10}^{°}$

and

Table 3. Parameter list of the SELDnet algorithm.

Parameter	Value
Sampling Rate	24,000 Hz
Frame Length	480
Label Frame Length	2400
Dimension of Mel-spectrograms	64
Unify Threshold of Multi-ACCDOA	15
Feature Sequence Length	250
Label Sequence Length	50

, respectively.

The results are summarized in

Table 4. Performance on locating the sound of interest.

Method	Success Rate ↑	Distance to Target ↓
SEVD-MUSIC [18] (random)	24%	2.34 m
SEVD-MUSIC [18] (high response)	40%	2.02 m
GSVD-MUSIC [19] (random)	28%	2.28 m
GSVD-MUSIC [19] (high response)	42%	1.96 m
SELDnet [20,21]	62%	1.68 m
AV-SSLT (ours)	82%	1.28 m

. It can be seen that our AV-SSLT has a much higher success rate in locating the specific target and has the ability to get closer to the target than its counterparts relying solely on the microphone array. That is to say, by combining voiceprint and face features, our approach performs well in locating a specific speaker among multiple speakers, which is the first challenge in the cocktail party problem. Furthermore, in order to demonstrate the capability of our AV-SSLT system to work in real time, the average response time of each module was measured during the experiments and is presented in

Table 5. The average response time of each module.

Module	Response Time
Keywords Detection	101 ms
Voiceprint Recognition	216 ms
Face Recognition	91 ms
DoA Estimation	160 ms

. It can be seen that the total response time of the entire system is about 0.5 s, which is sufficient for real-time interaction with humans.

4.2.2. The Accuracy of Speech Recognition at the End of Tracking

For the second challenge in the cocktail party problem, the clarity of the target audio can be reflected by the accuracy of speech recognition. To verify the effectiveness of our motion module, the WER was calculated at the end of tracking. Two strategies were applied to control the robot to track the target, the robot went directly to the target or the robot moved relying on the motion module. Figure 6 shows the trajectories of the robot in two different episode settings, one with the target speaker outside of the interference sources and the other with the target speaker in the middle of the interference sources. It can be seen that the motion module can control the robot to move it toward the target while keeping away from other interfering sound sources in the surroundings.

Table 6. Word Error Rates (WERs) of speech recognition at the end of tracking.

Method	WER
Going directly to the target	32%
Moving with motion module	18%

reports the average WERs for these two strategies. Obviously, the WER is lower when the robot was guided by our motion module, implying that AV-SSLT can improve the quality of captured speech audio under the control of the motion module.

5. Discussion

Although the success rate of our system in locating the specific sound source is significantly higher compared to the methods without using voiceprint and face features, the overall success rate still has room for improvement. By reviewing the process of the experiments, we identified two main reasons for the failure as follows.

The voiceprint features extracted directly from the mixed signal sometimes cannot be matched with the features in the database. This happened particularly when the interfering signal in the environment was dominant. In other words, the ambient noise was much louder than the desired speech. At this point, it was difficult to extract the exact voiceprint of the target speaker and to obtain the correct face features of the target from the database, leading to the failure in localization and tracking of the target. We hold the view that differential microphone arrays can address this difficulty to some extent. In detail, by designing a specific beam pattern of the arrays, it is expected that the response from the non-target direction will be significantly reduced.

In order to ensure the accuracy of face recognition, a high threshold was set for the face verification process. This resulted in the robot being unable to properly detect and recognize the target face when the face was in an extreme pose, thereby failing to locate and track the target speaker. To deal with this problem, an additional robot motion controller needs to be designed to equip the robot with the ability to actively move towards the front of a person to catch the target’s face when there is no frontal human face in the FOVs of the camera, for example, when the target speakers have their backs to the robot.

6. Conclusions

In this paper, we propose AV-SSLT to localize and track the specific sound source in a cocktail party environment. In order to focus on the target of interest, we present a way by combining voiceprint recognition and face recognition to locate the sound source. To obtain high-quality audio of the target for downstream tasks such as speech recognition, we designed a motion module to control the robot’s movement towards the target while keeping away from other sound sources in the environment. The effectiveness of AV-SSLT has been fully corroborated via extensive practical experiments, which is demonstrated by the higher success rate in locating a specific target, the closer distance to the target at the end of tracking and the lower WER of speech recognition at the end of tracking than its counterparts. However, we have to admit that our system fails in some cases, such as when the target speaker has their back to the camera. Meanwhile, there is still room for improvement in the accuracy of speech recognition. Our future work will focus on improving the accuracy of face recognition when the face is in extreme poses to increase the success rate of specific target localization and on designing the geometry of the microphone array to reduce the interference of noises from other directions.