Variational Reward Estimator Bottleneck: Towards Robust Reward Estimator for Multidomain Task-Oriented Dialogue

Abstract

Despite its significant effectiveness in adversarial training approaches to multidomain task-oriented dialogue systems, adversarial inverse reinforcement learning of the dialogue policy frequently fails to balance the performance of the reward estimator and policy generator. During the optimization process, the reward estimator frequently overwhelms the policy generator, resulting in excessively uninformative gradients. We propose the variational reward estimator bottleneck (VRB), which is a novel and effective regularization strategy that aims to constrain unproductive information flows between inputs and the reward estimator. The VRB focuses on capturing discriminative features by exploiting information bottleneck on mutual information. Quantitative analysis on a multidomain task-oriented dialogue dataset demonstrates that the VRB significantly outperforms previous studies.

1. Introduction

While deep reinforcement learning (RL) has emerged as a viable solution for complicated and high-dimensional decision-making problems [1], including games such as Go [2], chess [3], checkers [4], and poker [5,6], robotic locomotion [7,8], autonomous driving [9,10], and recommender system [11,12], the determination of an effective reward function remains a challenge, especially in multidomain task-oriented dialogue systems. Many recent studies have struggled in sparse-reward environments and employed a handcrafted reward function as a breakthrough [13,14,15,16]. However, such approaches typically are not capable of guiding the dialogue policy through user goals. For instance, as shown in Figure 1, the user cannot attain the goal because the system (S1) that exploits the handcrafted rewards completes the dialogue session too early. Moreover, as the dialog progresses, the user goal will frequently vary.

Due to these problems, systems that exploit the handcrafted rewards fail to assimilate user goals and guide users through user goals, achieving low performance, while humans self-judge from dialog context using well-defined reward function in their minds and generate appropriate responses despite multidomain circumstances.

MaxEnt-IRL [17] and Inverse reinforcement learning (IRL) [18,19] tackle the problem of recovering the reward function automatically and using this reward function to generate optimal behavior. Although generative adversarial imitation learning (GAIL) [20], which applies the GANs framework [21], has proven that the discriminator can be defined as a reward function, GAIL fails to generalize and recover the reward function. Adversarial inverse reinforcement learning (AIRL) [22] enables GAIL to take advantage of disentangled rewards. Guided dialogue policy learning (GDPL) [23] uses the AIRL framework to construct the reward estimator for multidomain task-oriented dialogues. However, these approaches often encounter difficulties in balancing the performance of the reward estimator and policy generator and produce excessively uninformative gradients.

In this paper, we propose the variational reward estimator bottleneck (VRB), a novel and effective regularization algorithm. The VRB uses information bottleneck [24,25,26] to constrain unproductive information flows between dialogue internal representations and state–action pairs of the reward estimator, thereby ensuring highly informative gradients and robustness. The experiments show that the VRB achieves state-of-the-art (SOTA) performances on a multidomain task-oriented dataset.

The remainder of this paper is organized as follows: Section 2 presents the brief background to set the stage for our model. Section 3 describes the proposed method in detail along with mathematical calculations. Section 4 outlines the experimental setup, whereas Section 5 presents the experiments and the results thereof. Section 6 provide discussions and the conclusions of this study.

2. Background

2.1. Dialogue State Tracker

The dialogue state tracker (DST) [27,28,29], which takes dialogue action a and dialogue history as input, updates the dialogue state x and belief state b for each slot. For example, as shown in Figure 2, DST observes the user goal where the user aims to go. At dialogue turn t, the dialogue action is represented as a slot and value pair (e.g., Attraction: (area, centre), (type, concert hall)). Given the dialogue action, DST encodes the dialogue state as $x_t=[a_t^u;a_{t-1};b_t;q_t]$.

2.2. User Simulator

Mimicking various and human-like actions is essential with respect to training task-oriented dialogue systems and evaluating these models automatically. The user simulator $\mu (a^u,t^u|x^u)$ [30,31] in Figure 2 extracts the dialogue action $a^u$ corresponding to the dialogue state $x^u$. $t^u$ stands for whether the user goal is achieved during a conversation. Note that the DST and the user simulator cannot meet the user in the absence of well-defined reward estimation.

2.3. Policy Generator

The policy generator [32,33] encourages the dialogue policy ${\pi }_{\theta }$ to determine the next action that maximizes the reward function ${\widehat{r}}_{\zeta ,\psi }(x_t,a_t,x_{t+1})=f_{\zeta ,\psi }(x_t,a_t,x_{t+1})-log{\pi }_{\theta }\left(a_t\right|x_t)$:

$$\begin{array}{c}\hfill L_{\pi }^{CLIP}\left(\theta \right)={\mathbb{E}}_{x,a\sim \pi }[min({\xi }_t\left(\theta \right){\widehat{A}}_t,\widetilde{{\xi }_t}\left(\theta \right){\widehat{A}}_t)]\\ \hfill L_t^{VF}\left(\theta \right)=-{\left(V_{\theta }-\sum _{k=t}^T{\gamma }^{k-t}{\widehat{r}}_k\right)}^2\end{array}$$

where $\widetilde{{\xi }_t}\left(\theta \right)$ = $\mathrm{clip}({\xi }_t\left(\theta \right),1-ϵ,1+ϵ)$, ${\widehat{A}}_t={\delta }_t+\gamma \lambda {\widehat{A}}_{t+1}$, ${\delta }_t={\widehat{r}}_{\zeta ,\psi }+\gamma V\left(x_{t+1}\right)-V\left(x_t\right)$, and $\delta$ is the TD residual [34]. ${\xi }_t\left(\theta \right)=\frac{{\pi }_{\theta }\left(a_t\right|x_t)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(a_t\right|x_t)}$ and $V_{\theta }$ is the state-value function. Epsilon and $\lambda$ are hyperparameters. The reward function ${\widehat{r}}_{\zeta ,\psi }$ can be simplified in the following manner:

$$\begin{array}{cc}\hfill {\widehat{r}}_{\zeta ,\psi }(x_t,a_t,x_{t+1})& =log\left[D_{\zeta ,\psi }(x_t,a_t,x_{t+1})\right]\hfill \\ \hfill & -log[1-D_{\zeta ,\psi }(x_t,a_t,x_{t+1})]\hfill \\ \hfill & =log\left[-1+\frac{1}{1-D_{\zeta ,\psi }(x_t,a_t,x_{t+1})}\right]\hfill \\ \hfill & =log\left[\frac{exp\left[f_{\zeta ,\psi }(x_t,a_t,x_{t+1})\right]}{{\pi }_{\theta }\left(a_t\right|x_t)}\right]\hfill \\ \hfill & =f_{\zeta ,\psi }(x_t,a_t,x_{t+1})-log{\pi }_{\theta }\left(a_t\right|x_t)\hfill \end{array}$$

where $D_{\zeta ,\psi }(x_t,a_t,x_{t+1})$ is the reward estimator, which is defined as follows [22]:

$$\begin{array}{c}\hfill D_{\zeta ,\psi }(x_t,a_t,x_{t+1})=\frac{exp[f_{\zeta ,\psi }(x_t,a_t,x_{t+1})]}{exp[f_{\zeta ,\psi }(x_t,a_t,x_{t+1})]+{\pi }_{\theta }\left(a_t\right|x_t)}\end{array}$$

3. Proposed Method

3.1. Notations on MDP

To represent inverse reinforcement learning (IRL) as a Markov decision process (MDP), we consider a tuple $\mathcal{M}$ = $(\mathcal{X},\mathcal{A},T,\mathcal{R},{\rho }_0,\gamma )$, where $\mathcal{X}$ is state space, and $\mathcal{A}$ is the action space. The transition probability $T\left(x_{t+1}\right|x_t,a_t)$ defines the distribution of the next state $x_{t+1}$ given state $x_t$, and $a_t$ at time-step t. $\mathcal{R}(x_t,a_t)$ is the reward function of the state–action pair, ${\rho }_0$ is the distribution of the initial state $x_0$, and $\gamma$ is the discount factor. The stochastic policy $\pi \left(a_t\right|x_t)$ maps a state to a distribution over actions. Supposing we are given an optimal policy ${\pi }^{*}$, the goal of IRL is to estimate the reward function $\mathcal{R}$ from the trajectory $\tau =\{x_0,a_0,x_1,a_1,\cdots ,x_T,a_T\}\sim {\pi }^{*}$. However, building an effective reward function is challenging, especially in a multidomain task-oriented dialogue system.

3.2. Reward Estimator

The reward estimator [23], which is an essential component of multidomain task-oriented dialogue systems, evaluates dialogue state–action pairs at dialogue turn t and estimates the reward that is used for guiding the dialogue policy through the user goal. Based on MaxEnt-IRL [17], each dialogue session $\tau$ in a set of human dialogue sessions $\mathcal{D}=\{{\tau }_1,{\tau }_2,\cdots ,{\tau }_H\}$ can be modeled as a Boltzmann distribution that does not exhibit additional preferences for any dialogue sessions.

$$\begin{array}{c}\hfill f_{\zeta }\left(\tau \right)=log\left(\frac{exp\left({\mathcal{R}}_{\zeta }\right)}{Z}\right)\end{array}$$

where ${\mathcal{R}}_{\zeta }={\sum }_{t=0}^T{\gamma }^t{r}_{\zeta }(x_t,a_t)$, Z is a partition function, $\zeta$ is a parameter of the reward function, and ${\mathcal{R}}_{\zeta }$ denotes a discounted cumulative reward. To imitate human behaviors, the reward estimator should learn the distributions of human dialogue sessions using the KL divergence loss:

$$\begin{array}{cc}\hfill L_{\pi }\left(\theta \right)& \approx -\mathrm{KL}\left({\pi }_{\theta }\left(\tau \right)\left|\right|\frac{exp\left({\mathcal{R}}_{\zeta }\right)}{Z}\right)\hfill \\ \hfill & =\sum {\pi }_{\theta }\left(\tau \right)log\left(\frac{\frac{exp\left({\mathcal{R}}_{\zeta }\right)}{Z}}{\frac{{\pi }_{\theta }\left(\tau \right)}{1}}\right)\hfill \\ \hfill & ={\mathbb{E}}_{\tau \sim \pi }[log\left(\frac{exp\left({\mathcal{R}}_{\zeta }\right)}{Z}\right)-log{\pi }_{\theta }\left(\tau \right)]\hfill \\ \hfill & ={\mathbb{E}}_{\tau \sim \pi }[f_{\zeta }\left(\tau \right)-log{\pi }_{\theta }\left(\tau \right)]\hfill \\ \hfill & ={\mathbb{E}}_{x,a\sim \pi }\left[f_{\zeta ,\psi }(x_t,a_t,x_{t+1})\right]\hfill \\ \hfill & +{\mathbb{E}}_{x,a\sim \pi }[-log{\pi }_{\theta }(x_t,a_t,x_{t+1})]\hfill \\ \hfill & ={\mathbb{E}}_{x,a\sim \pi }\left[f_{\zeta ,\psi }(x_t,a_t,x_{t+1})\right]+H\left({\pi }_{\theta }\right)\hfill \end{array}$$

where $H\left({\pi }_{\theta }\right)$ is the entropy of dialogue policy ${\pi }_{\theta }$. The reward estimator maximizes the entropy, which indicates maximizing the likelihood of observed dialogue sessions. Therefore, the reward estimator is learned to discern between human dialogue sessions $\mathcal{D}$ and dialogue sessions that are generated by the dialogue policy.

$$\begin{array}{cc}\hfill L_f(\zeta ,\psi )& =-\mathrm{KL}\left(\mathcal{D}\left(\tau \right)\left|\right|\frac{exp\left({\mathcal{R}}_{\zeta }\right)}{Z}\right)\hfill \\ \hfill & -\left(-\mathrm{KL}\left({\pi }_{\theta }\left(\tau \right)\left|\right|\frac{exp\left({\mathcal{R}}_{\zeta }\right)}{Z}\right)\right)\hfill \\ \hfill & ={\mathbb{E}}_{x,a\sim \mathcal{D}}\left[f_{\zeta ,\psi }(x_t,a_t,x_{t+1})\right]+H\left(\mathcal{D}\right)\hfill \\ \hfill & -{\mathbb{E}}_{s,a\sim \pi }\left[f_{\zeta ,\psi }(x_t,a_t,x_{t+1})\right]-H\left({\pi }_{\theta }\right)\hfill \end{array}$$

Note that $H\left(\mathcal{D}\right)$ and $H\left({\pi }_{\theta }\right)$ are not dependent on the parameters $\zeta$ and $\psi$. Thus, the reward estimator can be trained using gradient-based optimization as follows:

$$\begin{array}{c}\hfill L_f(\zeta ,\psi )={\mathbb{E}}_{x,a\sim \mathcal{D}}\left[f_{\zeta ,\psi }(x_t,a_t,x_{t+1})\right]\\ \hfill -{\mathbb{E}}_{x,a\sim \pi }\left[f_{\zeta ,\psi }(x_t,a_t,x_{t+1})\right]\end{array}$$

(1)

3.3. Variational Reward Estimator Bottleneck

The variational information bottleneck [24,25,26] is an theoretical information approach that restricts unproductive information flow between the discriminator and inputs. Inspired by this approach, we propose a regularized objective that constrains the mutual information between encoded original inputs and state–action pairs, thereby ensuring highly informative internal representations and a robust adversarial model. Our proposed method trains an encoder that is maximally informative regarding human dialogues.

To this end, we employ a stochastic encoder and an upper bound constraint on the mutual information between the dialogue states X and latent variables $\mathbf{Z}$:

$$\begin{array}{c}\hfill L_{f,\mathbf{E}}(\zeta ,\psi )={\mathbb{E}}_{x,a\sim \mathcal{D}}\left[{\mathbb{E}}_{\mathbf{z}\sim \mathbf{E}\left(\mathbf{z}\right|x_t,x_{t+1})}\left[f_{\zeta ,\psi }({\mathbf{z}}_g,{\mathbf{z}}_h^{\prime },{\mathbf{z}}_h)\right]\right]\\ \hfill -{\mathbb{E}}_{x,a\sim \pi }\left[{\mathbb{E}}_{\mathbf{z}\sim \mathbf{E}\left(\mathbf{z}\right|x_t,x_{t+1})}\left[f_{\zeta ,\psi }({\mathbf{z}}_g,{\mathbf{z}}_h^{\prime },{\mathbf{z}}_h)\right]\right]\\ \hfill \mathrm{s}.\mathrm{t}.I(\mathbf{Z},X)\le I_c\end{array}$$

(2)

where $f_{\zeta ,\psi }({\mathbf{z}}_g,{\mathbf{z}}_h^{\prime },{\mathbf{z}}_h)=D_g\left({\mathbf{z}}_g\right)+\gamma D_h\left({\mathbf{z}}_h^{\prime }\right)+D_h\left({\mathbf{z}}_h\right)$, and D is modeled with nonlinear function. Note that $f_{\zeta ,\psi }({\mathbf{z}}_g,{\mathbf{z}}_h^{\prime },{\mathbf{z}}_h)$ is divided into the three terms $D_g\left({\mathbf{z}}_g\right)$, $\gamma D_h\left({\mathbf{z}}_h^{\prime }\right)$, and $D_h\left({\mathbf{z}}_h\right)$, based on GANs [21], GAN-GCL [35], and AIRL [22]. $D_g$ represents the encoded disentangled reward approximator with the parameter $\zeta$, and $D_h$ is the encoded shaping term with the parameter $\psi$. Stochastic encoder $\mathbf{E}\left(\mathbf{z}\right|x_t,x_{t+1})$ can be defined as $\mathbf{E}\left(\mathbf{z}\right|x_t,x_{t+1})={\mathbf{E}}_g\left({\mathbf{z}}_g\right|x_t)·{\mathbf{E}}_h\left({\mathbf{z}}_h\right|x_t)·{\mathbf{E}}_h\left({\mathbf{z}}_h^{\prime }\right|x_{t+1})$, which maps states to a latent distribution $\mathbf{z}$: $\mathbf{E}\left(\mathbf{z}\right|x_t)=\mathcal{N}({\mu }_{\mathbf{E}}\left(x_t\right),{\Sigma }_{\mathbf{E}}\left(x_t\right))$. $r\left(\mathbf{z}\right)=\mathcal{N}(0,I)$ is standard Gaussian, and $I_c$ stands for an enforced upper bound on mutual information.

To optimize $L_{f,\mathbf{E}}(\zeta ,\psi )$, VRB introduces a Lagrange multiplier $\phi$ as follows:

$$\begin{array}{c}\hfill L_{f,\mathbf{E}}(\zeta ,\psi )={\mathbb{E}}_{x,a\sim \mathcal{D}}\left[{\mathbb{E}}_{\mathbf{z}\sim \mathbf{E}\left(\mathbf{z}\right|x_t,x_{t+1})}\left[f_{\zeta ,\psi }({\mathbf{z}}_g,{\mathbf{z}}_h^{\prime },{\mathbf{z}}_h)\right]\right]\\ \hfill -{\mathbb{E}}_{x,a\sim \pi }\left[{\mathbb{E}}_{\mathbf{z}\sim \mathbf{E}\left(\mathbf{z}\right|x_t,x_{t+1})}\left[f_{\zeta ,\psi }({\mathbf{z}}_g,{\mathbf{z}}_h^{\prime },{\mathbf{z}}_h)\right]\right]\\ \hfill +\phi ({\mathbb{E}}_{x,a\sim \pi }[\mathrm{KL}\left[\mathbf{E}\left(\mathbf{z}\right|x_t,x_{t+1})\right]\left|\right|r\left(\mathbf{z}\right)]-I_c)\end{array}$$

(3)

where the mutual information between dialogue states X and latent variable $\mathbf{Z}$ is

$$\begin{array}{cc}\hfill I(\mathbf{Z},X)& =\mathrm{KL}\left[p\right(\mathbf{z},x\left)\right|\left|p\right(\mathbf{z}\left)p\right(x\left)\right]\hfill \\ \hfill & {\displaystyle =\int d\mathbf{z}dxp(\mathbf{z},x)log\frac{p(\mathbf{z},x)}{p\left(\mathbf{z}\right)p\left(x\right)}}\hfill \\ \hfill & {\displaystyle =\int d\mathbf{z}dxp\left(x\right)\mathbf{E}\left(\mathbf{z}\right|x)log\frac{\mathbf{E}\left(\mathbf{z}\right|x)}{p\left(\mathbf{z}\right)}}\hfill \\ \hfill & {\displaystyle \le I_c=\int d\mathbf{z}dx{\pi }_{\theta }\left(x\right)\mathbf{E}\left(\mathbf{z}\right|x)log\frac{\mathbf{E}\left(\mathbf{z}\right|x)}{r\left(\mathbf{z}\right)}}\hfill \\ \hfill & ={\mathbb{E}}_{x,a\sim \pi }\left[\mathrm{KL}\right[\mathbf{E}\left(\mathbf{z}\right|x)\left|\right|r\left(\mathbf{z}\right)\left]\right]\hfill \end{array}$$

In Equation (3), the VRB minimizes the mutual information with dialogue states to focus on discriminative features. The VRB also minimizes the KL divergence with the human dialogues, while maximizing the KL divergence with the generated dialogues, thereby distinguishing effectively between samples from dialogue policy and human dialogues. Our proposed model is summarized in Algorithm 1.

Algorithm 1 Algorithm of Variational Reward Estimator Bottleneck

4. Experimental Setup

4.1. Dataset Details

We evaluated our method on multidomain wizard of oz [36] (MultiWOZ), which contained approximately 10,000 large-scale, multidomain, and multiturn conversational dialogue corpora. MultiWOZ consisted of 7 distinct task-oriented domains, 24 slots, and 4510 slot values. The dialogue sessions were randomly divided into training, validation, and test set. The validation and test sets contained 1000 sessions, respectively.

4.2. Models Details

We used the agenda-based user simulator [30] and VHUS-based user simulator [31]. The policy network ${\pi }_{\theta }$ and value network V are MLPs with two hidden layers. $g_{\zeta }$ and $h_{\psi }$ are MPLs with one hidden layer each. We used the ReLu activation function and Adam optimizer for the MLPs. We trained our model using a single NVIDIA GTX 1080ti GPU. Detailed hyperparameters are shown in

Table 1. Detail description of VRB hyperparameters.

Hyperparameters	Value
Lagrange multiplier $\phi$	0.001
Upper bound $I_c$	0.5
Learning rate of dialogue policy	0.0001
Learning rate of reward estimator	0.0001
Learning rate of user simulator	0.001
Clipping component $ϵ$ for dialogue policy	0.02
GAE component $\lambda$ for dialogue policy	0.95

.

We compare the proposed method with the following previous studies: GP-MBCM [37], ACER [38], PPO [33], ALDM [39], and GDPL [23]. GP-MBCM [37] trains a number of policies on different datasets based on the Bayesian committee machine [40]. ACER [38] suggests the importance of weight truncation with bias correction for sampling efficiency. PPO [33] employs an effective algorithm that attains the data’s robust and efficient performance using only a first-order optimizer. ALDM [39] shows an adversarial learning method to learn dialogue rewards directly from dialogue samples. GDPL [23] is the current SOTA model that consists of a dialogue reward estimator based on IRL.

4.3. Evaluation Details

To evaluate the performances of these models, we introduce four metrics: (i) Turns: we record the average number of dialogue turns between the user simulator and dialogue agent. (ii) Match rate: we conduct match rate experiments to analyze whether the booked entities are matched with the corresponding constraints in the multidomain environment. For instance, in Figure 2, entertainment should be matched with concert hall in the center. The match rate ranges from 0 to 1 and scores 0 if an agent is unable to book the entity. (iii) Inform F1: we test the ability of the model to inform all of the requested slot values. For example, as shown in Figure 1, the price range, food type, and area should be informed if the user wishes to visit a high-end Cuban restaurant in Cambridge. (iv) Success rate: in the success rate experiment, a dialogue session scores 0 or 1. We obtain 1 if all required information is presented, and every entity is booked successfully.

5. Main Results

5.1. Experimental Results of Agenda-Based User Simulators

Table 2. Results on agenda-based user simulators.

Model	Agenda
	Turns	Match	Inform	Success
GP-MBCM [37]	2.99	44.29	19.04	28.9
ACER [38]	10.49	62.83	77.98	50.8
PPO [33]	9.83	69.09	83.34	59.1
ALDM [39]	12.47	62.60	81.20	61.2
GDPL [23]	7.64	83.90	94.97	86.5
VRB (Ours)	7.59	90.87	90.97	90.4
Human	7.37	95.29	66.89	75.0

presents the empirical results on both simulators and MultiWOZ. In the agenda-based setting, we observe that our proposed method achieves a new SOTA performance. Note that an outstanding model should obtain high scores in every metric, not just a single one, because to regard a dialogue as having ended successfully, every request should be informed precisely, thereby guiding a dialogue through the user goal. Although GDPL achieves the highest score in Inform F1, our proposed model acts more human-like with respect to Turns, which is close to the human evaluation score: 7.37, and provides more accurate slot values and matched entities than the other methods.

5.2. Experimental Results of VHUS-Based User Simulators

On the other hand, in the VHUS setting, though PPO behaves more human-like in Turns, PPO exhibits greater difficulty in providing accurate information, while our model does not because our approach constrains unproductive information flows. Results in

Table 3. Results on VHUS-based user simulators.

Model	VHUS
	Turns	Match	Inform	Success
GP-MBCM [37]	-	-	-	-
ACER [38]	22.35	33.08	55.13	18.6
PPO [33]	19.23	33.08	56.31	18.3
ALDM [39]	26.90	24.15	54.37	16.4
GDPL [23]	22.43	36.21	52.58	19.7
VRB (Ours)	20.96	44.93	56.93	20.1

demonstrate that our proposed model outperforms existing models, providing more definitive information than the other methods. Similar to the agenda-based setting, the VHUS-based model also showed the best performance. It demonstrates that our methodology reflecting human-like characteristics is a very effective methodology.

5.3. Verification of Robustness

As shown in Figure 3 and Figure 4, to evaluate the robustness of the models, we conduct experiments over 30 times for each model and visualize the results using a violin plot. Experimental results show that our proposed method outperforms PPO in every metric, despite some negative outliers, and has a much lower standard deviation than PPO. An example of a dialogue session comparison between VRB and PPO is available in

Table 4. A comparison between VRB and PPO with respect to the dialogue act.

VRB		PPO
U:	{‘attraction-inform-area-1’: ’south’}	U:	{‘attraction-inform-area-1’: ‘south’}
S:	{‘attraction-inform-choice-1’: ‘8’, ‘attraction-request-type-?’: ‘?’}	S:	{‘attraction-inform-choice-1’: ‘8’, ‘attraction-request-type-?’: ‘?’}
U:	{‘attraction-request-post-?’: ‘?’, ‘attraction-request-phone-?’: ‘?’, ’attraction-request-addr-?’: ‘?’, ‘attraction-request-fee-?’: ‘?’, ‘attraction-request-type-?’: ‘?’}	U:	{‘attraction-request-post-?’: ‘?’, ‘attraction-request-phone-?’: ‘?’, ‘attraction-request-addr-?’: ‘?’, ‘attraction-request-fee-?’: ‘?’, ‘attraction-request-type-?’: ‘?’}
S:	{‘attraction-inform-name-1’: ‘the junction’, ‘attraction-recommend-name-1’: ‘the junction’, ‘attraction-recommend-addr-1’: ‘clifton way’}	S:	{‘attraction-inform-name-1’: ‘the junction’, ‘attraction-inform-fee-1’: ‘?’, ‘attraction-recommend-name-1’: ‘the junction’}
U:	{‘attraction-request-post-?’: ‘?’, ‘attraction-request-phone-?’: ‘?’, ‘attraction-request-fee-?’: ‘?’, ‘attraction-request-type-?’: ‘?’}	U:	{‘attraction-request-post-?’: ‘?’, ‘attraction-request-phone-?’: ‘?’, ‘attraction-request-addr-?’: ‘?’, ‘attraction-request-type-?’: ‘?’}
S:	{‘attraction-inform-fee-1’: ‘?’, ‘attraction-inform-phone-1’: ‘01223511511’, ‘attraction-inform-post-1’: ‘cb17gx’}	S:	{‘attraction-inform-name-1’: ‘the junction’, ‘attraction-inform-addr-1’: ‘clifton way’, ‘attraction-inform-phone-1’: ‘01223511511’, ‘attraction-inform-post-1’: ‘cb17gx’}
U:	{‘attraction-request-type-?’: ‘?’}	U:	{‘attraction-request-type-?’: ‘?’}
S:	{‘attraction-inform-type-1’: ‘museum’}	S:	{}
U:	{‘hotel-inform-price-1’: ‘cheap’, ‘hotel-inform-area-1’: ‘centre’}	U:	{‘attraction-request-type-?’: ‘?’}
S:	{‘hotel-inform-name-1’: ‘alexander bed and breakfast’}	S:	{}
U:	{‘hotel-request-post-?’: ‘?’, ‘hotel-request-phone-?’: ‘?’}	U:	{‘attraction-request-type-?’: ‘?’}
S:	{‘general-reqmore-none-none’: ‘none’, ‘hotel-inform-phone-1’: ‘01223525725’, ‘hotel-inform-post-1’: ‘cb12de’}	S:	{}
U:	{‘hotel-inform-stay-1’: ‘dont care’, ‘hotel-inform-day-1’: ‘dont care’, ‘hotel-inform-people-1’: ‘dont care’}	U:	{‘attraction-request-type-?’: ‘?’}
S:	{‘booking-book-ref-1’: ‘none’}	S:	{}
U:	{‘general-bye-none-none’: ‘none’}	U:	{‘attraction-request-type-?’: ‘?’}
		S:	{}
		U:	{‘attraction-request-type-?’: ‘?’}
		S:	{}
		U:	{‘attraction-request-type-?’: ‘?’}
		S:	{}
		U:	{‘attraction-request-type-?’: ‘?’}
		S:	{}
		U:	{‘attraction-request-type-?’: ‘?’}
		S:	{}
		U:	{‘attraction-request-type-?’: ‘?’}
		S:	{}
		U:	{‘attraction-request-type-?’: ‘?’}
		S:	{}
		U:	{‘attraction-request-type-?’: ‘?’}
		S:	{}
		U:	{‘attraction-request-type-?’: ‘?’}
		S:	{}
		U:	{‘attraction-request-type-?’: ‘?’}
		S:	{}
		U:	{‘attraction-request-type-?’: ‘?’}
		S:	{}
		U:	{‘attraction-request-type-?’: ‘?’}
		S:	{}
		U:	{‘attraction-request-type-?’: ‘?’}
		S:	{}
		U:	{‘attraction-request-type-?’: ‘?’}
		S:	{}
		U:	{‘general-bye-none-none’: ‘none’}
turn: 8 match: 1.0 inform: (1.0, 1.0, 1.0)		turn: 22 match: 0.0 inform: (0, 0, 0)
Success		Failure

.

6. Conclusions

In this paper, we present a novel and effective regularization method known as the variational reward estimator bottleneck (VRB) for multidomain task-oriented dialogue systems. The VRB includes a stochastic encoder, which enables the reward estimator to be maximally informative, and provides information bottleneck regularization, which constrains unproductive information flows between the reward estimator and the inputs. The quantitative results show that VRB achieves new SOTA performances on two different user simulators and a multiturn and multidomain task-oriented dialogue dataset. Despite great improvements, training dialog policy via VHUS setting remains a hurdle to overcome. We leave this for future works.