CG-BERT: Conditional Text Generation with BERT for Generalized Few-shot Intent Detection

The input representation follows that of BERT Devlin et al. 2018. In order to encode both the intent and the utterance, the input in our model is a pair of intent and utterance text sentences packed together.

Given an intent $y=(L_{1},L_{2},...,L_{T_{1}})$ with $T_{1}$ words and an utterance $x=(X_{1},X_{2},...,X_{T_{2}})$ with $T_{2}$ words, we add special start-of-sequence ([CLS]) token at the beginning of the input and a special end-of-sequence ([SEP]) token at the end of each sentence. The first intent sentence, ([CLS], $L_{1}$, $L_{2}$, …, $L_{T_{1}}$, [SEP]), is referred to as $S_{1}$ and the second utterance sentence, ($X_{1}$, $X_{2}$, …, $X_{T_{2}}$, [SEP]) is named as $S_{2}$. As shown in Figure 2, $S_{1}$ and $S_{2}$ are concatenated together as the whole input. [SEP] not only marks the sentence boundary, but also is used for the model to learn when to terminate the decoding process. [CLS] is used as the representation for the whole input and we encode the embeddings for [CLS] to the latent variable $z$.

Texts are tokenized into subword units by WordPiece Wu et al. 2016. As illustrated in Figure 2, three types of embedddings will be obtained for each token: token embeddings, position embeddings Vaswani et al. 2017, and segment embeddings Devlin et al. 2018 which identifies the intent and the utterance. The input representation of a given token is constructed by summing these three embeddings and represented as $\mathbf{H}^{0}=[\mathbf{h}^{0}_{1},...,\mathbf{h}^{0}_{T}]$ with a total length of $T$ tokens.

As shown in Figure 2, the encoder in our model encodes the input (both the intent and the utterance) into a latent variable that models the distribution of diverse utterances for a given intent. To obtain deep bidirectional context information, we utilize multiple bidirectional Transformer encoders Vaswani et al. 2017, namely Transformer blocks in this paper, to encode the input. These Transformer blocks not only extract the semantic information for the input, but also models the attention between the intent tokens and the utterance tokens.

The BERT-base model is used as the backbone of CG-BERT to incorporate the information pre-trained from a large amount of unlabeled data. As BERT-base has 12 transformer blocks, we utilize the first 6 transformer blocks in the encoder and the last 6 blocks are involved in the decoder.

The input representation $\mathbf{H}^{0}=[\mathbf{h}^{0}_{1},...,\mathbf{h}^{0}_{T}]$ is encoded into contextual layer representations through the transformer blocks. For each layer, the layer representation $\mathbf{H}^{l}=[\mathbf{h}^{l}_{1},...,\mathbf{h}^{l}_{T}]$ is computed with an $l$-layer Transformer block $\mathbf{H}^{l}=$ Transformer${}_{l}(\mathbf{H}^{l-1}$), $l\in\{1,2,\cdots,6\}$. In each Transformer block, multiple self-attention heads are used to aggregate the output vectors of the previous layer. For the $l$-th Transformer layer, the output of a self-attention head $\mathbf{A}_{l}$ is computed via:

where the output of the previous layer $\mathbf{H}^{l-1}{\in}\mathbb{R}^{T\times d_{h}}$ is linearly projected to a triple of queries, keys and values parameterized by matrices $\mathbf{W}^{Q}_{l},\mathbf{W}^{l}_{K},\mathbf{W}^{l}_{V}{\in}\mathbb{R}^{d_{h}{\times}d_{k}}$. Following BERT, the embeddings for the [CLS] token in the 6-th transformer block $\mathbf{h}^{6}_{1}$ is regarded as the sentence-level representation.

By modeling the true distribution $p(z|x,y)$ using a known distribution that is easy to sample from Kingma et al. 2014, we encode the sentence-level representation $\mathbf{h}^{6}_{1}$ into a latent variable $z$ whose prior distribution $p(z|y)$ is a multivariate standard Gaussian distribution. We use the reparametrization trick Kingma and Welling 2013 to generate the latent vector $z$. The encoder predicts the parameters $\mu$ and $\sigma$ in the Gaussian distribution such that the proxy posterior $q(z|x,y)=\mathcal{N}(\mu,\sigma)$:

where $\mu$ and $\log(\sigma^{2})$ are projected from $\mathbf{h}^{6}_{1}$ through parameters $\textbf{W}_{\mu}\in\mathbb{R}^{d_{h}\times d_{h}}$, $\textbf{W}_{\sigma}\in\mathbb{R}^{d_{h}\times d_{h}}$, $b_{\mu}\in\mathbb{R}^{d_{h}}$, $b_{\sigma}\in\mathbb{R}^{d_{h}}$, $\varepsilon\in\mathcal{N}(0,1)$ are used to sample $z$.

The decoder aims to reconstruct the input utterance $x$ using the latent variable $z$ and the intent $y$. As shown in Figure 2, we add a residual connection from the input representation $\mathbf{H}^{0}$ to the input of the decoder $\mathbf{H}^{6^{\prime}}$. The input of the decoder, $\mathbf{H}^{6^{\prime}}=[z,\mathbf{h}^{0}_{2},...,\mathbf{h}^{0}_{T}]$, is constructed by concatenating the latent variable $z$ and the input representation $\mathbf{H}^{0}$ which represents the input tokens. The input representation for the [CLS] token, $\mathbf{h}^{0}_{1}$, is ignored since its position is taken by $z$.

The input for the decoder $\mathbf{H}^{6^{\prime}}$ is fed into the last six transformer blocks of BERT-base, as shown in Figure 2. To reconstruct the input utterance, the decoder is trained to generate the utterance in a left-to-right manner, since we can only use the left context to predict the next token. The original BERT model is trained on a large corpus to predict a randomly masked token given its contexts in the sentence. We try to leverage the conditional prediction ability and modify it for text generation in a left-to-right manner.

Inspired by the Unified Language model Dong et al. 2019, which utilizes specific self-attention masks to control what context the prediction conditions on, we adopt the attention mask which helps the transformer blocks fit into the conditional text generation task. Instead of applying the whole bidirectional attention to the input, a mask matrix $\mathbf{M}\in\mathbb{R}^{T\times T}$ is added to determine whether a pair of tokens can be attended to each other. We update the calculation for the attentions in transformer blocks from Equation 3 to:

where

The proposed attention mask matrix for the transformer blocks in the decoder is shown in the upper right corner of Figure 2. To keep the transformers in the encoder having the same structure, we apply the bidirectional self-attention mask for the encoder as shown in the lower right corner of Figure 2. The bidirectional self-attention mask allows all the tokens in $S_{1}$ and $S_{2}$ to attend to all the tokens.

In the attention mask for the decoder, the tokens in the first intent sentence $S_{1}$ can only attend to all the tokens in $S_{1}$, while the tokens in the second sentence $S_{2}$ can attend to both $S_{1}$ and all the left tokens in $S_{2}$. With this attention matrix, we are able to control the decoder to generate the output from left to right conditioned on the given intent.

A special token here is the first token which holds the latent variable $z$, it is only allowed to attend to itself. This is due to the vanishing latent variable problem Zhao et al. 2017 when adapting VAE/CVAE to natural language generators. Previous works Zhao et al. 2017 reported the problem that the LSTM decoder tends to ignore the latent variable. We met the same issue when applying the variational autoencoder with transformers. To alleviate this problem, we only allow the latent variable $z$ to attend to itself to avoid it been overwhelmed by the information of other tokens.

The output of the 12-th transformer block in the decoder is represented as $\mathbf{H}^{12}=[\mathbf{h}^{12}_{1},\mathbf{h}^{12}_{2},...,\mathbf{h}^{12}_{T}]$, where $\mathbf{h}^{12}_{1}$ is the embeddings for the latent variable $z$. To further increase the impact of $z$ and alleviate the vanishing latent variable problem, we concatenate the embedings of $z$ with all the tokens: $\mathbf{H}^{12^{\prime}}=[\mathbf{h}^{12}_{1}\mathbin{\|}\mathbf{h}^{12}_{1},\mathbf{h}^{12}_{2}\mathbin{\|}\mathbf{h}^{12}_{1},...,\mathbf{h}^{12}_{T}\mathbin{\|}\mathbf{h}^{12}_{1}]$. Two fully-connected layers with a layer normalization Ba et al. 2016 are applied to get the final representation:

where $\mathbf{W}_{1}\in\mathbb{R}^{2d_{h}\times d_{h}}$, $\mathbf{W}_{2}\in\mathbb{R}^{d_{h}}$, $f$ is a Gelu Hendrycks and Gimpel 2016 activation function and $g$ is for layer normalization. The embeddings in $\mathbf{H}^{f}$ at position at position $t$ is used to predict the next token at position $t+1$:

where $\mathbf{W_{e}}{\in}\mathbb{R}^{V{\times}d_{h}}$ is the token embedding in the input representation, $b_{e}{\in}\mathbb{R}^{V}$, $V$ is vocabulary size.

We train this model to maximize $\log p(x|y)$, the conditional log-likelihood of the utterance $x$ given an intent $y$, which involves a marginalization over the latent variable $z$. As proposed in Kingma et al. 2014, CVAE can be effectively trained with the Stochastic Gradient Variational Bayes framework by maximizing the evidence lower bound (ELBO) of the log-likelihood. The loss function for the proposed model is:

which consists of the KL divergence loss and the reconstruction loss.

The distributions of the intents are learned after training the CG-BERT. Through sampling from these distributions, we are able to generate utterances for novel intents. As shown in Figure 2, we use the modules in the purple boxes (Input Embedding and Decoder) to generate new utterances. For a given novel intent $y$, an intent sentence $S_{1}$ is fed into the Input Embedding module to get the input representation $\mathbf{H}^{0}$. The input for the decoder is constructed through $\mathbf{H}^{6^{\prime}}=[z,\mathbf{h}^{0}_{2},...,\mathbf{h}^{0}_{T_{1}}]$, where $z$ is sampled from a multivariate standard Gaussian distribution instead of encoded from the input. With the decoder, the new utterance is generated sequentially from left to right.

To generate more utterances, we sample $z$ for $s$ times, and for each $z$, we get the top 20 results using beam search. To increase diversity, we will delete the generated sentences which already exist in the training data. The procedure for generating new utterances for the novel intents is illustrated as:

The generated dataset $\mathcal{D}_{ge}$ are merged with the original training data $\mathcal{D}_{seen},\mathcal{D}_{novel}$ to do intent detection with fine-tuning on BERT.

To demonstrate the effectiveness of our proposed model, we evaluate CG-BERT on two real-word datasets for the generalized few-shot intent detection task: SNIPS-NLU Coucke et al. 2018 and NLU-Evaluation-Data (NLUED) Xingkun Liu and Rieser 2019. These two datasets were collected to benchmark the performance of natural language understanding services offering customized solutions. The statistical information of these two datasets is shown in Table 2.

SNIPS-NLU¹¹ 1 https://github.com/snipsco/nlu-benchmark/ Following Xia et al. 2018, we select two intents (RateBook and AddToPlaylist) as emerging intents (the few-shot classes), while the other five intents are regarded as existing intents. For the GFSID task, we need to do classification among the whole 7 intents. We randomly choose 80% of the whole data as the training data and 20% as the test data. For the few-shot intents, we randomly sample 1 or 5 examples as the few-shots.

To the best of our knowledge, we are the first one to study the GFSID task. Following Shi et al. 2019, we extend two state-of-the-art FSL methods to the GFSID setup as our baselines. From the data augmentation perspective, our model is an over-sampling model which samples sentence level examples from latent distributions. To show the effectiveness of our proposed model, we also compare it with other over-sampling methods.

Five baselines are considered in total: 1) IN⁺. Induction Networks Geng et al. 2019 (IN) is a state-of-the-art few-shot text classification model. It can take any-way any-shot inputs, hence we can easily apply a readily trained IN model to solve the GFSL task. 2) BERT-PN⁺. Prototypical Networks Snell et al. 2017 (PN) is a simple but effective FSL model. To provide a fair comparison, we use BERT as the encoder in PN and finetune BERT together with the PN model. 3) BERT. BERT Devlin et al. 2018 is the state-of-the-art text classification model. We simply over sample the few-shots by copying the few-shot examples to the maximum training examples per class and use the over-sampled dataset to do intent detection with fine-tuning on BERT. 4) BERT-SMOTE. SMOTE Chawla et al. 2002 is an over-sampling method that samples features based on the nearest neighbors. BERT-SMOTE uses fixed pre-trained BERT features. A fully connected layer is used for intent detection with the over-sampled features. 5) SVAE. SVAE Bowman et al. 2015 is a text generation model that utilizes Variational Auto-Encoder (VAE) to model the distribution of the utterances. It can generate similar sentences by sampling from the posterior or get intermediate sentences between two sentences.

We use BERT-base and Adam optimizer (Kingma and Ba 2014) for all the experiments. For CG-BERT, the hidden dimension $d_{h}$ is 768 and the batch size is 16. CG-BERT is trained with learning rate equals 1e-5 in 100 epochs and each epoch has 1000 steps. We sample the latent variable $s=10$ times and choose the top $k=20$ utterances when generating new utterances. For fine-tuning on BERT, we set batch size as 32, learning rate as 2e-5 and the number of the training epochs as 3. For the FSL baselines, we train these models in the episode training method. For SNIPS-NLU, we train these models in 3-way for 1000 episodes. For NLUED, we train these models in 10-way for 1500 episodes.

The generalized few-shot intent detection results on two datasets with 1-shot/5-shot settings are reported in Table 1. In addition to the overall accuracy (Overall) measured on all intents, we follow the convention in GFZL Xian et al. 2017 and report the accuracy on seen intents (Seen), the accuracy on novel intents (Novel) together with their harmonic mean (H-Mean). We report the average and the standard deviation for SNIPS-NLU over 5 runs. The results on NLUED are reported over 10 Folds.

On both two datasets with 1-shot and 5-shot settings, our proposed model CG-BERT achieves state-of-the-art performance on overall accuracy, novel accuracy, and harmonic mean. The improvement mainly stems from the high quality of the generated examples for novel intents, which leads to significantly increased novel accuracy and harmonic mean. CG-BERT achieves the best performance for the seen accuracy on the NLUED 1-shot setting, while obtaining comparable results with BERT on three other settings.

Among the extended few-shot learning models, BERT-PN+ achieves decent performance on the novel accuracy compared to other baselines while it sacrifices the performance on the seen intents. Compared to BERT-PN+, CG-BERT achieves good performance consistently by training the model on both seen intents and novel intents altogether. IN+ lacks the ability to extend from the FSL training to the GFSID setting since it uses a parametric method other than distance-based methods like PN to do classification. It performs poorly on the NLUED dataset ($<$ 20% on the overall accuracy, thus not reported in Table 1).

For the text generation baseline, SVAE performs worse than CG-BERT on the novel intents due to the generated sentences are too similar to the input few-shots. BERT-SMOTE also does not perform well since it uses fixed BERT features and does over-sampling by exploring the interpolation among the few-shots in the feature space.

We show some examples generated by CG-BERT for two novel intents in Table 3. The generated utterances for Rate Book show that CG-BERT is able to generate similar utterances to the few-shots by adding expressions like “can you” or replacing words (“like” and “want”). The cases in Alarm Query are pretty diverse. It learns from other intents like “Calendar Query” that expressions like “show me” can be used for request information. It also generates utterances that are totally different from the few-shots, like “do i have any alarms”.

There are also some bad cases when the intent is not fully conditioned as expected. For example, “am i going to need a jacket tomorrow” which has the intent of Query Weather, but generated for Alarm Query. In other cases, sentences are not well generated and may have syntax errors, like “alarms me are there any alarms for me”.

We further evaluate the quality of the generated utterances both quantitatively and qualitatively. Quantitative metrics like the percentage of unique sentences and n-grams are used. Sentences are manually annotated for qualitative analysis. We use the Fold 7 of the NLUED dataset with the 5 shot setting as an example. There are 16 novel intents in total and each novel intent has 5 examples. We randomly sample the hidden variable z for once and get the top 20 generated sentences for each intent. After deduplication, we got 257 unique sentences out of 320, which is around 80%.

Quantitatively, we compare the average percentage of new n-grams in the generated sentences to a text generation model SVAE Bowman et al. 2015. As shown in Table 4, CG-BERT is able to generate more new n-grams which present the diversity of the generated sentences. Qualitatively, we asked human annotators to annotate the syntax quality (result: 4.3 out of 5) and intent coherency (result: 4.7 out of 5) of the generated utterances.

We visualize the embedding space of the generated utterances using t-SNE for BERT-SMOTE and CG-BERT. As shown in Figure 3, the generated examples from CG-BERT for novel intents are diverse and well separated from the seen intents, while BERT-SMOTE tends to generate similar examples on the feature level as in Figure 4. SMOTE only generates new features within the few-shots, while CG-BERT is able to generate diverse examples beyond these five shots by transfer expressions from existing intents.