Information
WSDM 2020
Samsung-PDMI Joint AI Center, Steklov Institute of Mathematics at St. Petersburg, Russia
Samsung-PDMI Joint AI Center는 삼성과 러시아의 스테클로프 수학연구소(Steklov Institute of Mathematics) 사이의 협력으로 설립된 연구 센터입니다. 이 센터는 인공지능(AI) 분야의 연구와 개발을 촉진하는 것을 목표로 하며, 수학적 이론과 AI 기술을 융합하여 혁신적인 알고리즘 및 시스템을 개발하는 데 중점을 둡니다.\ (https://news.samsung.com/global/samsung-electronics-launches-ai-center-in-russia)
스테클로프 수학연구소(Steklov Institute of Mathematics)는 러시아의 수학 연구를 선도하는 기관 중 하나로, 이론 수학과 응용 수학 분야에서 중요한 연구를 진행하는 세계적인 연구소입니다. 특히, 수학적 모델링과 AI 관련 수학적 기법을 활용한 연구가 활발히 이루어지고 있습니다.
ABSTRACT & 1 INTRODUCTION
In this work, we propose the Recommender VAE (RecVAE) model for collaborative filtering with implicit feedback based on the variational autoencoder (VAE) and specifically on the Mult-VAE approach.
RecVAE introduces several novel ideas to improve Mult-VAE, including
1) a new architecture for the encoder network
2) a novel composite prior distribution for the latent codes: The composite prior is a mixture of a standard Gaussian prior and the latent code distribution with parameters fixed from the previous iteration of the model.
3) a new approach to setting the β hyperparameter for the β-VAE framework: We have found that $\beta$ should be user-specific, $\beta = \beta(x_u)$, and should depend on the amount of data (implicit feedback) available for a given user.
4) a new approach to training based on alternating updates: training is done by alternating updates for the encoder and decoder
3 PROPOSED APPROACH
3.2 Model Architecture
Our model is inherited from Mult-VAE, but we also suggest some architecture changes. The general architecture is shown on Figure 1; the figure reflects some of the novelties we will discuss below in this section
1) Denoising VAE:
$$L_{Mult-VAE} = \mathbb{E}_{q_\phi(z_u | \tilde{x}_u)} \mathbb{E}_{p(\tilde{x}_u | x_u)} \left[ \log p_\theta(x_u | z_u) - \beta \text{KL} \left( q_\phi(z_u | x_u) \| p(z_u) \right) \right] $$
2) change the underlying neural networks as shown in Figure 3: it uses the ideas of densely connected layers from dense CNNs[14], swish activation functions [26], and layer normalization[20].
3.3 Composite prior
Mult-VAE의 입력과 출력는 높은 차원의 sparse vectors이다. 또한 posterior를 학습하는 동안, 데이터의 일부에 대한 posterior 업데이트가 또 다른 데이터와 관련된 파라미터에 악영향을 끼칠 수 있다. 이는 훈련의 불안정성을 가져오며 "forgetting" 현상으로 잘 알려져있다. 이 문제를 해결하기 위해 파라미터를 업데이트하기 위해서 다음과 같은 식을 사용한다.
$$ p(z|\phi_{\text{old}}, x) = \alpha \mathcal{N}(z|0, I) + (1 - \alpha) q_{\phi_{\text{old}}}(z|x), 0 \leq \alpha \leq 1 $$
첫 번째 항은 Gaussian 분포로부터 샘플링된 latent code $z$를 더해 모델이 훈련 데이터에 너무 과도하게 적응되지 않도록 도와 새로운 데이터에 대해 잘 일반화할 수 있게 한다. 두 번째 항은 이전 epoch의 파라미터 $\phi_{old}$에 대한 분포에 $1-\alpha$를 곱함으로써 파라미터를 최적화하는 과정에서 너무 큰 변화가 일어나지 않도록 조절해 학습의 안정성을 높인다.
3.4 Rescaling KL divergence
the question of how to choose or change $\beta$ is still not solved conclusively. In our experiments, we have not found any improvements when $\beta$ is set to increase over some schedule, so we propose to keep scale factor fixed. we propose an alternative view on KL divergence rescaling.
scaling factor는 $|X_u^o| / |X_u^f| $로 결정된다. observed data $x_{u}^{o}$가 주어졌을 때 $z_u$일 확률 분포는 unobserved data도 포함하는 $x_u^f$가 주어졌을 때 $z_u$일 확률 분포와 비슷하거나 같아야한다, $q_{\phi}(z_u|x_u)$. 그러므로 observerd data를 기반으로 계산되는 $KL$ 항을 $|X_u^o|/|X_u^f|$로 조정하여 두 분포의 유사성이 반영되도록 한다. 여기서 $|X_u^o|$는 user $u$가 상호작용한 모든 아이템 $i$의 개수이며, $|X_u^f|$는 user와 상관없이 데이터셋에 존재하는 모든 아이템의 개수이다. 그러므로 $|X_u^f|$는 모든 유저에 대해 공통된 파라미터 $ \gamma $로 표현할 수 있다. 결론적으로 $\beta$는 다음과 같이 표현된다.
$$ \beta' = \beta'(x_u) = \gamma |X_{o_u}| = \gamma \sum_i x_{ui} $$
3.5 Alternating Training and Regularization by Denoising
We propose to update $\phi$ and $\theta$ alternately with a different number of iterations: since the encoder is a much more complex network than the decoder, we make multiple updates of $\phi$ for each update of $\theta$.
we noticed that performance improves if we do not corrupt input data during the training of $\theta$ and leave the denoising purely for $\phi$. Thus, we propose to train the decoder as part of a basic vanilla VAE, with no denoising applied. As for the encoder, however, we train it as part of the denoising variational autoencoder. The dropout layer on Figure 3 serves for noising; it is turned off during evaluation, decoder learning phase, and in the composite prior.
3.6 Summary
To summarize the proposed regularizers and changes, we first write down the ELBO for our model:
$$ \mathcal{L} = \mathbb{E}_{q_\phi(z | \tilde{x})} \mathbb{E}_{q(\tilde{x} | x)} \left[ \log p_\theta(x | z) - \beta' (x) \text{KL} \left( q_\phi(z | \tilde{x}) \| p(z | \phi_{\text{old}}, x) \right) \right]$$
To keep the input uncorrupted while training the decoder, we introduce a modified objective function for θ updates (we skip the KL term entirely because it does not depend on θ):
$$ \mathcal{L}_{\text{dec}} = \mathbb{E}_{q_\phi(z | x)} \log p_\theta(x | z)$$
Since we use the multinomial likelihood, the main component of the loss function is the classification cross-entropy as shown on Figure 1. Similar to Mult-VAE, our model is also able to make predictions for users whose feedback was not observed during training.
4 EXPERIMENTAL EVALUATION
4.5 Results
1) RecVAE outperforms all previous autoencoderbased models across all datasets in the comparison.
2) new features of RecVAE and RaCT are independent and can be used together for even higher performance.
3) the proposed model significantly outperforms EASE( Embarrassingly Shallow Autoencoder)only on MovieLens-20M datasets, and shows competitive performance on the Netflix Prize Dataset.
4.6 Ablation study and negative results
1) the new architecture does not bring much improvement by itself but facilitates other new features
2) the composite prior fixes the “forgetting” problem
3) $\beta$ rescaling is dataset-sensitive, sometimes improving a lot and sometimes doing virtually nothing.
4) $\beta$ rescaling and alternating training degrade the scores when applied individually, but together improve them.
5 CONCLUSION
As a result, performance of RecVAE is comparable to EASE and significantly outperforms other models on classical collaborative filtering datasets such as MovieLens-20M, Netflix Prize Dataset, and Million Songs Dataset.
We note that while we have provided certain theoretical motivations for our modifications, these motivations are sometimes incomplete, and some of our ideas have been primarily motivated by practical improvements.