Riemannian Normalizing Flow on Variational Wasserstein Autoencoder for Text Modeling

R
i
e
m
a
n
n
i
a
n
 
N
o
r
m
a
l
i
z
i
n
g
 
F
l
o
w
 
o
n
V
a
r
i
a
t
i
o
n
a
l
 
W
a
s
s
e
r
s
t
e
i
n
 
A
u
t
o
e
n
c
o
d
e
r
 
f
o
r
T
e
x
t
 
M
o
d
e
l
i
n
g
Prince Wang, William Wang
UC Santa Barbara
1
Outline
VAE and the KL vanishing problem
Motivation: why Riemannian Normalizing flow/WAE
Details
Experimental Results
2
VAE: KL vanishing
 
KL term, gap between posterior and prior
3
Can generate sentences given latent
codes z
i were to buy any groceries . 
horses are to buy any groceries .
horses are to buy any animal .
horses the favorite any animal .
Previous works
 
Generating sentences from Continuous Space, (2015,
Bowman)
Improved Variational Autoencoder for text Modeling
using Dilated Convolution, (2017, Yang)
Spherical Latent Spaces for Stable Variational
Autoencoder, (2018, Xu)
Semi-Amortized VAE, (2018, Kim)
Cyclical Annealing Schedule: A Simple Approach to
Mitigating KL Vanishing, (2019, Fu)
4
Riemannian Normalizing Flow/Wasserstein Distance
 
 
5
Normalizing Flow
https://lilianweng.github.io/lil-log/2018/10/13/flow-
based-deep-generative-models.html
6
Making posterior harder to collapse to a standard Gaussian prior
Normalizing Flow
Tighter likelihood approximation
7
Reconstruction
    KL
Jacobian
Why Riemannian VAE?
 
The Latent space is not flat Euclidean. It should be
curved.
 
 
8
 
9
Riemannian Metric
Jacobian
Rie. Metric
Match latent manifold with input manifold
10
Curve
Length
Modeling curvature by NF
Planar Flow
              
Curvature:
11
Modeling curvature by NF
To match geometry of latent space with input space, we
need this determinant to be large when input manifold has
high curvature
Jacobian
12
Wasserstein Distance
Replace KL with Maximum Mean Discrepancy
(MMD)
Wasserstein Autoencoder, (ICLR 2018, 
Ilya Tolstikhin)
13
Wasserstein RNF
Reconstruction
MMD loss
KLD loss with NF
14
15
Results
Language Models: Negative Log-likelihood/KL/Perplexity
Results: KL divergence
 
16
PTB
Yelp
Results: Negative log-likelihood
17
PTB
Yelp
   WAE
   WAE-NF
   WAE-RNF
   WAE
   WAE-NF
   WAE-RNF
104
   92
 91
198
184
183
Mutual Information
 
Mutual information
18
Conclusion
 
Propose to use Normalizing Flow and Wasserstein
Distance for variational language model
Design Riemannian Normalizing Flow to learn a smooth
latent space
Empirical results indicate that Riemannian Normalizing
Flow with Wasserstein Distance help avert KL vanishing
Code: 
https://github.com/kingofspace0wzz/wae-rnf-lm
19
 
20
 
Thank you!     Q & A   :)
Code: 
https://github.com/kingofspace0wzz/wae-rnf-lm
Paper: 
https://arxiv.org/abs/1904.02399
Slide Note
Embed
Share

This study explores the use of Riemannian Normalizing Flow on Variational Wasserstein Autoencoder (WAE) to address the KL vanishing problem in Variational Autoencoders (VAE) for text modeling. By leveraging Riemannian geometry, the Normalizing Flow approach aims to prevent the collapse of the posterior distribution to a standard Gaussian prior, enabling more accurate sentence generation from latent codes. Experimental results demonstrate the effectiveness of this method in mitigating the KL vanishing issue.


Uploaded on Nov 25, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Riemannian Normalizing Flow on Variational Wasserstein Autoencoder for Text Modeling Prince Wang, William Wang UC Santa Barbara 1

  2. Outline VAE and the KL vanishing problem Motivation: why Riemannian Normalizing flow/WAE Details Experimental Results 2

  3. VAE: KL vanishing Can generate sentences given latent codes z i were to buy any groceries . horses are to buy any groceries . horses are to buy any animal . horses the favorite any animal . KL term, gap between posterior and prior 3

  4. Previous works Generating sentences from Continuous Space, (2015, Bowman) Improved Variational Autoencoder for text Modeling using Dilated Convolution, (2017, Yang) Spherical Latent Spaces for Stable Variational Autoencoder, (2018, Xu) Semi-Amortized VAE, (2018, Kim) Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing, (2019, Fu) 4

  5. Riemannian Normalizing Flow/Wasserstein Distance 5

  6. Normalizing Flow Making posterior harder to collapse to a standard Gaussian prior https://lilianweng.github.io/lil-log/2018/10/13/flow- based-deep-generative-models.html 6

  7. Normalizing Flow Tighter likelihood approximation Reconstruction Jacobian KL 7

  8. Why Riemannian VAE? The Latent space is not flat Euclidean. It should be curved. 8

  9. Riemannian Metric Jacobian Rie. Metric 9

  10. Match latent manifold with input manifold Curve Length 10

  11. Modeling curvature by NF Planar Flow Curvature: 11

  12. Modeling curvature by NF To match geometry of latent space with input space, we need this determinant to be large when input manifold has high curvature Jacobian 12

  13. Wasserstein Distance Wasserstein Autoencoder, (ICLR 2018, Ilya Tolstikhin) Replace KL with Maximum Mean Discrepancy (MMD) 13

  14. Wasserstein RNF Reconstruction MMD loss KLD loss with NF 14

  15. Results Language Models: Negative Log-likelihood/KL/Perplexity 15

  16. Results: KL divergence Yelp PTB 16

  17. Results: Negative log-likelihood PTB Yelp 104 198 92 91 184 183 WAE WAE-NF WAE-RNF WAE WAE-NF WAE-RNF 17

  18. Mutual Information Mutual information 18

  19. Conclusion Propose to use Normalizing Flow and Wasserstein Distance for variational language model Design Riemannian Normalizing Flow to learn a smooth latent space Empirical results indicate that Riemannian Normalizing Flow with Wasserstein Distance help avert KL vanishing Code: https://github.com/kingofspace0wzz/wae-rnf-lm 19

  20. Thank you! Q & A :) Code: https://github.com/kingofspace0wzz/wae-rnf-lm Paper: https://arxiv.org/abs/1904.02399 20

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#