variational_autoencoder.txt

variational_autoencoder.txt (2253B)
      1 # Variational Autoencoder (VAE)
      2 
      3 A variational autoencoder is a generative model that learns to map data x into
      4 a low-dimensional latent distribution z, then reconstruct x from samples of z.
      5 Unlike a plain autoencoder (deterministic bottleneck), the encoder outputs
      6 parameters of a distribution (usually Gaussian: mean and log-variance), and the
      7 decoder is trained on stochastic latent samples.
      8 
      9   x -> Encoder q(z|x) -> z ~ N(mu, sigma^2) -> Decoder p(x|z) -> x_hat
     10 
     11 Training maximizes the evidence lower bound (ELBO):
     12 
     13   L = E_q[log p(x|z)] - KL(q(z|x) || p(z))
     14 
     15 The reconstruction term encourages faithful outputs; the KL term regularizes
     16 latents toward a prior p(z) (typically N(0, I)), making the latent space
     17 continuous and sampleable. At inference, draw z ~ p(z) and decode to generate
     18 new data. The reparameterization trick (sample z = mu + sigma * epsilon,
     19 epsilon ~ N(0,I)) makes gradients flow through the stochastic encoder.
     20 
     21 
     22 ## Key papers
     23 
     24   Kingma & Welling (2013/2014)
     25     "Auto-Encoding Variational Bayes"
     26     ICLR 2014, arXiv:1312.6114
     27     https://arxiv.org/abs/1312.6114
     28     Foundational VAE: amortized variational inference + reparameterization.
     29 
     30   Rezende, Mohamed & Wierstra (2014)
     31     "Stochastic Backpropagation and Approximate Inference in Deep Generative Models"
     32     ICML 2014, arXiv:1401.4082
     33     https://arxiv.org/abs/1401.4082
     34     Independent development of the same variational inference idea for deep
     35     generative models.
     36 
     37   Higgins et al. (2017)
     38     "beta-VAE: Learning Basic Visual Concepts with a Constrained Variational
     39     Framework"
     40     ICLR 2017, arXiv:1804.03599
     41     https://arxiv.org/abs/1804.03599
     42     beta-VAE: scales KL weight to encourage disentangled latent factors.
     43 
     44   van den Oord, Vinyals et al. (2017)
     45     "Neural Discrete Representation Learning" (VQ-VAE)
     46     NeurIPS 2017, arXiv:1711.00937
     47     https://arxiv.org/abs/1711.00937
     48     Important variant: discrete latent codes via vector quantization instead of
     49     continuous Gaussians.
     50 
     51 
     52 ## Typical uses
     53 
     54   - generative modeling (sample novel images, shapes, audio)
     55   - learning compressed continuous representations for downstream models
     56   - latent-space interpolation and editing
     57   - component in larger pipelines (diffusion latents, 3D shape models)
	notes
	Log \| Files \| Refs
notes
