notes

Log | Files | Refs

variational_autoregressive_transformer.txt (4130B)

      1 # Variational Autoregressive Transformer (VART)
      2 
      3 Source paper:
      4   B-repLer: Language-guided Editing of CAD Models
      5   Liu et al., arXiv:2508.10201 (SIGGRAPH 2026)
      6   https://arxiv.org/abs/2508.10201
      7   https://arxiv.org/pdf/2508.10201
      8 
      9 
     10 ## What it is
     11 
     12 A variational autoregressive transformer (VART) is a sequence model that
     13 generates outputs one token at a time (autoregressive), but samples each
     14 token from a continuous distribution (variational) instead of predicting a
     15 single deterministic value or a discrete codebook entry.
     16 
     17 It combines:
     18   - a transformer encoder/decoder for sequence structure and conditioning
     19   - a flow matching network for stochastic token generation at each step
     20 
     21 Related idea: MAR (Autoregressive Image Generation without Vector Quantization,
     22 Li et al., NeurIPS 2024) -- generate continuous latents autoregressively
     23 without vector quantization.
     24 
     25 
     26 ## Core idea
     27 
     28 Standard autoregressive transformers predict the next token directly (often
     29 discrete). A VART splits each step into two parts:
     30 
     31   1. Transformer decoder predicts an intermediate conditioning feature
     32      from context + previously generated tokens.
     33 
     34   2. A flow matching network samples the next continuous latent token by
     35      denoising Gaussian noise, conditioned on that intermediate feature.
     36 
     37 The "variational" part comes from sampling z ~ N(0, I) and learning a flow
     38 from noise to the target token. This models uncertainty when one input can
     39 map to many valid outputs (one-to-many).
     40 
     41 
     42 ## Architecture (B-repLer example)
     43 
     44 Task: text-guided editing of CAD B-rep models in a learned latent space
     45 (HoLa-BRep encoder, 32-dim latent per face).
     46 
     47 Encoder (multimodal context):
     48   - rendered image (DINOv2)
     49   - text instruction (mLLM embedding)
     50   - source B-rep face latents (32-d -> 768-d projection)
     51   - per-face image crops (RoIAlign from image feature map)
     52   - optional 2D bounding box
     53   -> fused by transformer encoder -> context F_src
     54 
     55 Decoder (autoregressive + variational):
     56   - causal transformer decoder attends to F_src and prior tokens
     57   - at step t, predicts intermediate feature F_inter^t (not the token itself)
     58   - flow matching network conditions on F_inter^t and generates H_a^t in R^32
     59     by integrating a learned flow from noise over ~100 denoising steps
     60   - generated token is projected back to 768-d and fed to the next step
     61   - binary EOS classifier stops generation for variable-length outputs
     62 
     63 Post-decode:
     64   - latent sequence -> HoLa-BRep decoder -> edited B-rep CAD model
     65 
     66 
     67 ## Why use it
     68 
     69 Handles three hard problems at once:
     70 
     71   1. Variable-length sequences
     72      B-rep models have different numbers of faces. Autoregressive decoding
     73      with an EOS token avoids fixed-length padding.
     74 
     75   2. One-to-many ambiguity
     76      The same text edit ("make it rounder") can yield multiple valid shapes.
     77      Stochastic token sampling captures output diversity.
     78 
     79   3. Continuous latents without VQ
     80      Works directly on continuous 32-d face embeddings. No vector
     81      quantization step, unlike many AR generative models on images or 3D.
     82 
     83 
     84 ## Inference loop (per token)
     85 
     86   t = 1, 2, ... until EOS:
     87     F_inter^t = TransformerDecoder(F_src, {H_a^0 ... H_a^{t-1}})
     88     H_a^t     = FlowMatch.sample(F_inter^t)   # noise -> latent, 100 steps
     89     append H_a^t to history
     90 
     91 
     92 ## Ablations from the paper
     93 
     94 Compared on BrepEDIT-240K text-driven B-rep editing:
     95 
     96   Deterministic AR transformer:
     97     - cannot model one-to-many edits
     98     - validity drops; often missing geometry
     99 
    100   Pure flow matching (no AR):
    101     - struggles with variable-length latents
    102     - padding/unpadding adds noise; worse metrics
    103 
    104   Full VART:
    105     - best validity and edit quality
    106     - diverse, prompt-aligned outputs
    107 
    108 
    109 ## When to reach for a VART
    110 
    111 Use when you need to:
    112   - generate or edit structured sequences (faces, patches, tokens)
    113   - operate in a continuous latent space (VAE/autoencoder latents)
    114   - support ambiguous or multimodal outputs from one conditioning input
    115   - avoid discrete codebooks / vector quantization
    116 
    117 B-repLer applies this to CAD: natural language + source geometry ->
    118 edited latent face sequence -> valid B-rep model, without construction history.