commit 4b858ed1beca1e453e62cc6c09367ba3f4f4d8d6
parent dfb5b0b7e5675f2d282eeb3e5281d4a9b093135f
Author: ling0x <ling0x@users.noreply.github.com>
Date: Fri, 10 Jul 2026 11:51:42 +0100
ai
Diffstat:
7 files changed, 278 insertions(+), 0 deletions(-)
diff --git a/machine_learning/concepts.txt b/artificial_intelligence/concepts.txt
diff --git a/artificial_intelligence/kl_regularization.txt b/artificial_intelligence/kl_regularization.txt
@@ -0,0 +1,84 @@
+# KL Regularization
+
+KL regularization adds a Kullback-Leibler (KL) divergence penalty to a loss
+function. It measures how much one probability distribution diverges from
+another. Used as regularization, it pulls a learned distribution toward a
+target (usually a simple prior), preventing the model from collapsing to
+degenerate or overly complex representations.
+
+
+## Definition
+
+For distributions q and p over the same variable z:
+
+ KL(q || p) = E_q[log q(z) - log p(z)]
+
+ - KL >= 0, and KL = 0 only when q = p almost everywhere
+ - not symmetric: KL(q || p) != KL(p || q)
+ - penalizes q for placing mass where p has little mass
+
+
+## In variational autoencoders
+
+In a VAE, the encoder learns an approximate posterior q(z|x) (e.g. Gaussian
+with learned mean and variance). KL regularization matches it to a prior p(z)
+(usually N(0, I)):
+
+ L = E_q[log p(x|z)] - KL(q(z|x) || p(z))
+ ^ reconstruction ^ KL regularization
+
+Effects:
+ - makes the latent space smooth and continuous (nearby z decode to similar x)
+ - lets you sample z ~ p(z) at inference to generate new data
+ - prevents the encoder from memorizing x as a unique deterministic code
+
+Tradeoff: too much KL weight hurts reconstruction; too little yields
+unstructured latents or posterior collapse (encoder ignores z, decoder
+reconstructs from bias alone).
+
+beta-VAE scales the KL term by beta > 1 to push stronger regularization and
+encourage more disentangled factors (Higgins et al., 2017).
+
+
+## Closed form (Gaussian case)
+
+When q(z|x) = N(mu, diag(sigma^2)) and p(z) = N(0, I):
+
+ KL(q || p) = -0.5 * sum_i (1 + log(sigma_i^2) - mu_i^2 - sigma_i^2)
+
+This is cheap to compute and differentiable, so it is added directly to the
+training loss each batch.
+
+
+## Other common uses
+
+ - RL / policy optimization: KL(policy_new || policy_old) caps how far the
+ policy moves per update (TRPO, PPO)
+ - knowledge distillation: KL(student || teacher) transfers soft label
+ distributions
+ - variational inference generally: any model with an intractable posterior
+ can be trained by minimizing KL between an approximate and true posterior
+
+
+## Key papers
+
+ Kingma & Welling (2013/2014)
+ "Auto-Encoding Variational Bayes"
+ arXiv:1312.6114
+ https://arxiv.org/abs/1312.6114
+ VAE loss = reconstruction - KL to prior; foundational use of KL as
+ regularization in deep generative models.
+
+ Higgins et al. (2017)
+ "beta-VAE: Learning Basic Visual Concepts with a Constrained Variational
+ Framework"
+ arXiv:1804.03599
+ https://arxiv.org/abs/1804.03599
+ Tunable KL weight (beta) to balance reconstruction vs latent structure.
+
+ Bowman et al. (2016)
+ "Generating Sentences from a Continuous Space"
+ arXiv:1511.06349
+ https://arxiv.org/abs/1511.06349
+ Early discussion of KL annealing to avoid posterior collapse in
+ sequence VAEs.
diff --git a/machine_learning/variational_autoencoders.txt b/artificial_intelligence/vae_vs_3d_morph.txt
diff --git a/artificial_intelligence/variational_autoencoder.txt b/artificial_intelligence/variational_autoencoder.txt
@@ -0,0 +1,57 @@
+# Variational Autoencoder (VAE)
+
+A variational autoencoder is a generative model that learns to map data x into
+a low-dimensional latent distribution z, then reconstruct x from samples of z.
+Unlike a plain autoencoder (deterministic bottleneck), the encoder outputs
+parameters of a distribution (usually Gaussian: mean and log-variance), and the
+decoder is trained on stochastic latent samples.
+
+ x -> Encoder q(z|x) -> z ~ N(mu, sigma^2) -> Decoder p(x|z) -> x_hat
+
+Training maximizes the evidence lower bound (ELBO):
+
+ L = E_q[log p(x|z)] - KL(q(z|x) || p(z))
+
+The reconstruction term encourages faithful outputs; the KL term regularizes
+latents toward a prior p(z) (typically N(0, I)), making the latent space
+continuous and sampleable. At inference, draw z ~ p(z) and decode to generate
+new data. The reparameterization trick (sample z = mu + sigma * epsilon,
+epsilon ~ N(0,I)) makes gradients flow through the stochastic encoder.
+
+
+## Key papers
+
+ Kingma & Welling (2013/2014)
+ "Auto-Encoding Variational Bayes"
+ ICLR 2014, arXiv:1312.6114
+ https://arxiv.org/abs/1312.6114
+ Foundational VAE: amortized variational inference + reparameterization.
+
+ Rezende, Mohamed & Wierstra (2014)
+ "Stochastic Backpropagation and Approximate Inference in Deep Generative Models"
+ ICML 2014, arXiv:1401.4082
+ https://arxiv.org/abs/1401.4082
+ Independent development of the same variational inference idea for deep
+ generative models.
+
+ Higgins et al. (2017)
+ "beta-VAE: Learning Basic Visual Concepts with a Constrained Variational
+ Framework"
+ ICLR 2017, arXiv:1804.03599
+ https://arxiv.org/abs/1804.03599
+ beta-VAE: scales KL weight to encourage disentangled latent factors.
+
+ van den Oord, Vinyals et al. (2017)
+ "Neural Discrete Representation Learning" (VQ-VAE)
+ NeurIPS 2017, arXiv:1711.00937
+ https://arxiv.org/abs/1711.00937
+ Important variant: discrete latent codes via vector quantization instead of
+ continuous Gaussians.
+
+
+## Typical uses
+
+ - generative modeling (sample novel images, shapes, audio)
+ - learning compressed continuous representations for downstream models
+ - latent-space interpolation and editing
+ - component in larger pipelines (diffusion latents, 3D shape models)
diff --git a/artificial_intelligence/variational_autoregressive_transformer.txt b/artificial_intelligence/variational_autoregressive_transformer.txt
@@ -0,0 +1,118 @@
+# Variational Autoregressive Transformer (VART)
+
+Source paper:
+ B-repLer: Language-guided Editing of CAD Models
+ Liu et al., arXiv:2508.10201 (SIGGRAPH 2026)
+ https://arxiv.org/abs/2508.10201
+ https://arxiv.org/pdf/2508.10201
+
+
+## What it is
+
+A variational autoregressive transformer (VART) is a sequence model that
+generates outputs one token at a time (autoregressive), but samples each
+token from a continuous distribution (variational) instead of predicting a
+single deterministic value or a discrete codebook entry.
+
+It combines:
+ - a transformer encoder/decoder for sequence structure and conditioning
+ - a flow matching network for stochastic token generation at each step
+
+Related idea: MAR (Autoregressive Image Generation without Vector Quantization,
+Li et al., NeurIPS 2024) -- generate continuous latents autoregressively
+without vector quantization.
+
+
+## Core idea
+
+Standard autoregressive transformers predict the next token directly (often
+discrete). A VART splits each step into two parts:
+
+ 1. Transformer decoder predicts an intermediate conditioning feature
+ from context + previously generated tokens.
+
+ 2. A flow matching network samples the next continuous latent token by
+ denoising Gaussian noise, conditioned on that intermediate feature.
+
+The "variational" part comes from sampling z ~ N(0, I) and learning a flow
+from noise to the target token. This models uncertainty when one input can
+map to many valid outputs (one-to-many).
+
+
+## Architecture (B-repLer example)
+
+Task: text-guided editing of CAD B-rep models in a learned latent space
+(HoLa-BRep encoder, 32-dim latent per face).
+
+Encoder (multimodal context):
+ - rendered image (DINOv2)
+ - text instruction (mLLM embedding)
+ - source B-rep face latents (32-d -> 768-d projection)
+ - per-face image crops (RoIAlign from image feature map)
+ - optional 2D bounding box
+ -> fused by transformer encoder -> context F_src
+
+Decoder (autoregressive + variational):
+ - causal transformer decoder attends to F_src and prior tokens
+ - at step t, predicts intermediate feature F_inter^t (not the token itself)
+ - flow matching network conditions on F_inter^t and generates H_a^t in R^32
+ by integrating a learned flow from noise over ~100 denoising steps
+ - generated token is projected back to 768-d and fed to the next step
+ - binary EOS classifier stops generation for variable-length outputs
+
+Post-decode:
+ - latent sequence -> HoLa-BRep decoder -> edited B-rep CAD model
+
+
+## Why use it
+
+Handles three hard problems at once:
+
+ 1. Variable-length sequences
+ B-rep models have different numbers of faces. Autoregressive decoding
+ with an EOS token avoids fixed-length padding.
+
+ 2. One-to-many ambiguity
+ The same text edit ("make it rounder") can yield multiple valid shapes.
+ Stochastic token sampling captures output diversity.
+
+ 3. Continuous latents without VQ
+ Works directly on continuous 32-d face embeddings. No vector
+ quantization step, unlike many AR generative models on images or 3D.
+
+
+## Inference loop (per token)
+
+ t = 1, 2, ... until EOS:
+ F_inter^t = TransformerDecoder(F_src, {H_a^0 ... H_a^{t-1}})
+ H_a^t = FlowMatch.sample(F_inter^t) # noise -> latent, 100 steps
+ append H_a^t to history
+
+
+## Ablations from the paper
+
+Compared on BrepEDIT-240K text-driven B-rep editing:
+
+ Deterministic AR transformer:
+ - cannot model one-to-many edits
+ - validity drops; often missing geometry
+
+ Pure flow matching (no AR):
+ - struggles with variable-length latents
+ - padding/unpadding adds noise; worse metrics
+
+ Full VART:
+ - best validity and edit quality
+ - diverse, prompt-aligned outputs
+
+
+## When to reach for a VART
+
+Use when you need to:
+ - generate or edit structured sequences (faces, patches, tokens)
+ - operate in a continuous latent space (VAE/autoencoder latents)
+ - support ambiguous or multimodal outputs from one conditioning input
+ - avoid discrete codebooks / vector quantization
+
+B-repLer applies this to CAD: natural language + source geometry ->
+edited latent face sequence -> valid B-rep model, without construction history.
diff --git a/commands/gawk.txt b/commands/gawk.txt
@@ -1,3 +1,9 @@
+Awk
+
+Remove all ollama models locally:
+
+ollama list | awk 'NR>1 {print $1}' | xargs -r ollama rm
+
================================================================================
GAWK — quick notes (Arch Linux)
diff --git a/commands/time.txt b/commands/time.txt
@@ -0,0 +1,12 @@
+time
+
+- Action: A shell built-in that measures how long the subsequent command takes to execute.
+- Output: After the build finishes, it will print three values to the
+ terminal:
+ - real: The actual elapsed "wall clock" time.
+ - user: The amount of CPU time spent in user-mode.
+ - sys: The amount of CPU time spent in kernel-mode.
+
+Example:
+
+time wasm-pack build --target web
+\ No newline at end of file
